CN105426419A - System and method for data promotion among heterogeneous systems - Google Patents
System and method for data promotion among heterogeneous systems Download PDFInfo
- Publication number
- CN105426419A CN105426419A CN201510735226.0A CN201510735226A CN105426419A CN 105426419 A CN105426419 A CN 105426419A CN 201510735226 A CN201510735226 A CN 201510735226A CN 105426419 A CN105426419 A CN 105426419A
- Authority
- CN
- China
- Prior art keywords
- data
- promotion
- transferred
- promoted
- major key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a system and method for data promotion among heterogeneous systems. The system for data promotion among heterogeneous systems comprises a dynamic paging controller, a promotion manner management unit, a promotion core processor and a promotion log management unit, wherein the dynamic paging controller is used for reading data by adopting dynamic secondary paging; the promotion manner management unit is used for selecting the promotion manner required by each promotion and selecting synchronous or asynchronous promotion according to the configuration conditions of the new and old systems so as to generate a scheduling configuration file; the promotion core processor is used for processing the data according to the scheduling configuration file generated by the promotion manner management unit; and the promotion log management unit is used for collecting the key log data of each promotion operation. The four components in the system cooperate with one another to realize the advantages of ensuring the data integrity and consistency and improving the system expansibility.
Description
Technical field
The present invention relates to computer realm, particularly, relate to data promotion system and method between a kind of heterogeneous system.
Background technology
At present, along with the development of infotech, network technology, mechanics of communication, database technology.Competition between enterprise is not only competition in product quality and service, the more important thing is and is combined the core competitiveness improving enterprise with management philosophy, fast responding market demand by infotech, network technology, mechanics of communication and database technology.The each mechanism of promoting enterprise, collaboration capabilities between all departments and each employee and link up whenever and wherever possible.
In the process of IT application in enterprises, due to the expansion of business administration and scope of business, may successively use different infosystems.Under current large data environment background, data have become and have improved enterprise competitiveness, explore a kind of valuable source that business growth point, promoting enterprise make the transition to Intranet.So along with the update of system, how to ensure that old system data can continue to use in new system, integration subsystems data are the important demands of one in Ways of Computer-based Information in Modern Enterprise.
In order to the data using old system in new system can be continued, need to carry out data promotion, generally have three kinds of methods.
First method is the DBLink technology utilizing database, the database of old system with new system is connected, directly import in new system by the data of old system database table, such data promotion mode is the fastest in time, and cost of development is also minimum.But, this method have a defect the most fatal to require two systems exactly especially back-end data database table structure must be closely similar even completely the same, otherwise cannot the method be used.For in practical application, seldom can there is back-end data list structure and similar situation thereof in two heterogeneous system generalized cases, and institute is in this way inapplicable in a lot of situation again.
Second method is that old system data is exported to EXCEL or XML format file, then is imported into new system by the import feature of new system.The backstage middle table data of this method inefficiency and a lot of business association cannot import, and can cause that the business datum that imports is follow-up cannot carry out operation flow operation.
The third method is exploitation one sets of data promotion system, resolve the data of old system in a program, and the data layout being organized into new system requirements is saved in new system database.
Carry out data promotion, generally can run into following problem:
First, be integrality and the consistency problem of promotion data.To be transferred and promoted the data of new system, must to ensure it is completely the same and correct with old system data.Business not complete in old system can continue equally in new system.This be data be transferred and promoted one be substantially also most important requirement.But in actual promotion process, often promotion system and new system are not same set of developers, two cover systems often can not well in conjunction with shared, relatively independent, so just causing promotion system often can not well utilize, new system is existing much guarantees data integrity and conforming function, and need again to develop, this not only causes overlapping development, increase cost of development, and also can there is potential risk for the integrality of the data of coming up of being transferred and promoted and consistance.
Secondly, be scalability problem.Deepening continuously and expanding along with business event, need the data of being transferred and promoted can become more and more, the field related to and module also can get more and more, and need the goal systems of being transferred and promoted also can change simultaneously.At this time, former promotion system needs to support that mode of extension is to tackle different business and development requirement flexibly, and reduces cost and the time of regarded as output controlling as much as possible.
Again, be promotion time point and promotion protocol questions.Data promotion is the process of a long-time continuous, does not generally accomplish to accomplish in one move.Client generally wishes the use as far as possible not affecting old system when data are transferred and promoted, this certain difficulty of causing in fact to promotion work.Being select synchronous promotion or asynchronous promotion, how ensureing that completing data promotion work when user does not shut down or the short time of trying one's best shuts down will be all the actual key element needing consideration and balance when specifying promotion scheme.No matter finally select which kind of scheme, in fact all need to select when the support of promotion system, this just requires that promotion system is enough powerful, can support multiple different schemes, to tackle disparity items conditions of demand simultaneously.
Finally, be big data quantity problem.When enterprise needs to carry out inter-system data promotion, general old system has used for a long time, and the historical data of accumulation also can be quite huge, and often reach hundred million grades of levels, database generally can at more than 10T.So old system business data are often first carried out merging and gathered by existing promotion system, reduce actual business datum amount of being transferred and promoted, then be transferred and promoted.Data unavailability to a certain extent after being transferred and promoted can be caused like this.When user needs concrete detailed data, still need to check in primal system, the waste of the resource caused.
Summary of the invention
The object of the invention is to, for the problems referred to above, propose data promotion system and method between a kind of heterogeneous system, to realize guaranteeing data integrity, consistance improve the advantage of set expandability.
For achieving the above object, the technical solution used in the present invention is:
Data promotion system between a kind of heterogeneous system, comprises dynamic paging controller, promotion mode manager, promotion core processor and promotion log manager;
Described dynamic paging controller: adopt dynamic secondary paging to read data;
Described promotion mode manager: select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
Described promotion core processor: the process completing data according to the scheduling configuration file of promotion mode manager generation;
Described promotion log manager: the key log data of each operation of being transferred and promoted of collection.
Preferably, the dynamic secondary paging that described dynamic paging controller adopts is specially:
First will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, read the major key ID of some again in internal memory from temporary table in batches, when secondary paging, from internal memory, directly read the major key ID of buffer memory in batches, utilize the major key ID read actual traffic data to be read in internal memory from legacy data storehouse.
Preferably, read in the major key ID of buffer memory from internal memory, the major key ID quantity of reading produces after analyzing actual traffic data amount dynamic calculation in batches.
Preferably, the promotion mode of promotion mode manager administration, comprising:
Once be transferred and promoted: i.e. promotion system is disposable is all transferred and promoted in new system by had been friends in the past system data;
Repeatedly be transferred and promoted: be about to overall promotion time period fractionation, the promotion time period according to splitting repeatedly carries out data promotion.
Preferably, to be describedly repeatedly transferred and promoted, to comprise repeatedly newly-increased promotion and multi-fold promotion,
Described repeatedly newly-increased promotion: namely within the scope of the fixed time, in the database of new system, all data are all newly-increased from old system;
Described multi-fold is transferred and promoted: namely carry out synchronized update for the history document of being transferred and promoted within the scope of the fixed time, ensures the consistance of new-old system data.
Preferably, described promotion core processor, comprising:
Data query unit: by reading the sql statement of the automatic generated query of data query configuration file needing the business paper of being transferred and promoted, according to the concrete major key ID read in Database Dynamic paging, the business paper read in legacy data storehouse is detailed, and the data of reading are transferred to Date Conversion Unit;
Date Conversion Unit: by the mapping relations table of old system document table and each field of new system document table, convert the data of the data query unit of reception to new system discernible data structure, so that the various application codes making full use of that new system developed;
Data translation unit: the data receiving Date Conversion Unit, translates into the archives major key ID of new system by the data Archives Field ID after conversion;
Data processing unit: fill and the correctness of checking data carrying out backstage field in the data after the process of data translation unit;
Data saving unit: classification is carried out to the data after data processing unit processes and preserves.
Preferably, also comprise: early warning and repair unit: being translated into by archives Field ID before the archives major key ID of new system when data translation unit needs the archives field of document in old system to analyze, when the archives field of record upper in old system is wrong, carry out abnormal data early warning and reparation.
Preferably, in described Date Conversion Unit, the mapping relations table in the mapping relations table of old system document table and each field of new system document table is Microsoft Excel.
Technical solution of the present invention also discloses a kind of data promotion method between heterogeneous system simultaneously, comprising:
Dynamic secondary paging is adopted to read data;
Select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
The promotion process of data is completed according to the scheduling configuration file generated;
Collect the key log data of each operation of being transferred and promoted.
Preferably, described dynamic secondary paging is specially:
First will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, read the major key ID of some again in internal memory from temporary table in batches, when secondary paging, from internal memory, directly read the major key ID of buffer memory in batches, wherein, the quantity of the major key ID of the buffer memory of each reading is according to analysis actual traffic data amount dynamic calculation, utilizes the major key ID read actual traffic data to be read in internal memory from legacy data storehouse.
Technical scheme of the present invention has following beneficial effect:
1: make full use of new systematic difference development function, ensure the integrality of promotion data;
2: support flexible configuration, when promotion demand constantly changes, support the fast Development that new business data are transferred and promoted, reduce cost of development, elevator system extensibility;
3: support different promotion scheme, no matter be synchronously transferred and promoted or asynchronous promotion, ensure that user completes data promotion work when not shutting down as far as possible and not affecting user's use, and ensure data consistency;
4: support that the data of hundred million DBMS big data quantities are transferred and promoted.Under the prerequisite of guaranteed efficiency, new system of original mass data all can being transferred and promoted to, and ensure the stability of promotion process.
Below by drawings and Examples, technical scheme of the present invention is described in further detail.
Accompanying drawing explanation
Fig. 1 is the theory diagram of data promotion system between the heterogeneous system described in the embodiment of the present invention;
Fig. 2 is the Database Dynamic paging process flow diagram described in the embodiment of the present invention;
Fig. 3 is the promotion mode manager workflow diagram described in the embodiment of the present invention;
Fig. 4 is the promotion core processor workflow diagram described in the embodiment of the present invention.
Embodiment
Below in conjunction with accompanying drawing, the preferred embodiments of the present invention are described, should be appreciated that preferred embodiment described herein is only for instruction and explanation of the present invention, is not intended to limit the present invention.
As shown in Figure 1, data promotion system between a kind of heterogeneous system, comprises dynamic paging controller, promotion mode manager, promotion core processor and promotion log manager;
Dynamic paging controller: adopt dynamic secondary paging to read data;
Promotion mode manager: select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
Promotion core processor: the process completing data according to the scheduling configuration file of promotion mode manager generation;
Promotion log manager: the key log data of each operation of being transferred and promoted of collection.
Wherein, the dynamic secondary paging that dynamic paging controller adopts is specially:
First will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, read the major key ID of some again in internal memory from temporary table in batches, when secondary paging, from internal memory, directly read the major key ID of buffer memory in batches, utilize the major key ID read actual traffic data to be read in internal memory from legacy data storehouse.Quantity wherein in some is determined according to setting.
The quantity of the major key ID in batches read from internal memory produces after analyzing actual traffic data amount dynamic calculation.
The promotion mode of promotion mode manager administration, comprising:
Once be transferred and promoted: i.e. promotion system is disposable is all transferred and promoted in new system by had been friends in the past system data;
Repeatedly be transferred and promoted: be about to overall promotion time period fractionation, the promotion time period according to splitting repeatedly carries out data promotion.
Repeatedly be transferred and promoted, comprise repeatedly newly-increased promotion and multi-fold promotion,
Described repeatedly newly-increased promotion: namely within the scope of the fixed time, in the database of new system, all data are all newly-increased from old system;
Described multi-fold is transferred and promoted: namely carry out synchronized update for the history document of being transferred and promoted within the scope of the fixed time, ensures the consistance of new-old system data.
Promotion core processor, comprising:
Data query unit: by reading the sql statement of the automatic generated query of data query configuration file needing the business paper of being transferred and promoted, according to the concrete major key ID read in Database Dynamic paging, the business paper read in legacy data storehouse is detailed, and the data of reading are transferred to Date Conversion Unit;
Date Conversion Unit: by the mapping relations table of old system document table and each field of new system document table, convert the data of the data query unit of reception to new system discernible data structure, so that the various application codes making full use of that new system developed;
Data translation unit: the data receiving Date Conversion Unit, translates into the archives major key ID of new system by the data Archives Field ID after conversion;
Data processing unit: fill and the correctness of checking data carrying out backstage field in the data after the process of data translation unit;
Data saving unit: classification is carried out to the data after data processing unit processes and preserves.
Preferably, also comprise: early warning and repair unit: being translated into by archives Field ID before the archives major key ID of new system when data translation unit needs the archives field of document in old system to analyze, when the archives field of record upper in old system is wrong, carry out abnormal data early warning and reparation.
In Date Conversion Unit, the mapping relations table in the mapping relations table of old system document table and each field of new system document table is Microsoft Excel.
Technical solution of the present invention also discloses a kind of data promotion method between heterogeneous system simultaneously, comprising:
Dynamic secondary paging is adopted to read data;
Select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
The promotion process of data is completed according to the scheduling configuration file generated;
Collect the key log data of each operation of being transferred and promoted.
The invention provides the multi-period batch data promotion system based on Database Dynamic paging technique between a kind of heterogeneous system.Contrast existing data promotion system, the present invention makes full use of memory source mainly for the promotion of support big data quantity, multiple promotion mode is supported flexibly, better utilize new system existing capability enhancing these three aspects of promotion set expandability to improve.In order to realize above-mentioned improvement, this system four core components, comprising: dynamically paging controller, promotion mode manager, promotion core processor and promotion log manager.Four assemblies cooperate mutually, jointly complete the promotion work of a mass data, and provide various configurations and extended mode to tackle different promotion demands, reduce promotion cost.
The technical program specific works is as follows:
The first, the support that magnanimity business datum is transferred and promoted is realized by Database Dynamic paging controller.Due to the otherness of data structure between heterogeneous system, in order to batch resolves promotion raw data, need first by digital independent in middleware internal memory, carry out program traffic logical process, then preserve into new system.When data volume is very large, it is limited that middleware can carry maximum memory, all can not reading in internal memory, adopting the technology of database paging to come data to be read in internal memory like this with regard to needing in batches by disposable for all data needing to be transferred and promoted.The data volume (business paper number) of existing Database Pagination Technology every page is constant, when business paper distributed data amount and uneven time (as: the table body line number size variation of document sublist is very large), overflow to prevent internal memory, steady state value is needed to remove the limit, so just can not utilize memory headroom more fully, increase a point number of pages, and then affect efficiency.For the problems referred to above, the present invention devises a kind of dynamically secondary paging technique, as shown in Figure 2.
Because the business datum of being transferred and promoted is all generally document, so first will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, the major key ID in batches reading some (can adjust according to configuration) again from temporary table, in internal memory, uses when reading data detail again as secondary paging in batches.When secondary paging, directly can read a part of major key ID from internal memory in batches, utilize this major key ID to read in internal memory by actual traffic data from legacy data storehouse, for follow-up promotion operation provides data encasement.Why carry out secondary paging, because when data volume is very large, often very repeatedly from database, read document detail, so just need acquisition major key ID very repeatedly, major key ID is done one deck memory cache, can effectively reduce the number of times inquiring about major key ID from database, and then promote paging query efficiency.
Dynamically refer to and to produce after analyzing actual traffic data amount dynamic calculation in each major key ID number obtained from internal memory, object is also to utilize internal memory as far as possible fully, reduces paging query number of times, promotes paging query efficiency.Because document is all generally boss's list structure, same document is not quite similar, so the actual amount of data (i.e. gauge outfit+table body line number summation) representated by a gauge outfit major key ID can have a long way to go due to the table body line number of often opening document.If each document quantity (i.e. gauge outfit major key quantity) read is a definite value, this just must arrange a lower value with solution when the indivedual document table body of appearance capable very large time the internal memory overflow problem that causes.And when the distribution of actual document table body line number amount is extremely unequal, internal memory waste significantly will be caused, invisible the number of times that improve paging query, greatly reduce search efficiency.For this situation, the present invention devises according to presetting memory value dynamic calculation document quantity, at every turn dynamically from secondary Paged Memory, reads a certain amount of major key ID, carries out paging query.Make use of internal memory as far as possible fully, decrease paging query number of times, improve Paging system search efficiency.
The second, by promotion mode manager, the promotion system making this invent can support multiple promotion mode, to tackle multiple different actual promotion demand.Existing traditional promotion system does not often consider the problem of multiple promotion scheme, like this often can not supporting flexibly for during difference promotion scheme.Such as user requires that promotion process can not be shut down, when amount of user data is very large, in not shutdown situation, general need multiple promotion scheme with the use of: to be repeatedly in batches newly-increasedly transferred and promoted, the secondary renewal etc. of promotion data.The promotion mode manager of the present invention's design is exactly to solve the problem.This manager can intelligence judgement or manual select each promotion to need to use any promotion mode, and select synchronous or asynchronous promotions execution according to configuring condition, and then the difference promotion core processing that schedule subsequent designs for different modes.
Support of the present invention is once transferred and promoted and two large class promotion modes of being repeatedly transferred and promoted.
Once be transferred and promoted, as the term suggests promotion system is disposable is all transferred and promoted had been friends in the past system data in new system.This promotion mode is often less for old system data amount, and the shut down promotion that client allows is had plenty of time.All data, often by old system-down, to be disposablely transferred and promoted in new system by promotion scheme.
Repeatedly be transferred and promoted, when the data volume of being transferred and promoted is very huge, when the stop time of customer requirement is shorter, is once transferred and promoted due to overlong time, often can not satisfies the demands.This just needs the promotion time period of entirety to split, and plans time range of being at every turn transferred and promoted.Technically, chronic owing to recovering a database, so may need to support that the online backup database of production environment is transferred and promoted as data source.Because be online promotion, so practical business document is in constantly change, for ensureing the consistance of data, being repeatedly transferred and promoted and being divided into repeatedly newly-increased promotion and multi-fold to be transferred and promoted.
Repeatedly newly-increased promotion, specifies a time range, and all documents are all less than in the database of new system, and at this moment all data are all finally newly-increased preservations, and system can record successful daily record and error log.
Multi-fold is transferred and promoted, i.e. the upgrading of history promotion data.Carry out synchronized update within the scope of the fixed time for the history document of being transferred and promoted, ensure the consistance of new-old system data.System can synchronously be recorded as merit daily record and error log, and these daily records also involve newly-increased, deletes, amendment etc.Specifically as shown in Figure 3.
3rd, by promotion core processor, achieve the process of new system of old system business data being transferred and promoted to.In order to strengthen the extensibility of this promotion system further, promotion procedural abstraction has been become several function by the promotion core processor in the present invention, when actual development and expansion, research staff only needs to develop in conjunction with the existing service logic of new system for concrete steps simultaneously.By translation function wherein, old system organization can be become new system should have the discernible data structure of program, and then it is existing much about the API that data check, data stuffing, data are preserved during new system just can be utilized to apply, so just take full advantage of new systematic difference exploitation code, ensure the integrality of promotion data.Meanwhile, each committed step of core processing is all loaded and scheduling by configuration file, and this just makes whole promotion process height easily extensible, according to actual promotion demand, flexible configuration, supports the fast Development that new business data are transferred and promoted, reduce cost of development, improve the system expandability.
Just be described in detail for several major functions of core processor below, as shown in Figure 4:
1, data query.Promotion system is by reading the sql statement of the automatic generated query of data query configuration file needing the business paper of being transferred and promoted, according to the concrete major key ID read in Database Dynamic paging, the business paper read in legacy data storehouse is detailed, data is passed to next step and carries out data conversion.
2, data conversion.By the mapping relations table of old system document table and each field of new system document table, promotion system converts the data read to new system discernible data structure, follow-up like this all operations is exactly the operation for new system data, and the API such as the existing various data stuffing of new system, verification, preservation then can directly use.In fact new-old system literary name section mapping relations table is exactly EXCEL table, and the maintenance of this table is got up very convenient, develops, implementing even client can flexible configuration according to the actual requirements.
3, data translation.The major key ID of a large amount of archives is have recorded in business datum table, major key may be inconsistent between heterogeneous system for these archives, cause this phenomenon may be often due to the archives of new system be again safeguard or archives when old system is transferred and promoted new system, do not consider to preserve former major key or list structure change greatly and cannot preserve former major key.At this moment, after converting new system data structure to, need the archives Field ID inside by document to translate into the archives major key ID of new system.Promotion system of the present invention can resolve the archives field in each field of new system document automatically by a metadata parser, call different archives ID translaters translate according to different archives fields.If the object system of being transferred and promoted changes, only need change metadata parser, can Quick Extended be realized.
Meanwhile, when analyzing the archives field of document, due to the uncertainty of old system, the archives field sometimes old document recorded is wrong, as these archives are deleted etc.When finding this problem in promotion system data translation process, abnormal data early warning can be carried out and repair step.By realizing the early warning of promotion system development and repairing interface, dealing of abnormal data can be carried out according to the actual requirements very flexibly.
4, data processing.Data processing is generally divided into data stuffing and data check, when a new document is before preservation, often need to fill some backstage fields and the data correctness verifying document, owing to converting the data structure that new system real-life program uses to, so some data stuffings of new system application and the API of verification directly can be loaded, the code of fast Development data processing link, can also ensure the correctness of data simultaneously.
5, data are preserved.Because promotion system of the present invention support is repeatedly transferred and promoted, thus data preserve may relate to newly-increased, amendment and delete.Promotion system has analyzed data to be processed by promotion mode manager and promotion daily record in the past specifically should carry out for which kind of preservation operation, so, analysis result before only needing basis when carrying out data and preserving, classification is called new system API and is carried out preserving.
4th, promotion log manager.Daily record can all kinds of operation of help system process time problem, help system analyzes current operating conditions, and promotion system is no exception.The key log data of each operation of being transferred and promoted of being collected by promotion log manager, it is newly-increased promotion or covering promotion that promotion system can analyze current operation very easily, and can log collection abnormal data be passed through, and then for repairing abnormal data, locator data or procedural problem provide foundation.
In sum, technical solution of the present invention has following effect:
1. be transferred and promoted for the data of big data quantity, make full use of memory headroom, technical solution of the present invention adopts Database Dynamic secondary paging technique.The data volume of inquiring about in each data page of existing Database Pagination Technology is certain, after changing dynamic paging into, can more effectively utilize default internal memory, can effectively reduce paging number of times according to real data situation, improves paging query efficiency.
2. in order to support various promotion scene demand flexibly, multiple promotion mode can flexible combination be used, technical solution of the present invention adopts promotion mode manager, like this when tackling promotion scene complicated and changeable, just can by multiple promotion mode flexible combination, enhance the business support scope of promotion system, decrease the secondary development of promotion system.
3., in order in conjunction with existing new system, cost of development can better be reduced, accelerate payment process.Promotion core process is abstracted into several function by the present invention.Wherein, data conversion process, converts the discernible data structure of new system to by old system data, so just can use the API of a lot of new system, makes new system and promotion system set up contact, and utilizes a lot of application API of new system to ensure data correctness.Contrast existing promotion system and new system is often relatively isolated, this just cause real implement certain promotion project time, need to develop in a large number former promotion system for new system, cause the waste of cost of development.
4. all promotions all carry out scheduled for executing by configuration file, and have opened multiple interface, to adapt to different promotion demand, carry out the secondary development of rapid data promotion.
Last it is noted that the foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, although with reference to previous embodiment to invention has been detailed description, for a person skilled in the art, it still can be modified to the technical scheme described in foregoing embodiments, or carries out equivalent replacement to wherein portion of techniques feature.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a data promotion system between heterogeneous system, is characterized in that, comprises dynamic paging controller, promotion mode manager, promotion core processor and promotion log manager;
Described dynamic paging controller: adopt dynamic secondary paging to read data;
Described promotion mode manager: select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
Described promotion core processor: the process completing data according to the scheduling configuration file of promotion mode manager generation;
Described promotion log manager: the key log data of each operation of being transferred and promoted of collection.
2. data promotion system between heterogeneous system according to claim 1, is characterized in that, the dynamic secondary paging that described dynamic paging controller adopts is specially:
First will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, read the major key ID of some again in internal memory from temporary table in batches, when secondary paging, from internal memory, directly read the major key ID of buffer memory in batches, utilize the major key ID read actual traffic data to be read in internal memory from legacy data storehouse.
3. data promotion system between heterogeneous system according to claim 2, is characterized in that, above-mentionedly reads in the major key ID of buffer memory from internal memory in batches, and the major key ID quantity of reading produces after analyzing actual traffic data amount dynamic calculation.
4. data promotion system between heterogeneous system according to claim 3, is characterized in that, the promotion mode of promotion mode manager administration, comprising:
Once be transferred and promoted: i.e. promotion system is disposable is all transferred and promoted in new system by had been friends in the past system data;
Repeatedly be transferred and promoted: be about to overall promotion time period fractionation, the promotion time period according to splitting repeatedly carries out data promotion.
5. data promotion system between heterogeneous system according to claim 4, is characterized in that, is describedly repeatedly transferred and promoted, and comprises repeatedly newly-increased promotion and multi-fold promotion,
Described repeatedly newly-increased promotion: namely within the scope of the fixed time, in the database of new system, all data are all newly-increased from old system;
Described multi-fold is transferred and promoted: namely carry out synchronized update for the history document of being transferred and promoted within the scope of the fixed time, ensures the consistance of new-old system data.
6. data promotion system between heterogeneous system according to claim 3, is characterized in that, described promotion core processor, comprising:
Data query unit: by reading the sql statement of the automatic generated query of data query configuration file needing the business paper of being transferred and promoted, according to the concrete major key ID read in Database Dynamic paging, the business paper read in legacy data storehouse is detailed, and the data of reading are transferred to Date Conversion Unit;
Date Conversion Unit: by the mapping relations table of old system document table and each field of new system document table, convert the data of the data query unit of reception to new system discernible data structure, so that the various application codes making full use of that new system developed;
Data translation unit: the data receiving Date Conversion Unit, translates into the archives major key ID of new system by the data Archives Field ID after conversion;
Data processing unit: fill and the correctness of checking data carrying out backstage field in the data after the process of data translation unit;
Data saving unit: classification is carried out to the data after data processing unit processes and preserves.
7. data promotion system between heterogeneous system according to claim 6, it is characterized in that, also comprise: early warning and repair unit: being translated into by archives Field ID before the archives major key ID of new system when data translation unit needs the archives field of document in old system to analyze, when the archives field of record upper in old system is wrong, carry out abnormal data early warning and reparation.
8. data promotion system between heterogeneous system according to claim 6, is characterized in that, in described Date Conversion Unit, the mapping relations table in the mapping relations table of old system document table and each field of new system document table is Microsoft Excel.
9. a data promotion method between heterogeneous system, is characterized in that, comprising:
Dynamic secondary paging is adopted to read data;
Select each promotion mode needing to use of being transferred and promoted, and select synchronous or asynchronous promotion according to the configuring condition of new-old system, thus generate scheduling configuration file;
The promotion process of data is completed according to the scheduling configuration file generated;
Collect the key log data of each operation of being transferred and promoted.
10. data promotion method between heterogeneous system according to claim 9, is characterized in that, described dynamic secondary paging is specially:
First will all gauge outfit major key ID of the business paper of being transferred and promoted be needed to be stored in a temporary table when a paging, read the major key ID of some again in internal memory from temporary table in batches, when secondary paging, from internal memory, directly read the major key ID of buffer memory in batches, wherein, the quantity of the major key ID of the buffer memory of each reading is according to analysis actual traffic data amount dynamic calculation, utilizes the major key ID read actual traffic data to be read in internal memory from legacy data storehouse.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510735226.0A CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510735226.0A CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105426419A true CN105426419A (en) | 2016-03-23 |
| CN105426419B CN105426419B (en) | 2018-11-20 |
Family
ID=55504631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510735226.0A Active CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105426419B (en) |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106407321A (en) * | 2016-08-31 | 2017-02-15 | 东软集团股份有限公司 | Data synchronization method and device |
| CN108280142A (en) * | 2017-12-29 | 2018-07-13 | 武汉钢铁工程技术集团通信有限责任公司 | A kind of method of new-old system transaction data docking |
| CN108595487A (en) * | 2018-03-14 | 2018-09-28 | 摇了购（武汉）电子商务有限公司 | The method and system of data are accessed under a kind of big data high concurrent |
| CN109684528A (en) * | 2018-12-04 | 2019-04-26 | 深圳店匠科技有限公司 | Inter-library paging query method, system and storage medium |
| CN109783779A (en) * | 2018-12-27 | 2019-05-21 | 北京航天福道高技术股份有限公司 | A kind of data EXCEL imports and exports the tool of conversion |
| CN112069190A (en) * | 2019-06-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Batch data acquisition method, device, equipment and medium |
| CN113010607A (en) * | 2021-04-06 | 2021-06-22 | 工银科技有限公司 | Method, device, computer system and storage medium for data synchronization between systems |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183371A (en) * | 2007-12-12 | 2008-05-21 | 中兴通讯股份有限公司 | Method for quick finishing large data-handling and reporting system |
| CN101261693A (en) * | 2007-03-09 | 2008-09-10 | 上海宝信软件股份有限公司 | A data archiving method in computer application system |
| CN102236672A (en) * | 2010-05-06 | 2011-11-09 | 深圳市腾讯计算机系统有限公司 | Method and device for importing data |
| CN103489061A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Batch data processing device and batch data processing method |
| US8868506B1 (en) * | 2010-06-17 | 2014-10-21 | Evolphin Software, Inc. | Method and apparatus for digital asset management |
-
2015
- 2015-11-03 CN CN201510735226.0A patent/CN105426419B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101261693A (en) * | 2007-03-09 | 2008-09-10 | 上海宝信软件股份有限公司 | A data archiving method in computer application system |
| CN101183371A (en) * | 2007-12-12 | 2008-05-21 | 中兴通讯股份有限公司 | Method for quick finishing large data-handling and reporting system |
| CN102236672A (en) * | 2010-05-06 | 2011-11-09 | 深圳市腾讯计算机系统有限公司 | Method and device for importing data |
| US8868506B1 (en) * | 2010-06-17 | 2014-10-21 | Evolphin Software, Inc. | Method and apparatus for digital asset management |
| CN103489061A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Batch data processing device and batch data processing method |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106407321A (en) * | 2016-08-31 | 2017-02-15 | 东软集团股份有限公司 | Data synchronization method and device |
| CN108280142A (en) * | 2017-12-29 | 2018-07-13 | 武汉钢铁工程技术集团通信有限责任公司 | A kind of method of new-old system transaction data docking |
| CN108280142B (en) * | 2017-12-29 | 2022-01-04 | 武汉钢铁工程技术集团通信有限责任公司 | Method for butt-jointing transaction data of new and old systems |
| CN108595487A (en) * | 2018-03-14 | 2018-09-28 | 摇了购（武汉）电子商务有限公司 | The method and system of data are accessed under a kind of big data high concurrent |
| CN109684528A (en) * | 2018-12-04 | 2019-04-26 | 深圳店匠科技有限公司 | Inter-library paging query method, system and storage medium |
| CN109783779A (en) * | 2018-12-27 | 2019-05-21 | 北京航天福道高技术股份有限公司 | A kind of data EXCEL imports and exports the tool of conversion |
| CN109783779B (en) * | 2018-12-27 | 2023-12-22 | 北京航天福道高技术股份有限公司 | Tool for importing, exporting and converting data EXCEL |
| CN112069190A (en) * | 2019-06-11 | 2020-12-11 | 腾讯科技（深圳）有限公司 | Batch data acquisition method, device, equipment and medium |
| CN112069190B (en) * | 2019-06-11 | 2023-06-09 | 腾讯科技（深圳）有限公司 | Batch data acquisition method, device, equipment and medium |
| CN113010607A (en) * | 2021-04-06 | 2021-06-22 | 工银科技有限公司 | Method, device, computer system and storage medium for data synchronization between systems |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105426419B (en) | 2018-11-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105426419A (en) | System and method for data promotion among heterogeneous systems | |
| US8832020B2 (en) | Method and system for safely transporting legacy data to an object semantic form data grid | |
| CN101446971B (en) | Method for building content management system and device thereof | |
| CN104461671A (en) | Method and system for periodically managing code modification report | |
| CN103218402B (en) | General database data structure and data mover system and method thereof | |
| CN103514223A (en) | Data synchronism method and system of database | |
| US8607217B2 (en) | Incremental upgrade of entity-relationship systems | |
| CN102591864B (en) | Data updating method and device in comparison system | |
| CN105159831A (en) | Method for automatically monitoring quality of software code | |
| CN102521218B (en) | File combining method and file combining device | |
| CN113408890A (en) | Artificial intelligence-based method and system for generating evaluation report after industrial investment project | |
| CN109445771A (en) | A kind of off line network layers block code Auto-Generation Tool of CS framework and method | |
| CN102156744A (en) | Method for eliminating repetition of memory dialog list | |
| CN115712623A (en) | Batch data fault-tolerant acquisition method based on capture metadata change | |
| CN102254046A (en) | Webpage data acquiring method and system | |
| CN104573002A (en) | Data organization model for filing based on human, event and object | |
| CN111625596B (en) | Multi-source data synchronous sharing method and system for real-time new energy consumption scheduling | |
| CN103559574A (en) | Method and system for operating workflow | |
| CN105260411A (en) | XBRL-based system and method for achieving detail data aggregation and drill-down of financial report | |
| CN103809915A (en) | Read-write method and device of magnetic disk files | |
| CN109615241A (en) | A kind of software Bug allocating method based on convolution sum Recognition with Recurrent Neural Network | |
| CN109828953A (en) | Picture retrieval system and its method based on distributed memory database | |
| CN201974852U (en) | Financial voucher generating system | |
| CN105630997A (en) | Data parallel processing method, device and equipment | |
| CN115205030A (en) | Wind-controlled user portrait system based on configurable big data analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |