CN105426419B - Data promotion system and method between heterogeneous system - Google Patents
Data promotion system and method between heterogeneous system Download PDFInfo
- Publication number
- CN105426419B CN105426419B CN201510735226.0A CN201510735226A CN105426419B CN 105426419 B CN105426419 B CN 105426419B CN 201510735226 A CN201510735226 A CN 201510735226A CN 105426419 B CN105426419 B CN 105426419B
- Authority
- CN
- China
- Prior art keywords
- data
- promotion
- transferred
- promoted
- major key
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses data promotion system and method between a kind of heterogeneous system, wherein data promotion system includes dynamic page controller, promotion mode manager, promotion core processor and promotion log manager between heterogeneous system；The dynamic page controller：Data are read out using dynamic secondary paging；The promotion mode manager：Selection is transferred and promoted every time needs promotion mode to be used, and selects synchronous or asynchronous promotion according to the configuring condition of new-old system, to generate scheduling configuration file；The promotion core processor：The scheduling configuration file generated according to promotion mode manager completes the processing of data；The promotion log manager：Collect the key log data for operation of being transferred and promoted every time.In system four components cooperate realize guarantee data integrity, consistency and improve set expandability the advantages of.
Description
Technical field
The present invention relates to computer fields, and in particular, to data promotion system and method between a kind of heterogeneous system.
Background technique
Currently, with the continuous development of information technology, network technology, mechanics of communication, database technology.It is competing between enterprise
Strive is not only to compete in product quality and service, it is often more important that by information technology, network technology, mechanics of communication sum number
The core competitiveness of enterprise, fast responding market demand are improved in conjunction with management philosophy according to library technology.Each mechanism of promotion enterprise,
It collaboration capabilities between each department and each employee and links up whenever and wherever possible.
It, may be successively using to not due to the expansion of business administration and business scope during IT application in enterprises
Same information system.Under current big data environmental background, data, which have become, improves enterprise competitiveness, the growth of exploration business
Point, a kind of valuable source for promoting enterprise to make the transition to Intranet.So how to guarantee old with the update of system
System data can continue to use in new system, and integrating subsystems data is one of Ways of Computer-based Information in Modern Enterprise
Important demand.
In order to continue to use the data of old system in new system, need to carry out data promotion, general there are three types of sides
Method.
First method is the DBLink technology using database, and the database of old system and new system is attached,
The data of old system database table are introduced directly into new system, such data promotion mode is most fast, exploitation in time
Cost is also minimum.But it is exactly to require two systems especially back-end data that this method, which has a most fatal defect,
Database table structure must be closely similar or even completely the same, is not otherwise available this method.For in practical application, two isomery systems
Back-end data table structure and its similar situation can seldom occur in system ordinary circumstance, and institute is uncomfortable in many cases again in this way
With.
Second method is the importing function that old system data is exported to EXCEL or XML format file, then passes through new system
It can import into new system.The backstage middle table data of this method inefficiency and many business associations can not import, and can lead
Cause the business datum imported is subsequent can not carry out operation flow operation.
The third method is one sets of data promotion system of exploitation, parses the data of old system in a program, and is organized into new
The data format of system requirements is saved in new system database.
Data promotion is carried out, following problems can be generally encountered：
Firstly, being the integrality and consistency problem of promotion data.It is transferred and promoted to the data of new system, it is necessary to guarantee and old
System data is completely the same and correct.Not complete business can equally continue to complete in new system in old system.
It is also most important requirement that this is one of data promotion substantially.But during being actually transferred and promoted, often promotion system and newly system
System is not same set of developer, and two systems are tended not to combine well and be shared, relatively independently, result in rising in this way
The system of moving tends not to need again using the existing function of much guaranteeing data integrity with consistency of new system well
Exploitation, this not only causes overlapping development, increases development cost, and can integrality for the data that promotion comes up with it is consistent
There is also potential risks for property.
Secondly, being scalability problem.Deepening continuously and expand with business event, the data for needing to be transferred and promoted can become
More and more, the field being related to and module also can be more and more, while the goal systems for needing to be transferred and promoted may also have change
Change.At this time, former promotion system needs support flexible mode of extension to cope with different business and exploitation demand, and as far as possible
The cost that expands and develops of reduction and the time.
It again, is promotion time point and promotion protocol questions.Data promotion is the continuous process of long-time, is generally done not
To accomplish in one move.For client it is generally desirable to not influence the use of old system as far as possible when data are transferred and promoted, this actually gives promotion work
Caused by certain difficulty.It is that selection is synchronous to be transferred and promoted or asynchronous promotion how to guarantee not shut down in user or short time as far as possible
It all will be the practical element for needing to consider and weigh in specified promotion scheme that data promotion work is completed in the case where shutdown.Nothing
Which kind of, by scheme is finally selected, actually require to select in the case where promotion system is supported, this requires promotion systems to want
It is powerful enough, while can support a variety of different schemes, to cope with disparity items conditions of demand.
Finally, being big data quantity problem.When enterprise needs to carry out inter-system data promotion, general old system has been used
For a long time, the historical data of accumulation also can be quite huge, often reaches hundred million grades of levels, and database generally can be in 10T or more.
Summarize so existing promotion system often first merges old system business data, reduces the business datum being actually transferred and promoted
Amount, then be transferred and promoted.It will cause the unavailability of data to a certain extent after being transferred and promoted in this way.When user needs specific detail number
According to when, it is still necessary to go in primal system to check, caused by resource waste.
Summary of the invention
It is an object of the present invention in view of the above-mentioned problems, propose data promotion system and method between a kind of heterogeneous system, with
Realization guarantees data integrity, consistency and the advantages of improve set expandability.
To achieve the above object, the technical solution adopted by the present invention is that：
Data promotion system between a kind of heterogeneous system, including dynamic page controller, promotion mode manager, promotion core
Processor and promotion log manager；
The dynamic page controller：Data are read out using dynamic secondary paging；
The promotion mode manager：Selection is transferred and promoted every time needs promotion mode to be used, and matching according to new-old system
Situation selection synchronization or asynchronous promotion are set, to generate scheduling configuration file；
The promotion core processor：The scheduling configuration file generated according to promotion mode manager completes the place of data
Reason；
The promotion log manager：The key log data for each operation of being transferred and promoted collected.
Preferably, the secondary paging of dynamic that the dynamic page controller uses is specially：
First all gauge outfit major key ID for the business paper being transferred and promoted will be needed to store into an interim table in a paging,
A certain number of major key ID are read in batches from interim table again into memory, in secondary paging, are directly read in batches from memory
The major key ID cached is taken, is read into memory actual traffic data from old database using the major key ID of reading.
Preferably, from being read in the major key ID cached in memory in batches, the major key ID quantity of reading is real by analysis
What border business datum amount dynamic generated after calculating.
Preferably, the promotion mode of promotion mode manager administration, including：
It is primary to be transferred and promoted：I.e. had been friends in the past system data is disposably all transferred and promoted into new system by promotion system；
Repeatedly it is transferred and promoted：The promotion period that will be whole splits, and repeatedly carries out data liter according to the promotion period of fractionation
It moves.
Preferably, the multiple promotion, including repeatedly increase newly to be transferred and promoted and be transferred and promoted with multi-fold,
The repeatedly newly-increased promotion：I.e. within the scope of specified time, all data are all from old system in the database of new system
It is increased newly in system；
The multi-fold is transferred and promoted：Update is synchronized for the history document being transferred and promoted i.e. within the scope of specified time,
Guarantee the consistency of new-old system data.
Preferably, the promotion core processor, including：
Data query unit：The data query configuration file for the business paper being transferred and promoted is needed to automatically generate inquiry by reading
Sql sentence the business paper detail in old database is read according to the specific major key ID read in Database Dynamic paging,
And the data of reading are transferred to Date Conversion Unit；
Date Conversion Unit：By the mapping table of old system document table and each field of new system document table, will receive
Data query unit data conversion at the identifiable data structure of new system, to make full use of new system developed each
Kind application code；
Data translation unit：The data for receiving Date Conversion Unit, archives Field ID in the data after conversion is translated into
The archives major key ID of new system；
Data processing unit：To carrying out field filling in backstage in data translation unit treated data and verify data
Correctness；
Data saving unit：Classification preservation is carried out to the data after data processing unit processes.
Preferably, further include：Early warning and reparation unit：When archives Field ID is translated into new system by data translation unit
The archives field to document in old system is needed to analyze before archives major key ID, when the archives field recorded upper in old system has
It is accidentally to carry out abnormal data early warning and reparation.
Preferably, in the Date Conversion Unit, the mapping relations of old system document table and each field of new system document table
Mapping table in table is Microsoft Excel.
A kind of data promotion method between heterogeneous system is also disclosed in technical solution of the present invention simultaneously, including：
Data are read out using dynamic secondary paging；
Selection is transferred and promoted every time needs promotion mode to be used, and synchronous or different according to the selection of the configuring condition of new-old system
Step is transferred and promoted, to generate scheduling configuration file；
The promotion processing of data is completed according to the scheduling configuration file of generation；
Collect the key log data for operation of being transferred and promoted every time.
Preferably, the secondary paging of the dynamic is specially：
First all gauge outfit major key ID for the business paper being transferred and promoted will be needed to store into an interim table in a paging,
A certain number of major key ID are read in batches from interim table again into memory, in secondary paging, are directly read in batches from memory
Take the major key ID cached, wherein the quantity of the major key ID cached read every time is according to analysis actual traffic data amount
Dynamic calculates, and is read into memory actual traffic data from old database using the major key ID of reading.
Technical solution of the present invention has the advantages that：
1：The application and development function of new system is made full use of, guarantees the integrality of promotion data；
2：Support flexible configuration, when promotion demand constantly changes, the quick exploitation for supporting new business data to be transferred and promoted is reduced
Development cost, lifting system scalability；
3：It supports different promotion schemes, no matter synchronizes promotion or asynchronous promotion, guarantee that user is not shutting down not as far as possible
It influences to complete data promotion work in the case where user's use, and guarantees data consistency；
4：The data of hundred million grades of data big data quantities are supported to be transferred and promoted.It, can be by original magnanimity number under the premise of guaranteed efficiency
New system is arrived according to being all transferred and promoted, and guarantees the stability of promotion process.
Below by drawings and examples, technical scheme of the present invention will be described in further detail.
Detailed description of the invention
Fig. 1 is the functional block diagram of data promotion system between heterogeneous system described in the embodiment of the present invention；
Fig. 2 is Database Dynamic paging flow chart described in the embodiment of the present invention；
Fig. 3 is promotion mode manager work flow diagram described in the embodiment of the present invention；
Fig. 4 is promotion core processor work flow diagram described in the embodiment of the present invention.
Specific embodiment
Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings, it should be understood that preferred reality described herein
Apply example only for the purpose of illustrating and explaining the present invention and is not intended to limit the present invention.
As shown in Figure 1, data promotion system between a kind of heterogeneous system, including dynamic page controller, promotion mode manage
Device, promotion core processor and promotion log manager；
Dynamic page controller：Data are read out using dynamic secondary paging；
Promotion mode manager：Selection is transferred and promoted every time needs promotion mode to be used, and according to the configuration feelings of new-old system
Condition selection synchronization or asynchronous promotion, to generate scheduling configuration file；
Promotion core processor：The scheduling configuration file generated according to promotion mode manager completes the processing of data；
Promotion log manager：The key log data for each operation of being transferred and promoted collected.
Wherein, dynamic page controller use the secondary paging of dynamic be specially：
First all gauge outfit major key ID for the business paper being transferred and promoted will be needed to store into an interim table in a paging,
A certain number of major key ID are read in batches from interim table again into memory, in secondary paging, are directly read in batches from memory
The major key ID cached is taken, is read into memory actual traffic data from old database using the major key ID of reading.Wherein
Quantity in certain amount is determined according to setting.
Quantity from the major key ID read in memory in batches is generated after being calculated by analysis actual traffic data amount dynamic
's.
The promotion mode of promotion mode manager administration, including：
It is primary to be transferred and promoted：I.e. had been friends in the past system data is disposably all transferred and promoted into new system by promotion system；
Repeatedly it is transferred and promoted：The promotion period that will be whole splits, and repeatedly carries out data liter according to the promotion period of fractionation
It moves.
It is repeatedly transferred and promoted, including repeatedly increases newly to be transferred and promoted and be transferred and promoted with multi-fold,
The repeatedly newly-increased promotion：I.e. within the scope of specified time, all data are all from old system in the database of new system
It is increased newly in system；
The multi-fold is transferred and promoted：Update is synchronized for the history document being transferred and promoted i.e. within the scope of specified time,
Guarantee the consistency of new-old system data.
Promotion core processor, including：
Data query unit：The data query configuration file for the business paper being transferred and promoted is needed to automatically generate inquiry by reading
Sql sentence the business paper detail in old database is read according to the specific major key ID read in Database Dynamic paging,
And the data of reading are transferred to Date Conversion Unit；
Date Conversion Unit：By the mapping table of old system document table and each field of new system document table, will receive
Data query unit data conversion at the identifiable data structure of new system, to make full use of new system developed each
Kind application code；
Data translation unit：The data for receiving Date Conversion Unit, archives Field ID in the data after conversion is translated into
The archives major key ID of new system；
Data processing unit：To carrying out field filling in backstage in data translation unit treated data and verify data
Correctness；
Data saving unit：Classification preservation is carried out to the data after data processing unit processes.
Preferably, further include：Early warning and reparation unit：When archives Field ID is translated into new system by data translation unit
The archives field to document in old system is needed to analyze before archives major key ID, when the archives field recorded upper in old system has
It is accidentally to carry out abnormal data early warning and reparation.
In Date Conversion Unit, the mapping in the mapping table of old system document table and each field of new system document table is closed
Be table be Microsoft Excel.
A kind of data promotion method between heterogeneous system is also disclosed in technical solution of the present invention simultaneously, including：
Data are read out using dynamic secondary paging；
Selection is transferred and promoted every time needs promotion mode to be used, and synchronous or different according to the selection of the configuring condition of new-old system
Step is transferred and promoted, to generate scheduling configuration file；
The promotion processing of data is completed according to the scheduling configuration file of generation；
Collect the key log data for operation of being transferred and promoted every time.
The present invention provides the multi-period batch data promotions between a kind of heterogeneous system based on Database Dynamic paging technique
System.Existing data promotion system is compared, present invention is generally directed to support big data quantity promotion to make full use of memory source, more
Kind promotion mode is flexibly supported, preferably using new system existing capability enhancing promotion set expandability, these three aspects are carried out
It improves.In order to realize above-mentioned improvement, which devises four core components, including：Dynamic page controller, promotion side
Formula manager, promotion core processor and promotion log manager.Four components cooperate, and complete a mass data jointly
Promotion work, and provide various configurations and extended mode to cope with different promotion demands, reduce promotion cost.
The technical program specific works are as follows：
First, the support being transferred and promoted to magnanimity business datum is realized by Database Dynamic paging controller.Due to isomery
The otherness of inter-system data structure needs first to parse promotion initial data in batches by reading data to middleware memory
In, program traffic logical process is carried out, new system is then saved into.When data volume is very big, middleware can carry maximum memory
Be limited, it is impossible to by the data of promotion in need disposably all read in memory, thus need using database
The technology of paging in batches reads data in memory.The data volume of existing every page of Database Pagination Technology（Business paper item
Number）Be it is constant, when the distributed data amount of business paper and its it is uneven when（Such as：The table body line number size variation of document sublist
It is very big）, memory overflows in order to prevent, needs to remove the limit steady state value, thus cannot more fully utilize memory headroom, increases
Paging number, and then influence efficiency.In view of the above-mentioned problems, the present invention devises a kind of secondary paging technique of dynamic, such as Fig. 2 institute
Show.
Since the business datum of promotion is typically all document, so the business paper that first needs are transferred and promoted in a paging
All gauge outfit major key ID storage read certain amount in batches into an interim table, then from interim table（It can be adjusted according to configuration
It is whole）Major key ID into memory, as secondary paging again in batches read data detail when use.In secondary paging, Ke Yizhi
It connects and reads a part of major key ID in batches from memory, read actual traffic data from old database using this major key ID interior
In depositing, data preparation is provided for subsequent promotion operation.Why carry out secondary paging, be because when data volume is very big, it is past
Toward very multiple document detail can be read from database, very multiple acquisition major key ID is thus needed, by major key ID
One layer of memory cache is done, the number for inquiring major key ID from database can be effectively reduced, and then promote paging query efficiency.
Dynamic, which refers in ID number of the major key obtained from memory every time, to be counted by analysis actual traffic data amount dynamic
It is generated after calculation, purpose reduces paging query number also for memory is adequately utilized as far as possible, promotes paging query effect
Rate.Since document is typically all boss's table structure, same document is since the table body line number of every document is not quite similar, so one
Actual amount of data representated by a gauge outfit major key ID（That is gauge outfit+table body line number summation）It can have a long way to go.If read every time
Document quantity（That is gauge outfit major key quantity）As soon as being a definite value, it is individual single when occurring to solve that this must be provided with a lower value
Memory overflow problem caused by when very big according to table body row.And when practical document table body line number amount is distributed extremely unequal, it will
Significantly memory waste is caused, invisible improves the number of paging query, greatly reduces search efficiency.For this
Situation, the present invention are devised according to memory value dynamic calculating document quantity is preset, are dynamically read from secondary Paged Memory every time
A certain amount of major key ID is taken, paging query is carried out.Memory is adequately utilized as far as possible, reduces paging query number, improves
Paging system search efficiency.
Second, by promotion mode manager, the promotion system for inventing this supports a variety of promotion modes, to answer
To a variety of different practical promotion demands.Not the problem of existing traditional promotion system does not often account for a variety of promotion schemes, this
Sample tends not to flexibly support when being directed to different promotion schemes.For example user requires promotion process that cannot shut down, and works as number of users
When according to measuring very big, when not shutting down, generally requiring a variety of promotion schemes and being used cooperatively：It repeatedly increases newly in batches and is transferred and promoted, rises
Move the secondary update etc. of data.The promotion mode manager that the present invention designs is exactly to solve the above-mentioned problems.The manager can
In such a way that the judgement of intelligence or by hand selection are transferred and promoted need using any promotion every time, and it is synchronous also according to configuring condition selection
It is that asynchronous promotion executes, and then schedule subsequent is directed to the different promotion core processings of different modes design.
The present invention supports primary be transferred and promoted and multiple promotion two major classes promotion mode.
Primary to be transferred and promoted, as its name suggests, had been friends in the past system data is disposably all transferred and promoted into new system by promotion system.This
Kind of promotion mode is often directed to that old system data amount is smaller, and the promotion of shutting down that client allows is had plenty of time.Promotion scheme is often
It is that all data are disposably transferred and promoted into new system by old system-down.
Repeatedly be transferred and promoted, when the data volume of promotion is very huge, when the downtime of customer requirement is shorter, it is primary be transferred and promoted by
In overlong time, meet demand is tended not to.This just needs to split the whole promotion period, plans time model of being transferred and promoted every time
It encloses.Technically, very long due to restoring a database time, it is possible that needing support the online backup data of production environment
It is transferred and promoted as data source in library.Because being online promotion, practical business document is continually changing, to guarantee data
Consistency, repeatedly promotion is divided into repeatedly newly-increased be transferred and promoted again and is transferred and promoted with multi-fold.
It is repeatedly newly-increased to be transferred and promoted, a time range is specified, all documents do not exist in the database of new system,
At this moment all data are all finally newly-increased preservations, and system will record successfully log and error log.
Multi-fold is transferred and promoted, i.e. the upgrading of history promotion data.To within the scope of specified time for the history be transferred and promoted
Document synchronizes update, guarantees the consistency of new-old system data.System meeting synchronous recording success log and error log, this
A little logs also involve newly-increased, deletion, modification etc..It is specific as shown in Figure 3.
Third is realized and old system business data is transferred and promoted to the process of new system by promotion core processor.In order to
The scalability of the promotion system is further enhanced, the promotion core processor in the present invention is by promotion procedural abstraction at several
Function, in actual development and extension, research staff only needs to patrol for specific steps in combination with the existing business of new system
It collects and is developed.By conversion function therein, old system organization can there should be into the identifiable data knot of program at new system
Structure, and then can be using the API much saved about data check, data filling, data existing in new system application, this
Sample just takes full advantage of the application and development code of new system, to guarantee the integrality of promotion data.Meanwhile each of core processing
Committed step is to complete to load by configuration file and dispatch, this just makes entire promotion process height expansible, according to
Practical promotion demand, flexible configuration, the quick exploitation for supporting new business data to be transferred and promoted reduce development cost, and the system of improving can
Scalability.
It is just described in detail below for several major functions of core processor, as shown in Figure 4：
1, data query.Promotion system needs the data query configuration file for the business paper being transferred and promoted to give birth to automatically by reading
The business paper in old database is read according to the specific major key ID read in Database Dynamic paging at the sql sentence of inquiry
Detail passes data to and carries out data conversion in next step.
2, data conversion.Pass through the mapping table of old system document table and each field of new system document table, promotion system
By the data conversion read at the identifiable data structure of new system, all operations subsequent in this way are aiming at new system data
Operation, the existing various data filling of new system, verification the API such as save and then can directly use.New-old system literary name section is reflected
Penetrating relation table is actually an EXCEL table, and the maintenance of the table is got up very convenient, and exploitation is implemented even client and is ok
Flexible configuration according to actual needs.
3, data translation.The major key ID of a large amount of archives is had recorded in business datum table, these archives are main between heterogeneous system
Key may be inconsistent, to cause this phenomenon to be often possible to be due to the archives of new system be safeguard again or archives from old system
System, which does not account for saving former major key, when being transferred and promoted to new system or table structure changes greatly can not save former major key.At this moment, it is converting
After new system data structure, need to translate into the archives Field ID inside document the archives major key ID of new system.The present invention
Promotion system the archives field in each field of new system document can be automatically parsed by a metadata parser, according to difference
Archives field call different archives ID translaters to be translated.If the purpose system being transferred and promoted changes, only need to change
Quick Extended can be realized in metadata parser.
Meanwhile when analyzing the archives field of document, due to the uncertainty of old system, the shelves recorded on old document sometimes
Case field is wrong, such as the archives are deleted.When finding the problem in promotion system data translation process, will do it different
Regular data early warning and reparation step.It, can very flexible basis by realizing the early warning of promotion system development and repairing interface
Actual demand carries out dealing of abnormal data.
4, data processing.Data processing is generally divided into data filling and data check, past when a new document is before preservation
Toward needing to fill some backstage fields and verifying the data correctness of document, due to being converted into new system real-life program
The data structure used, it is possible to be loaded directly into the API of some data filling and verification of new system application program, quickly open
The code of data processing link is sent out, while also ensuring the correctness of data.
5, data save.Since promotion system of the invention is supported repeatedly to be transferred and promoted, thus data preservation may relate to it is new
Increase, modify and deletes.Promotion system, which has analyzed data to be processed by promotion mode manager and previous promotion log, to be had
Which kind of preservation operation body should carry out, so, when carrying out data preservation only need to analysis before as a result, classification is called newly
System API is saved.
4th, promotion log manager.Problem when log can handle all kinds of operations with help system, help system analysis are worked as
Preceding operating status, promotion system are no exception.Pass through the key log number for each operation of being transferred and promoted that promotion log manager is collected
According to it is that newly-increased promotion or covering are transferred and promoted, and can be received by log that promotion system, which can very easily analyze current operation,
Collect abnormal data, and then to repair abnormal data, location data or procedural problem provide foundation.
In conclusion technical solution of the present invention has the following effects that：
1. making full use of memory headroom to be transferred and promoted for the data of big data quantity, technical solution of the present invention uses database
Dynamically secondary paging technique.The data volume inquired in the existing each data page of Database Pagination Technology be it is certain, be changed to move
After state paging, preset memory can be more effectively utilized, paging number can be effectively reduced according to real data situation, improve and divide
Page search efficiency.
2. use a variety of promotion modes can with flexible combination to flexibly support various promotion scene demands, the present invention
Technical solution uses promotion mode manager, in this way when coping with promotion scene complicated and changeable, so that it may by a variety of promotion sides
Formula flexible combination enhances the business support range of promotion system, reduces the secondary development of promotion system.
3. reducing development cost in order to preferably combine existing new system, accelerate delivery process.The present invention will be transferred and promoted core
Heart process is abstracted into several functions.Wherein, old system data is converted into the identifiable data knot of new system by data conversion process
The API of many new systems thus can be used in structure, so that new system and promotion system is established connection, and very using new system
Mostly guarantee data correctness using API.It is often relatively isolated to compare existing promotion system and new system, this is resulted in
When really implementing some promotion project, former promotion system is largely developed for new system needs, causes development cost
Waste.
4. all promotions are scheduled execution by configuration file, and have opened multiple interfaces, to adapt to different promotions
Demand carries out the secondary development of rapid data promotion.
Finally it should be noted that：The foregoing is only a preferred embodiment of the present invention, is not intended to restrict the invention,
Although the present invention is described in detail referring to the foregoing embodiments, for those skilled in the art, still may be used
To modify the technical solutions described in the foregoing embodiments or equivalent replacement of some of the technical features.
All within the spirits and principles of the present invention, any modification, equivalent replacement, improvement and so on should be included in of the invention
Within protection scope.
Claims (8)
1. data promotion system between a kind of heterogeneous system, which is characterized in that managed including dynamic page controller, promotion mode
Device, promotion core processor and promotion log manager；
The dynamic page controller：Data are read out using dynamic secondary paging；
The promotion mode manager：Selection is transferred and promoted every time needs promotion mode to be used, and according to the configuration feelings of new-old system
Condition selection synchronization or asynchronous promotion, to generate scheduling configuration file；
The promotion core processor：The scheduling configuration file generated according to promotion mode manager completes the processing of data；
The promotion log manager：Collect the key log data for operation of being transferred and promoted every time；
The secondary paging of dynamic that the dynamic page controller uses is specially：
First all gauge outfit major key ID for the business paper being transferred and promoted will be needed to store into an interim table in a paging, then from
A certain number of major key ID are read in batches into memory in interim table, in secondary paging, are read in batches directly from memory
Actual traffic data is read into memory from old database by the major key ID of caching using the major key ID of reading.
2. data promotion system between heterogeneous system according to claim 1 in batches, which is characterized in that above-mentioned to be read from memory
It takes in the major key ID cached, the major key ID quantity of reading is generated after being calculated by analysis actual traffic data amount dynamic.
3. data promotion system between heterogeneous system according to claim 2, which is characterized in that promotion mode manager administration
Promotion mode, including：
It is primary to be transferred and promoted：I.e. had been friends in the past system data is disposably all transferred and promoted into new system by promotion system；
Repeatedly it is transferred and promoted：The promotion period that will be whole splits, and repeatedly carries out data promotion according to the promotion period of fractionation.
4. data promotion system between heterogeneous system according to claim 3, which is characterized in that the multiple promotion, including
Repeatedly newly-increased be transferred and promoted is transferred and promoted with multi-fold,
The repeatedly newly-increased promotion：I.e. within the scope of specified time, all data are all from old system in the database of new system
Newly-increased；
The multi-fold is transferred and promoted：Update is synchronized for the history document being transferred and promoted i.e. within the scope of specified time, is guaranteed
The consistency of new-old system data.
5. data promotion system between heterogeneous system according to claim 2, which is characterized in that the promotion core processing
Device, including：
Data query unit：The data query configuration file for the business paper being transferred and promoted is needed to automatically generate inquiry by reading
Sql sentence reads the business paper detail in old database according to the specific major key ID read in Database Dynamic paging, and
The data of reading are transferred to Date Conversion Unit；
Date Conversion Unit：By the mapping table of old system document table and each field of new system document table, by received number
According to the data conversion of query unit at the identifiable data structure of new system, to make full use of new system is developed various to answer
Use code；
Data translation unit：The data for receiving Date Conversion Unit, translate into new system for archives Field ID in the data after conversion
The archives major key ID of system；
Data processing unit：To carrying out field filling in backstage in data translation unit treated data and verify the correct of data
Property；
Data saving unit：Classification preservation is carried out to the data after data processing unit processes.
6. data promotion system between heterogeneous system according to claim 5, which is characterized in that further include：Early warning and reparation
Unit：It needs before archives Field ID is translated into the archives major key ID of new system by data translation unit to document in old system
Archives field is analyzed, and when the archives field recorded in old system is wrong, carries out abnormal data early warning and reparation.
7. data promotion system between heterogeneous system according to claim 5, which is characterized in that the Date Conversion Unit
In, the mapping table in the mapping table of old system document table and each field of new system document table is Microsoft Excel.
8. a kind of data promotion method between heterogeneous system, which is characterized in that including：
Data are read out using dynamic secondary paging；
Selection is transferred and promoted every time needs promotion mode to be used, and selects synchronous or asynchronous liter according to the configuring condition of new-old system
It moves, to generate scheduling configuration file；
The promotion processing of data is completed according to the scheduling configuration file of generation；
Collect the key log data for operation of being transferred and promoted every time；
The secondary paging of dynamic is specially：
First all gauge outfit major key ID for the business paper being transferred and promoted will be needed to store into an interim table in a paging, then from
A certain number of major key ID are read in batches into memory in interim table, in secondary paging, are read in batches directly from memory
The major key ID of caching, wherein the quantity of the major key ID cached read every time is according to analysis actual traffic data amount dynamic
It calculates, is read into memory actual traffic data from old database using the major key ID of reading.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510735226.0A CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201510735226.0A CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN105426419A CN105426419A (en) | 2016-03-23 |
| CN105426419B true CN105426419B (en) | 2018-11-20 |
Family
ID=55504631
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201510735226.0A Active CN105426419B (en) | 2015-11-03 | 2015-11-03 | Data promotion system and method between heterogeneous system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN105426419B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106407321B (en) * | 2016-08-31 | 2019-12-06 | 东软集团股份有限公司 | Method and device for synchronizing data |
| CN108280142B (en) * | 2017-12-29 | 2022-01-04 | 武汉钢铁工程技术集团通信有限责任公司 | Method for butt-jointing transaction data of new and old systems |
| CN108595487B (en) * | 2018-03-14 | 2022-04-29 | 武汉村助手科技有限公司 | Method and system for accessing data under high concurrency of big data |
| CN109684528A (en) * | 2018-12-04 | 2019-04-26 | 深圳店匠科技有限公司 | Inter-library paging query method, system and storage medium |
| CN109783779B (en) * | 2018-12-27 | 2023-12-22 | 北京航天福道高技术股份有限公司 | Tool for importing, exporting and converting data EXCEL |
| CN112069190B (en) * | 2019-06-11 | 2023-06-09 | 腾讯科技（深圳）有限公司 | Batch data acquisition method, device, equipment and medium |
| CN113010607A (en) * | 2021-04-06 | 2021-06-22 | 工银科技有限公司 | Method, device, computer system and storage medium for data synchronization between systems |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101183371A (en) * | 2007-12-12 | 2008-05-21 | 中兴通讯股份有限公司 | Method for quick finishing large data-handling and reporting system |
| CN101261693A (en) * | 2007-03-09 | 2008-09-10 | 上海宝信软件股份有限公司 | A data archiving method in computer application system |
| CN102236672A (en) * | 2010-05-06 | 2011-11-09 | 深圳市腾讯计算机系统有限公司 | Method and device for importing data |
| CN103489061A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Batch data processing device and batch data processing method |
| US8868506B1 (en) * | 2010-06-17 | 2014-10-21 | Evolphin Software, Inc. | Method and apparatus for digital asset management |
-
2015
- 2015-11-03 CN CN201510735226.0A patent/CN105426419B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101261693A (en) * | 2007-03-09 | 2008-09-10 | 上海宝信软件股份有限公司 | A data archiving method in computer application system |
| CN101183371A (en) * | 2007-12-12 | 2008-05-21 | 中兴通讯股份有限公司 | Method for quick finishing large data-handling and reporting system |
| CN102236672A (en) * | 2010-05-06 | 2011-11-09 | 深圳市腾讯计算机系统有限公司 | Method and device for importing data |
| US8868506B1 (en) * | 2010-06-17 | 2014-10-21 | Evolphin Software, Inc. | Method and apparatus for digital asset management |
| CN103489061A (en) * | 2013-09-04 | 2014-01-01 | 用友软件股份有限公司 | Batch data processing device and batch data processing method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN105426419A (en) | 2016-03-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN105426419B (en) | Data promotion system and method between heterogeneous system | |
| CN104866599B (en) | The production method and system of Visual Report Forms | |
| CN109034993A (en) | Account checking method, equipment, system and computer readable storage medium | |
| CN101576918A (en) | Data buffering system with load balancing function | |
| CN103514223A (en) | Data synchronism method and system of database | |
| CN106021445A (en) | Cached data loading method and apparatus | |
| CN102880683B (en) | The automatic network of a kind of feasibility study report generates system and the method for generation thereof | |
| CN109325200B (en) | Method and device for acquiring data and computer readable storage medium | |
| CN110209668A (en) | Dimension table correlating method, device, equipment and readable storage medium storing program for executing based on stream calculation | |
| CN105787058A (en) | User label system and data pushing system based on same | |
| CN101685452A (en) | Data warehouse scheduling method and scheduling system | |
| CN102156744A (en) | Method for eliminating repetition of memory dialog list | |
| CN116107993A (en) | Data migration evaluation method and system in MySQL protocol database | |
| CN105279138A (en) | Automatic generation system of message research report | |
| CN101710322A (en) | Method and system for associating information | |
| CN113918532A (en) | Portrait label aggregation method, electronic device and storage medium | |
| CN111309740A (en) | Data processing method, device, equipment and storage medium | |
| CN105260411A (en) | XBRL-based system and method for achieving detail data aggregation and drill-down of financial report | |
| WO2003058399A2 (en) | Method and system for adaptive software system interface and external database synchronization | |
| CN113793213B (en) | Method and device for implementing decision mode of asynchronous credit wind control breakpoint continuous operation | |
| CN112559641B (en) | Pull chain table processing method and device, readable storage medium and electronic equipment | |
| CN112612841A (en) | Knowledge extraction construction method, device, equipment and storage medium | |
| CN111767703A (en) | Form data acquisition method, device and system | |
| CN109634606A (en) | A kind of method and device of defined function menu | |
| CN112561368B (en) | Visual performance calculation method and device for OA approval system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |