CN104361133A - Data extraction device and method - Google Patents
Data extraction device and method Download PDFInfo
- Publication number
- CN104361133A CN104361133A CN201410750223.XA CN201410750223A CN104361133A CN 104361133 A CN104361133 A CN 104361133A CN 201410750223 A CN201410750223 A CN 201410750223A CN 104361133 A CN104361133 A CN 104361133A
- Authority
- CN
- China
- Prior art keywords
- data
- state
- extracted
- type
- extraction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Abstract
The invention provides a data extraction device comprising an on-off state loading unit, a data extraction controller and a data extraction unit, wherein the on-off state loading unit is used for loading an on-off state of a control switch during the current data extraction, and storing the on-off state of the control switch during previous data extraction; the data extraction controller is used for judging whether the type of extracted data in the current data extraction is a total extraction type or an increment extraction type according to the loaded on-off state, and respectively calculating to-be-extracted, screened, filed and appointed data when the type of the extracted data is the increment extraction type; and the data extraction unit is used for extracting the data according to the calculated to-be-extracted, screened, filed and appointed data. The invention also provides a data extraction method. Through the technical scheme disclosed by the invention, the data extraction of a plurality of object types can be finished by fully utilizing a single object type on the basis of an existing data extraction mode; and a universal and uniform extraction ides into which a plurality of object types participate is built.
Description
Technical field
The present invention relates to technical field of data processing, particularly, relate to a kind of data pick-up device and a kind of data pick-up method.
Background technology
Large data are seen everywhere at present, and increasing enterprise starts layout data warehouse, assist enterprise to make wise business business decision by data mining and business analysis instrument.The data that each operation system of enterprise produces provide high-quality data by ETL instrument to data warehouse.ETL comprises the extraction of data, conversion and loading 3 links, data pick-up is towards various Data Source, such as heterogeneous database, file etc., how from the extracted data of the correct stability and high efficiency of these data sources be one of key issue must considered ETL design process.Abstracting method popular at present all has some limitations.
1. full dose extracts: each by all data of operation system all by ETL tool loads to data warehouse, simply, but when data volume is increasing, there is performance bottleneck in this mode in extraction, and causing can not real-time update data.
2. based on the pushing-type increment extraction of triggering mode: the performance of data pick-up is higher, energy real-time update data, but trigger is set up in requirement in Service Database or program coding pushes away data to data warehouse, need to adjust operation system or Service Database, not only affect the security of the stable of operation system and business datum, have certain performance impact to operation system simultaneously.
3. based on the increment extraction of timestamp: the same with similar trigger, performance is also relatively good, process is relatively clear simple, but if having time field is stabbed to operation system mandatory requirement, meanwhile, modification time stamp field during Data Update, and the data of deleting are needed to be recorded in storehouse, can not complete deletion, data accuracy has certain restriction.
4. full table comparison increment extraction: determine that the additions and deletions of data change, poor-performing by entirely showing comparison.
5. database journal contrast: judged the data changed by the daily record of analytical database self, by the restriction of type of database and version, do not support heterogeneous database.
Therefore, need a kind of new Data Extraction Technology, on existing data pick-up mode basis, the data pick-up that single object type completes multi-object type can be made full use of, set up general, the unified extraction thinking of the data pick-up that multi-object type participates in.
Summary of the invention
The present invention is just based on the problems referred to above, propose a kind of new Data Extraction Technology, on existing data pick-up mode basis, the data pick-up that single object type completes multi-object type can be made full use of, set up general, the unified extraction thinking of the data pick-up that multi-object type participates in.
In view of this, the present invention proposes a kind of data pick-up device, comprising: on off state is loaded into unit, for being loaded into the on off state of gauge tap when this extracted data, and preserving the on off state of gauge tap when extracted data last time; Data pick-up controller, for according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying; Data pick-up unit, for needing extraction according to what calculate, filter and file the data of specifying, carries out data pick-up.In this technical scheme, increment extraction can be carried out according to data switch information and business date when carrying out data pick-up, effectively evade the limitation of current extraction mode, and data warehouse and Service Database independent operating, be beneficial to the security of the stable of enterprise's service control system and business datum.
In technique scheme, preferably, described on off state is loaded into unit, specifically comprises: state insmods, for being loaded into the on off state of gauge tap when this extracted data; State preserves module, for based on the on off state of the gauge tap be loaded into when this extracted data, transfers and preserves the on off state of gauge tap when extracted data last time.In this technical scheme, also can carry out increment extraction even if exist physics is deleted in operation system, the data extrapolating meeting increment extraction is high, little on impacts such as source operation system performances.
In technique scheme, preferably, described data pick-up controller, specifically comprises: extract type judging module, for according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; Data computation module, for when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.In this technical scheme, can according to based on the foundation of the data switch in operation system as data increment extraction, when switch opening, up-to-date closedown or do the data of correspondence in pent three kinds of situations again after being switched on again extracts again, screening and filing.
In technique scheme, preferably, described data pick-up unit, specifically comprises: initialization full dose abstraction module, and for needing extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts; Daily increment extraction module, for the data extracted according to initialization full dose, carries out the daily increment extraction of data.In this technical scheme, mainly through comparing the data switch state of operation system, realize the increment extraction of corresponding business datum.
In technique scheme, preferably, the operation that the initialization full dose that described initialization full dose abstraction module carries out data extracts, comprises: the current state of (1) loading switch extracted state to last time further; (2) extract by full dose mode control data; (3) full dose extracts business datum, does the initialization operation of data warehouse; And/or described daily increment extraction module carries out the operation of the daily increment extraction of data, comprises further: (1) load this on off state and extract mode bit to this; (2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract; (3) according to the data pick-up time calculated, increment extraction, filtration loading data; (4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.In this technical scheme, the data volume of increment extraction is little, thus extracts the characteristics such as performance is high.
According to a further aspect of the invention, also proposed a kind of data pick-up method, comprising: step 202: be loaded into the on off state of gauge tap when this extracted data, and preserve the on off state of gauge tap when extracted data last time; Step 204: according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying; Step 206: need extraction according to what calculate, filter and file the data of specifying, carry out data pick-up.In this technical scheme, increment extraction can be carried out according to data switch information and business date when carrying out data pick-up, effectively evade the limitation of current extraction mode, and data warehouse and Service Database independent operating, be beneficial to the security of the stable of enterprise's service control system and business datum.
In technique scheme, preferably, described step 202, specifically comprises: step 302: be loaded into the on off state of gauge tap when this extracted data; Step 304: based on the on off state of the gauge tap be loaded into when this extracted data, transfer and preserve the on off state of gauge tap when extracted data last time.In this technical scheme, also can carry out increment extraction even if exist physics is deleted in operation system, the data extrapolating meeting increment extraction is high, little on impacts such as source operation system performances.
In technique scheme, preferably, described step 204, specifically comprises: step 402: according to the on off state be loaded into, and judges that the type of this extracted data is that full dose extracts type or increment extraction type; Step 404: when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.In this technical scheme, can according to based on the foundation of the data switch in operation system as data increment extraction, when switch opening, up-to-date closedown or do the data of correspondence in pent three kinds of situations again after being switched on again extracts again, screening and filing.
In technique scheme, preferably, described step 206, specifically comprises: step 502: need extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts; Step 504: the data extracted according to initialization full dose, carries out the daily increment extraction of data.In this technical scheme, mainly through comparing the data switch state of operation system, realize the increment extraction of corresponding business datum.
In technique scheme, preferably, the operation that the initialization full dose that described step 502 carries out data extracts, comprises: the current state of (1) loading switch extracted state to last time further; (2) extract by full dose mode control data; (3) full dose extracts business datum, does the initialization operation of data warehouse; And/or described step 504 carries out the operation of the daily increment extraction of data, comprises further: (1) load this on off state and extract mode bit to this; (2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract; (3) according to the data pick-up time calculated, increment extraction, filtration loading data; (4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.In this technical scheme, the data volume of increment extraction is little, thus extracts the characteristics such as performance is high.
By above technical scheme, on existing data pick-up mode basis, the data pick-up that single object type completes multi-object type can be made full use of, set up general, the unified extraction thinking of the data pick-up that multi-object type participates in.
Accompanying drawing explanation
Fig. 1 shows the block diagram of data pick-up device according to an embodiment of the invention;
Fig. 2 shows the process flow diagram of data pick-up method according to an embodiment of the invention;
Fig. 3 shows on off state according to an embodiment of the invention and is loaded into the process flow diagram of unit;
Fig. 4 shows the process flow diagram of data pick-up controller according to an embodiment of the invention;
Fig. 5 shows the process flow diagram of data pick-up unit according to an embodiment of the invention;
Fig. 6 shows the schematic diagram doing incremental data extraction according to an embodiment of the invention according to data switch;
Fig. 7 has gone out the process flow diagram of the logic of full dose extraction according to an embodiment of the invention;
Fig. 8 has gone out the process flow diagram of daily according to an embodiment of the invention increment extraction.
Embodiment
In order to more clearly understand above-mentioned purpose of the present invention, feature and advantage, below in conjunction with the drawings and specific embodiments, the present invention is further described in detail.It should be noted that, when not conflicting, the feature in the embodiment of the application and embodiment can combine mutually.
Set forth a lot of detail in the following description so that fully understand the present invention; but; the present invention can also adopt other to be different from other modes described here and implement, and therefore, protection scope of the present invention is not by the restriction of following public specific embodiment.
Fig. 1 shows the block diagram of data pick-up device according to an embodiment of the invention.
As shown in Figure 1, data pick-up device 100 according to an embodiment of the invention, comprising: on off state is loaded into unit 102, for being loaded into the on off state of gauge tap when this extracted data, and preserves the on off state of gauge tap when extracted data last time; Data pick-up controller 104, for according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying; Data pick-up unit 106, for needing extraction according to what calculate, filter and file the data of specifying, carries out data pick-up.In this technical scheme, increment extraction can be carried out according to data switch information and business date when carrying out data pick-up, effectively evade the limitation of current extraction mode, and data warehouse and Service Database independent operating, be beneficial to the security of the stable of enterprise's service control system and business datum.
In technique scheme, preferably, on off state is loaded into unit 102, specifically comprises: state insmods 1022, for being loaded into the on off state of gauge tap when this extracted data; State preserves module 1024, for based on the on off state of the gauge tap be loaded into when this extracted data, transfers and preserves the on off state of gauge tap when extracted data last time.In this technical scheme, also can carry out increment extraction even if exist physics is deleted in operation system, the data extrapolating meeting increment extraction is high, little on impacts such as source operation system performances.
In technique scheme, preferably, data pick-up controller 104, specifically comprises: extract type judging module 1042, for according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; Data computation module 1044, for when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.In this technical scheme, can according to based on the foundation of the data switch in operation system as data increment extraction, when switch opening, up-to-date closedown or do the data of correspondence in pent three kinds of situations again after being switched on again extracts again, screening and filing.
In technique scheme, preferably, data pick-up unit 106, specifically comprises: initialization full dose abstraction module 1062, and for needing extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts; Daily increment extraction module 1066, for the data extracted according to initialization full dose, carries out the daily increment extraction of data.In this technical scheme, mainly through comparing the data switch state of operation system, realize the increment extraction of corresponding business datum.
In technique scheme, preferably, the operation that the initialization full dose that initialization full dose abstraction module 1062 carries out data extracts, comprises: the current state of (1) loading switch extracted state to last time further; (2) extract by full dose mode control data; (3) full dose extracts business datum, does the initialization operation of data warehouse; And/or daily increment extraction module 1064 carries out the operation of the daily increment extraction of data, comprises further: (1) load this on off state and extract mode bit to this; (2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract; (3) according to the data pick-up time calculated, increment extraction, filtration loading data; (4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.In this technical scheme, the data volume of increment extraction is little, thus extracts the characteristics such as performance is high.
Fig. 2 shows the process flow diagram of data pick-up method according to an embodiment of the invention.
As shown in Figure 2, data pick-up method according to an embodiment of the invention, comprising: step 202: be loaded into the on off state of gauge tap when this extracted data, and preserve the on off state of gauge tap when extracted data last time; Step 204: according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying; Step 206: need extraction according to what calculate, filter and file the data of specifying, carry out data pick-up.In this technical scheme, increment extraction can be carried out according to data switch information and business date when carrying out data pick-up, effectively evade the limitation of current extraction mode, and data warehouse and Service Database independent operating, be beneficial to the security of the stable of enterprise's service control system and business datum.
In technique scheme, preferably, as shown in Figure 3, step 202, specifically comprises: step 302: be loaded into the on off state of gauge tap when this extracted data; Step 304: based on the on off state of the gauge tap be loaded into when this extracted data, transfer and preserve the on off state of gauge tap when extracted data last time.In this technical scheme, also can carry out increment extraction even if exist physics is deleted in operation system, the data extrapolating meeting increment extraction is high, little on impacts such as source operation system performances.
In technique scheme, preferably, as shown in Figure 4, step 204, specifically comprises: step 402: according to the on off state be loaded into, and judges that the type of this extracted data is that full dose extracts type or increment extraction type; Step 404: when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.In this technical scheme, can according to based on the foundation of the data switch in operation system as data increment extraction, when switch opening, up-to-date closedown or do the data of correspondence in pent three kinds of situations again after being switched on again extracts again, screening and filing.
In technique scheme, preferably, as shown in Figure 5, step 206, specifically comprises: step 502: need extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts; Step 504: the data extracted according to initialization full dose, carries out the daily increment extraction of data.In this technical scheme, mainly through comparing the data switch state of operation system, realize the increment extraction of corresponding business datum.
In technique scheme, preferably, the operation that the initialization full dose that step 502 carries out data extracts, comprises: the current state of (1) loading switch extracted state to last time further; (2) extract by full dose mode control data; (3) full dose extracts business datum, does the initialization operation of data warehouse; And/or step 504 carries out the operation of the daily increment extraction of data, comprises further: (1) load this on off state and extract mode bit to this; (2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract; (3) according to the data pick-up time calculated, increment extraction, filtration loading data; (4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.In this technical scheme, the data volume of increment extraction is little, thus extracts the characteristics such as performance is high.
Technical scheme of the present invention, provides a kind of data pick-up method and apparatus based on data switch.Enterprise is in real work, often need to carry out loading cleaning and conversion to business datum, but along with data volume is increasing, full dose extraction is carried out to data at every turn and become more and more difficult, and carry out increment extraction than equity there is a lot of limitation according to timestamp, trigger, full table.But consider in practical business, enterprises, usually can setting data switch in order to ensure the stability of business datum.Therefore, technical scheme of the present invention carries out increment extraction when carrying out data pick-up according to data switch information and business date, effectively evade the limitation of current extraction mode, and data warehouse and Service Database independent operating, be beneficial to the security of the stable of enterprise's service control system and business datum.
Along with the arriving of large data age, to data pick-up, filter with conversion demand also become increasingly complex, for the method for data pick-up requirement also more and more harshness.And the method for data pick-up presents very superior showed in some cases, and show in other cases not fully up to expectations, even completely inapplicable.
Technical scheme of the present invention, is mainly applicable to following situation:
1, there is data switch in operation system, and record the operation version of each switch; 2, data switch can allow again to open after closedown, also can allow to open, but finally should be tending towards closing; 3, the pent corresponding data of data switch cannot be revised, and must reopen data switch if will revise.Technical scheme of the present invention, mainly through comparing the data switch state of operation system, realizes the increment extraction of corresponding business datum.
In order to effectively solve prior art Problems existing, have devised a kind of device doing incremental data extraction according to data switch at this, this device is made up of, see Fig. 6 following three parts:
On off state is loaded into unit: its Main Function is to be loaded into the state of gauge tap when this extracted data, and preserve the on off state extracted last time, so that data pick-up controller judges the part data needing again to extract, screen and file respectively.
Data pick-up controller: it needs the on off state be loaded into according on off state loading unit to judge to do full dose extraction or increment extraction, when doing increment extraction, calculates the part data needing extraction, screening and file respectively.
Data pick-up unit: extract according to data pick-up controller, filter and file the data of specifying.The data pick-up one of this data pick-up unit is divided into two parts, i.e. initialization full dose extracts and daily increment extraction.Wherein the logic of full dose extraction is fairly simple, can be divided into two steps: concrete steps are see Fig. 7.
Daily increment extraction step is see Fig. 8.Note: the comparison principle of the method in the 2nd step is as follows:
(1) this switch opens, data need again corresponding extraction and filing;
This switch close, when last time switch open, need again extract corresponding data and file;
(3) this switch cuts out, and last time also closes, but centre is once opened, needs again extract corresponding data and file.
The object lesson of our inventory auditing below.Inventory auditing module needs to do the moon by the combination in each accounting moon and cost territory and to check out operation, just cannot revise once some information such as corresponding cost territory after monthly closing entry and the documents that combines by the accounting moon, and this operation of monthly closing entry is allowed to reopen amendment owing to there is mistake, and to open amendment be also regular, namely monthly closing entry can carry out monthly closing entry according to the order of the accounting moon, if latter one month namely will be tied, the previous moon must be checkout, if and to take in reef knot account one month, need in month below first to do the operation of reef knot account.In inventory auditing module, data volume very large, full dose extracts very consumption of natural resource and time, but due to data volume large, there are some operation systems directly to carry out physics deletion action to some data bank service tables, make to carry out the impossible of data increment extraction change by the mode of timestamp.
We carry out the increment extraction of data, filtration and filing with regard to using the present invention below.First determine that the list structure of the monthly closing entry tables of data primary fields as data switch is as follows:
table 1
。
The storage mode of table 1 stores data when being the combination monthly closing entry in each accounting moon and cost territory, if there is the phenomenon of anti-monthly closing entry, just data delete flag is labeled as deletion.The tables of data related to is for material monthly financial statement and detailed document, and the structure of their tables of data primary fields is as shown in the table:
table 2
。
table 3
。
Wherein table 2 is for the memory space of often kind of material in each accounting moon and cost territory and the amount of money, and their average unit price, this table does not use physics to delete, occur reef knot account time by delete flag for being designated as deletion, if there is the operation of reef knot account, just by this data modification and cancel (CANCL) segment mark be designated as and do not delete.And table 3 is detailed document table, each detailed document data, data volume very huge, therefore in operation system, employ physics delete.
The method in the present invention is just used to solve appeal problem below, tables of data 1 can as data switch table, if settled accounts, namely an effective record has been stored in tables of data 1, so be directed to this and record that corresponding cost territory and the data switch that combines by the accounting moon be considered to close, otherwise just think that it is opened.First initialized step is had a look:
Using tables of data 1 as clock switch, and its data full dose is loaded into the result that extracted as last time of data warehouse;
(2) extraction state is set to full dose extraction and filing by data pick-up controller;
(3) data pick-up device extracts the data to data warehouse of table 2 and table 3 according to the state full dose of controller, and data full dose is filed data exhibiting layer.
The operation of data initialization is relatively simple, but the mode still using full dose to extract when scheduler routine time will unusual consumption of natural resource, must use the mode of increment extraction.First, because tables of data 1 meets the prerequisite by increment extraction, so to tables of data 1 based on timestamp increment extraction data, the data due to increment extraction are all that after last extraction, data switch did variation, again extract and filing so the data that these Switch Controller are answered all should be done.In addition, the corresponding part business datum that after extracting last time, data switch is not closed also is likely change, so this part data also needs again to extract and file.Be exactly more than the strategy of daily increment extraction data, have a look the step of daily increment extraction below again:
(1) his-and-hers watches 1 do data increment extraction based on timestamp, are loaded in data warehouse.
(2) data pick-up mode is set to increment extraction by data pick-up controller, and the data markers corresponding to the combination in the accounting moon corresponding to data the 1st step extracted and cost territory again extracts for needs and files, and not had the part data markers of closedown again extract for needs and file in the data switch extracted last time.At this these are needed again to extract and be designated as set A with the cost territory of filing and the combination of the accounting moon.
(3) for tables of data 2, delete owing to there is no physics, so more efficient timestamp increment extraction mode can be used, by data interface tier by data pick-up in data warehouse, then the part data that the accounting moon and the combination of cost territory belong to set A are done again increment filing process.
(4) for tables of data 3, owing to there being physics to delete, so the increment extraction mode based on timestamp cannot be used.The combination in the accounting moon and cost territory in table 3 is belonged to detailed document in set A by data interface tier increment extraction in data warehouse, and this part data is filed data exhibiting layer data table again.
By the Data Update of the up-to-date change of increment extraction in the 1st step to extracted data on off state table last time, itself and business library are consistent, and the state as this data switch retains, so as next time use.
Technical scheme of the present invention, according to based on the foundation of the data switch in operation system as data increment extraction, when switch opening, up-to-date closedown or do the data of correspondence in pent three kinds of situations again after being switched on again extracts again, screening and filing.Even if exist physics is deleted in operation system like this and also can carry out increment extraction, the data extrapolating meeting increment extraction is high, little on impacts such as source operation system performances.Meanwhile, the data volume of increment extraction is little, thus extracts the characteristics such as performance is high.
More than be described with reference to the accompanying drawings technical scheme of the present invention, considered in correlation technique there is no easy, the unified solution extracted for complex type data.Existing data pick-up cannot complete the data extraction process that complicated type participates in.Therefore, the present invention proposes a kind of data pick-up device and a kind of data pick-up method, on existing data pick-up mode basis, the data pick-up that single object type completes multi-object type can be made full use of, set up general, the unified extraction thinking of the data pick-up that multi-object type participates in.
The foregoing is only the preferred embodiments of the present invention, be not limited to the present invention, for a person skilled in the art, the present invention can have various modifications and variations.Within the spirit and principles in the present invention all, any amendment done, equivalent replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (10)
1. a data pick-up device, is characterized in that, comprising:
On off state is loaded into unit, for being loaded into the on off state of gauge tap when this extracted data, and preserves the on off state of gauge tap when extracted data last time;
Data pick-up controller, for according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying;
Data pick-up unit, for needing extraction according to what calculate, filter and file the data of specifying, carries out data pick-up.
2. data pick-up device according to claim 1, is characterized in that, described on off state is loaded into unit, specifically comprises:
State insmods, for being loaded into the on off state of gauge tap when this extracted data;
State preserves module, for based on the on off state of the gauge tap be loaded into when this extracted data, transfers and preserves the on off state of gauge tap when extracted data last time.
3. data pick-up device according to claim 1 and 2, is characterized in that, described data pick-up controller, specifically comprises:
Extract type judging module, for according to the on off state be loaded into, judge that the type of this extracted data is that full dose extracts type or increment extraction type;
Data computation module, for when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.
4. data pick-up device according to claim 1 and 2, is characterized in that, described data pick-up unit, specifically comprises:
Initialization full dose abstraction module, for needing extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts;
Daily increment extraction module, for the data extracted according to initialization full dose, carries out the daily increment extraction of data.
5. data pick-up device according to claim 4, is characterized in that, the operation that the initialization full dose that described initialization full dose abstraction module carries out data extracts, and comprises further:
(1) the current state of loading switch extracted state to last time;
(2) extract by full dose mode control data;
(3) full dose extracts business datum, does the initialization operation of data warehouse;
And/or,
Described daily increment extraction module carries out the operation of the daily increment extraction of data, comprises further:
(1) load this on off state and extract mode bit to this;
(2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract;
(3) according to the data pick-up time calculated, increment extraction, filtration loading data;
(4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.
6. a data pick-up method, is characterized in that, comprising:
Step 202: be loaded into the on off state of gauge tap when this extracted data, and preserve the on off state of gauge tap when extracted data last time;
Step 204: according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type; When the type of this extracted data is increment extraction type, calculates respectively and need to extract, screen and file the data of specifying;
Step 206: need extraction according to what calculate, filter and file the data of specifying, carry out data pick-up.
7. data pick-up method according to claim 6, is characterized in that, described step 202, specifically comprises:
Step 302: be loaded into the on off state of gauge tap when this extracted data;
Step 304: based on the on off state of the gauge tap be loaded into when this extracted data, transfer and preserve the on off state of gauge tap when extracted data last time.
8. the data pick-up method according to claim 6 or 7, is characterized in that, described step 204, specifically comprises:
Step 402: according to the on off state be loaded into, judges that the type of this extracted data is that full dose extracts type or increment extraction type;
Step 404: when the type of this extracted data is increment extraction type, calculates respectively and needs to extract, screen and file the data of specifying.
9. the data pick-up method according to claim 6 or 7, is characterized in that, described step 206, specifically comprises:
Step 502: need extraction according to what calculate, filter and file the data of specifying, the initialization full dose of carrying out data extracts;
Step 504: the data extracted according to initialization full dose, carries out the daily increment extraction of data.
10. data pick-up method according to claim 9, is characterized in that, the operation that the initialization full dose that described step 502 carries out data extracts, and comprises further:
(1) the current state of loading switch extracted state to last time;
(2) extract by full dose mode control data;
(3) full dose extracts business datum, does the initialization operation of data warehouse;
And/or,
Described step 504 carries out the operation of the daily increment extraction of data, comprises further:
(1) load this on off state and extract mode bit to this;
(2) the time point that the state computation extracted by this and last time goes out to need incremental data to extract;
(3) according to the data pick-up time calculated, increment extraction, filtration loading data;
(4) current switch states data conversion storage was extracted state position to last time, used when convenient next time is extracted.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410750223.XA CN104361133B (en) | 2014-12-10 | 2014-12-10 | Data pick-up device and method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410750223.XA CN104361133B (en) | 2014-12-10 | 2014-12-10 | Data pick-up device and method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104361133A true CN104361133A (en) | 2015-02-18 |
| CN104361133B CN104361133B (en) | 2019-06-21 |
Family
ID=52528393
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410750223.XA Active CN104361133B (en) | 2014-12-10 | 2014-12-10 | Data pick-up device and method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104361133B (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512176A (en) * | 2015-11-24 | 2016-04-20 | 北京中电普华信息技术有限公司 | Increment extracting method and system based on information powercenter |
| CN105760485A (en) * | 2016-02-17 | 2016-07-13 | 上海携程商务有限公司 | Financial data extraction method and system |
| CN106126612A (en) * | 2016-06-22 | 2016-11-16 | 重庆秒银科技有限公司 | A kind of big ETL process dynamically divides the data pick-up method of timeslice |
| CN107229721A (en) * | 2017-06-02 | 2017-10-03 | 泰华智慧产业集团股份有限公司 | A kind of method and device for changing data pick-up |
| CN108876585A (en) * | 2018-09-29 | 2018-11-23 | 金蝶软件（中国）有限公司 | A kind of method and relevant device of across phase reef knot account |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177892A1 (en) * | 2007-01-19 | 2008-07-24 | International Business Machines Corporation | Method for service oriented data extraction transformation and load |
| CN101923566A (en) * | 2010-06-24 | 2010-12-22 | 浙江协同数据系统有限公司 | Data increment extraction method based on trigger |
| CN102375891A (en) * | 2011-11-15 | 2012-03-14 | 山东浪潮金融信息系统有限公司 | Implementation tool for unloading and loading incremental data |
| CN102508908A (en) * | 2011-11-11 | 2012-06-20 | 北京用友政务软件有限公司 | Method for acquiring subordinate financial business data and system for acquiring subordinate financial business data |
| CN104102737A (en) * | 2014-07-28 | 2014-10-15 | 中国农业银行股份有限公司 | Historical data storage method and system |
-
2014
- 2014-12-10 CN CN201410750223.XA patent/CN104361133B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080177892A1 (en) * | 2007-01-19 | 2008-07-24 | International Business Machines Corporation | Method for service oriented data extraction transformation and load |
| CN101923566A (en) * | 2010-06-24 | 2010-12-22 | 浙江协同数据系统有限公司 | Data increment extraction method based on trigger |
| CN102508908A (en) * | 2011-11-11 | 2012-06-20 | 北京用友政务软件有限公司 | Method for acquiring subordinate financial business data and system for acquiring subordinate financial business data |
| CN102375891A (en) * | 2011-11-15 | 2012-03-14 | 山东浪潮金融信息系统有限公司 | Implementation tool for unloading and loading incremental data |
| CN104102737A (en) * | 2014-07-28 | 2014-10-15 | 中国农业银行股份有限公司 | Historical data storage method and system |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105512176A (en) * | 2015-11-24 | 2016-04-20 | 北京中电普华信息技术有限公司 | Increment extracting method and system based on information powercenter |
| CN105512176B (en) * | 2015-11-24 | 2019-07-09 | 北京中电普华信息技术有限公司 | A kind of increment extraction method and system based on Informatica Powercenter |
| CN105760485A (en) * | 2016-02-17 | 2016-07-13 | 上海携程商务有限公司 | Financial data extraction method and system |
| CN106126612A (en) * | 2016-06-22 | 2016-11-16 | 重庆秒银科技有限公司 | A kind of big ETL process dynamically divides the data pick-up method of timeslice |
| CN107229721A (en) * | 2017-06-02 | 2017-10-03 | 泰华智慧产业集团股份有限公司 | A kind of method and device for changing data pick-up |
| CN107229721B (en) * | 2017-06-02 | 2019-10-29 | 泰华智慧产业集团股份有限公司 | A kind of method and device changing data pick-up |
| CN108876585A (en) * | 2018-09-29 | 2018-11-23 | 金蝶软件（中国）有限公司 | A kind of method and relevant device of across phase reef knot account |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104361133B (en) | 2019-06-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110933953B (en) | Systems and methods for extending blockchain utility through use of related sub-blockchains | |
| CN104361133A (en) | Data extraction device and method | |
| CN103370691A (en) | Managing buffer overflow conditions | |
| CN109213598A (en) | A kind of resource allocation methods, device and computer readable storage medium | |
| DiFrancesco et al. | Flexibility in water resources management: Review of concepts and development of assessment measures for flood management systems | |
| CN102521269A (en) | Index-based computer continuous data protection method | |
| CN106557272A (en) | A kind of efficient sensor historic data archiving method | |
| CN102332004B (en) | Data processing method and system for managing mass data | |
| CN103577402A (en) | Task adding, modifying and management method and task management system | |
| CN111639121A (en) | Big data platform and method for constructing customer portrait | |
| CN102810115A (en) | Method for implementing multi-layer distributed document management system | |
| CN105573673A (en) | Database based data cache system | |
| CN104537148A (en) | Statistical method for bolt information in PDMS (Product Data Management System) model | |
| CN105159820A (en) | Transmission method and device of system log data | |
| CA3200883A1 (en) | Multi-cache based digital output generation | |
| CN104050251B (en) | A kind of file management method and management system | |
| CN101957840B (en) | Storage and optimization method of MPI (Message Passing Interface) parallel data | |
| Pereshybkina et al. | Industry 4.0 Scenario Planning: How will the industry 4.0 transformations affect SMEs in Germany by 2030? | |
| Sharma et al. | Enhancing business intelligence using data warehousing: A Multi Case Analysis | |
| Abdul Azizurrofi et al. | Meeting the Strategy of Onshore Petroleum Exploration Based on Statistical Analysis of Operating Cost and Commercial Reserves in Indonesia | |
| CN107392750A (en) | A kind of enterprise's financial data memory management method | |
| Nwaokorie et al. | Gas Cap Development Guideline: Closing the Gap Between Effective Reservoir management and Contract Obligations | |
| Yuan et al. | Improvement of snapshot differential algorithm based on hadoop platform | |
| Kogler | A discrete event simulation model to test multimodal strategies for a greener and more resilient wood supply in Austria | |
| Lim et al. | Valuation and Optimal Timing of the Investment in Next Generation Telecommunication Service Using Real Options |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Applicant after: Yonyou Network Technology Co., Ltd. Address before: 100094 Beijing city Haidian District North Road No. 68, UFIDA Software Park Applicant before: UFIDA Software Co., Ltd. |
|
| COR | Change of bibliographic data | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |