CN102663020A - CDC data distribution method and device thereof - Google Patents
CDC data distribution method and device thereof Download PDFInfo
- Publication number
- CN102663020A CN102663020A CN2012100769289A CN201210076928A CN102663020A CN 102663020 A CN102663020 A CN 102663020A CN 2012100769289 A CN2012100769289 A CN 2012100769289A CN 201210076928 A CN201210076928 A CN 201210076928A CN 102663020 A CN102663020 A CN 102663020A
- Authority
- CN
- China
- Prior art keywords
- data
- database
- extraction
- extracts
- time window
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention relates to a CDC (changed data capture) data distribution method and a CDC data distribution device, wherein the method comprises the following steps: step (1) of configuring extraction information, configuring the extraction information for extracting data in a database through a configuration interface; step (2) of extracting process, reading the configured extraction information, extracting the data in a source database from the database of a system related to businesses to generate a text file; step (3) of configuring loading information, configuring the loading information for loading the data in the database through the configuration interface; step (4) of loading process, reading the loading information, and loading the text file derived during the extraction process into a target database for storing the extracted text file. According to the CDC data distribution method and device provided by the invention, a configuration mode is graphically and flexibly opened, data is extracted fast, data extraction is fully implemented according to source data API, extraction and overloading are performed by the way of pipelining operation.
Description
Technical field
The present invention relates to the data warehouse field, particularly the data integration in data warehouse field.
Background technology
CDC data distribution centring system is to do a product of data integration specially to the data warehouse field, and it is the ELT pattern, rather than the ETL pattern.ELT extracts earlier to load the last conversion of cleaning again; ETL extracts earlier to clean last the loading again, most of still ETL pattern in this field at present, and IS/BI-CDC data distribution centring system is the ELT pattern, through quick extraction and loading data, thereby in data warehouse, carries out data-switching work.It mainly is extraction and the loading that is used for doing data, and extraction is to export to text to data from the database of operation system; Loading is to import to the text that extracts to clean conversion process in the data warehouse.
It is the JDBC through database that prior art extracts data, and the ODBC interface extracts, and extraction efficiency is not high; Processing logic is complicated, and architecture is huge, light weight not, installation and maintenance trouble; Oracle database can only extract by individual process, and extraction speed is not high; Do not support distributed extraction and loading.
Summary of the invention
Technical matters to be solved by this invention provide a kind of graphical open flexibly data pick-up fast, fully based on source data API realize, architectural framework is simple, support distributed parallel to extract the CDC data distributing method that loads.
The technical scheme that the present invention solves the problems of the technologies described above is following: a kind of CDC data distributing method, and it may further comprise the steps:
1. dispose extraction information: the extraction information that is used for extracted data storehouse data through the configuration interface configuration;
2. extract process: read the extraction information of configuration, from the database of the system relevant, extract the generation text to the data in the source database with business;
3. dispose hosting Information: the hosting Information that is used for loading data storehouse data through the configuration interface configuration;
4. load process: read said hosting Information, and 1. dispose extraction information to the target database that the text that the extraction process derives is loaded into the text that is used for depositing extraction: the needed extraction information of configuration extraction process;
On the basis of technique scheme, the present invention can also do following improvement.
Further, said extraction process comprises that Mysql extracts, Sql server extracts, greenplum extracts, Oracle extracts, db2 extracts and/or group extracts.
Further, wherein said, said Mysql extracts and may further comprise the steps:
1. Mysql extracts process and initiates a connection to the Mysql database; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that Mysql connects, if 3. in the time window scope; Connect the Mysql database; Utilize the Mysql api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said Sql server extracts and may further comprise the steps:
1. Sql server extracts process and initiates a connection to Sql server database; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that Sql server connects, if 3. in the time window scope; Connect Sql server database; Utilize the freetds api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said greenplum extracts and may further comprise the steps:
1. greenplum extracts process and initiates a connection to greenplum distributed data warehouse; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that greenplum connects, if 3. in the time window scope; Connect the greenplum database; Utilize the copy command interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said Oracle extracts and may further comprise the steps:
1. Oracle extracts process and initiates a connection to oracle database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that Oracle connects; If 3. in the time window scope, connect oracle database, through the data recorded block address; The extraction process extracts a data source table; Data derive to generate text the most at last, if 4. not in the time window scope, directly finish extraction work;
Said db2 extracts and may further comprise the steps:
1. db2 extracts process and initiates a connection to the db2 database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that db2 connects; If 3. in the time window scope; Connect the db2 database, in db2 multi partition data, open the subregion extraction process of a plurality of correspondences automatically at all subregions, distributed derived data generates text; If 4. not in the time window scope, directly finish extraction work;
Said group of extraction may further comprise the steps:
1. organize the identical or connection of data of different types storehouse initiation of extraction process, 2. read the extraction information of configuration, judge whether system time extracts in the time window that connects at that time under group extracts to two or more; If 3. in the time window scope, connect corresponding database, after group extracts all down extraction completion; Whole group is extracted end; Otherwise be failure,, directly finish extraction work if 4. not in the time window scope.
Further, said loading data process comprises that Data Loading, db2 load, Oracle loads and/or greenplum loads.
Further, said when in carrying out said loading process, reading the hosting Information of configuration, the extraction information according to correspondence is loaded into target database to the text that extracts, if load failure, reads load configurations information again and loads.
Further, said Data Loading may further comprise the steps:
1. load process Mysql database and initiate a connection; 2. read the hosting Information of configuration, judge that system time at that time is whether in loading the time window that connects, if 3. in the time window scope; The load data that calls the Mysql database loads interface; Be loaded into file in the Mysql database,, directly finish extraction work if 4. not in the time window scope;
Said db2 loads and may further comprise the steps:
1. the db2 process of loading is initiated a connection to the db2 database; 2. read the hosting Information of configuration, judge that system time at that time is whether in db2 loads the time window that connects, if 3. in the time window scope; Call db2 and load interface; Be loaded into file in the db2 database,, directly finish extraction work if 4. not in the time window scope;
Said Oracle loads and may further comprise the steps:
1. the Oracle process of loading is initiated a connection to oracle database; 2. read the hosting Information of configuration, judge that system time at that time is whether in oracle database loads the time window that connects, if 3. in the time window scope; Call Oracle and load interface; Load interface interchange sqlloader interface at Oracle and be loaded into text in the oracle database,, directly finish extraction work if 4. not in the time window scope;
Said greenplum loads and may further comprise the steps:
1. the greenplum process of loading is initiated a connection to the greenplum database; 2. read the hosting Information of configuration; Judge system time at that time whether in greenplum loads the time window that connects,, call greenplum and load interface and connect the greenplum database if 3. in the time window scope; 4. after connecting the greenplum database; The loading process is created external table and object table in the greenplum database, the process of 5. loading is carried out insert into select operation in greenplum distributed data warehouse, be loaded into file in the greenplum distributed data warehouse; If 6. not in the time window scope, directly finish extraction work.
In addition, the present invention also provides a kind of CDC data delivery device, and this device comprises configuration extraction information module, abstraction module, and configuration hosting Information module, the loading data module, wherein:
Information module is extracted in configuration, is used for being used for through the configuration interface configuration extraction information of extracted data storehouse data;
Abstraction module is used to read the extraction information of configuration, from the database of the system relevant with business, extracts the generation text to the data in the source database;
Configuration hosting Information module is used for disposing the hosting Information that is used for loading data storehouse data through configuration interface;
Loading module is used to read said hosting Information, and is loaded into the target database of the text that is used for depositing extraction to the text that the extraction process derives
Further, said abstraction module comprises that Mysql extracts submodule, Sql server extracts submodule, greenplum extraction submodule, Oracle extraction submodule, db2 extraction submodule and/or group and extracts submodule.
Further, said Mysql extracts submodule, is used for exporting to text to the SQL statement of Mysql data of database through appointment to data from the Mysql database;
Said Sql server extracts submodule, is used for data integration to the data warehouse platform;
Said greenplum extracts submodule, be used for coming out the data pick-up in greenplum distributed data warehouse to Sql server type of database, with data distribution in other applied environment;
Said Oracle extracts submodule, is used for utilizing the extraction process that oracle database Oracle data source table is extracted;
Said db2 extracts submodule, is used for to db2 multi partition data, from the parallel derived data of feasible multi partition database;
Said group is extracted submodule; Be used for the data of a table database from two or more similar and different types; But only be loaded into the situation in the table in the object library; Extract the data of database of two or more similar and different types, generate two or more texts.
Further, said loading module comprises that Data Loading submodule, db2 load submodule, Oracle loads submodule and/or greenplum loads submodule.
Further, operation in the said loading module read the hosting Information of configuration the time, the extraction information according to correspondence is loaded into object library to the text that extracts, and fails if load, and reads load configurations information again and loads.
Further, said Data Loading submodule is used for after exporting to text to Data Warehouse, loads interface routine through Mysql, is distributed to text in the Mysql database;
Said db2 loads submodule, and being used for to the target data warehouse is the situation of db2 database, is loaded into each source data unification in the db2 database through this interface;
Said Oracle loads submodule, and being used for to the target data warehouse is the situation that oracle database perhaps will be loaded into Data Warehouse the Oracle background data base of other application, loads interface through Oracle and accomplishes the loading process;
Said greenplum loads submodule, is used for greenplum distributed data warehouse environment, all imports to data the data of various data sources in the greenplum distributed data warehouse through extracting interface through the greenplum data-interface.
The beneficial effect that adopts such scheme is to extract, load based on the configuration mode operation, can open to the outside world fully, can extract by multi-threaded parallel for the oracle database simultaneously, improves extraction efficiency.Based on patterned configuration interface, operation is loaded in the extraction of pipeline system.
Support the loading of greenplum data, can effectively utilize the high-performance calculation ability of greenplum database to come deal with data.
Description of drawings
Fig. 1 is overall flow figure of the present invention;
Fig. 2 extracts process flow diagram for Mysql of the present invention;
Fig. 3 extracts process flow diagram for Sql server of the present invention;
Fig. 4 extracts process flow diagram for greenplum of the present invention;
Fig. 5 extracts process flow diagram for Oracle of the present invention;
Fig. 6 extracts process flow diagram for db2 of the present invention;
Fig. 7 extracts process flow diagram for of the present invention group;
Fig. 8 is a Data Loading process flow diagram of the present invention;
Fig. 9 loads process flow diagram for db2 of the present invention;
Figure 10 loads process flow diagram for Oracle of the present invention;
Figure 11 loads process flow diagram for greenplum of the present invention;
Embodiment
Below in conjunction with accompanying drawing principle of the present invention and characteristic are described, institute gives an actual example and only is used to explain the present invention, is not to be used to limit scope of the present invention.
Embodiment 1
As shown in Figure 1, whole implementation process of the present invention may further comprise the steps:
1. dispose extraction information: dispose the extraction information that is used for extracted data storehouse data through configuration interface, information configured comprises: the table that extract, document storage catalogue after the extraction and filename form; Checking file storing directory and checking file name format, data file remember history, the SQL of extraction; Extract preposition SQL, extraction time, empty data closing times; Whether launch operation main frame, configure host;
2. extract process: read the extraction information of configuration, from the database of the system relevant with business, extract the generation text to the data in the source database, said and professional relevant system is: like ERP system; Financial system, business support system, OA system; EBS, logistics system, site shopping platform; Customer service system, wireless WAP platform or the like.；
3. dispose hosting Information: dispose the hosting Information that is used for loading data storehouse data through configuration interface, the information configured of comprises: the pairing extraction of Load Number is numbered, and is loaded into the stem of target database; Port numbers is loaded the start time, the number of times of reloading; Whether launch the target table name of loading, the target database connection name of loading; Clear table SQL, rearmounted SQL loads list of fields;
4. load process: read said hosting Information, and be loaded into the target database of the text that is used for depositing extraction to the text that the extraction process derives.
Said extraction process comprises that Mysql extracts, Sql server extracts, greenplum extracts, Oracle extracts, db2 extracts and/or group extracts.
Said loading data process comprises that Data Loading, db2 load, Oracle loads and/or greenplum loads.
Said when in carrying out said loading process, reading the hosting Information of configuration, the extraction information according to correspondence is loaded into target database to the text that extracts, if load failure, reads load configurations information again and loads.
Fig. 2 is the practical implementation step that said Mysql extracts:
1. Mysql extracts process and initiates a connection to the Mysql database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that Mysql connects; Said time window be meant sometime the section in, process can connect associated databases, if 3. in the time window scope; Connect the Mysql database; Utilize the Mysql api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope.
Fig. 3 be said Sql server extract practical implementation the time step:
1. Sql server extracts process and initiates a connection to Sql server database; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that Sql server connects, if 3. in the time window scope; Connect Sql server database; Utilize the freetds api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Fig. 4 is the said greenplum step when extracting practical implementation:
1. greenplum extracts process and initiates a connection to greenplum distributed data warehouse; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that greenplum connects, if 3. in the time window scope; Connect the greenplum database; Utilize the copy command interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Fig. 5 is the said Oracle step when extracting practical implementation:
1. Oracle extracts process and initiates a connection to oracle database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that Oracle connects; If 3. in the time window scope, connect oracle database, through the data recorded block address; The extraction process extracts a data source table; Data derive to generate text the most at last, if 4. not in the time window scope, directly finish extraction work;
Fig. 6 is the said db2 step when extracting practical implementation:
1. db2 extracts process and initiates a connection to the db2 database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that db2 connects; If 3. in the time window scope; Connect the db2 database, in db2 multi partition data, open the subregion extraction process of a plurality of correspondences automatically at all subregions, distributed derived data generates text; If 4. not in the time window scope, directly finish extraction work;
Step when Fig. 7 is said group of extraction practical implementation:
1. organize the identical or connection of data of different types storehouse initiation of extraction process, 2. read the extraction information of configuration, judge whether system time extracts in the time window that connects at that time under group extracts to two or more; If 3. in the time window scope, connect corresponding database, after group extracts all down extraction completion; Whole group is extracted end; Otherwise be failure,, directly finish extraction work if 4. not in the time window scope.
Step when Fig. 8 is said Data Loading practical implementation:
1. load process Mysql database and initiate a connection; 2. read the hosting Information of configuration, judge that system time at that time is whether in loading the time window that connects, if 3. in the time window scope; The load data that calls the Mysql database loads interface; Be loaded into file in the Mysql database,, directly finish extraction work if 4. not in the time window scope;
Fig. 9 is the practical implementation step that said db2 loads:
1. the db2 process of loading is initiated a connection to the db2 database; 2. read the hosting Information of configuration, judge that system time at that time is whether in db2 loads the time window that connects, if 3. in the time window scope; Call db2 and load interface; Be loaded into file in the db2 database,, directly finish extraction work if 4. not in the time window scope;
Figure 10 is the said Oracle step when loading practical implementation:
1. the Oracle process of loading is initiated a connection to oracle database; 2. read the hosting Information of configuration, judge that system time at that time is whether in oracle database loads the time window that connects, if 3. in the time window scope; Call Oracle and load interface; Load interface interchange sql loader interface at Oracle and be loaded into text in the oracle database,, directly finish extraction work if 4. not in the time window scope;
Figure 11 is the said greenplum step when loading practical implementation:
1. the greenplum process of loading is initiated a connection to the greenplum database; 2. read the hosting Information of configuration; Judge system time at that time whether in greenplum loads the time window that connects,, call greenplum and load interface and connect the greenplum database if 3. in the time window scope; 4. after connecting the greenplum database; The loading process is created external table and object table in the greenplum database, the process of 5. loading is carried out insert into select operation in greenplum distributed data warehouse, be loaded into file in the greenplum distributed data warehouse; If 6. not in the time window scope, directly finish extraction work.
The above is merely preferred embodiment of the present invention, and is in order to restriction the present invention, not all within spirit of the present invention and principle, any modification of being done, is equal to replacement, improvement etc., all should be included within protection scope of the present invention.
Claims (12)
1. CDC data distributing method is characterized in that it may further comprise the steps:
1. dispose extraction information: the extraction information that is used for extracted data storehouse data through the configuration interface configuration;
2. extract process: read the extraction information of configuration, from the database of the system relevant, extract the generation text to the data in the source database with business;
3. dispose hosting Information: the hosting Information that is used for loading data storehouse data through the configuration interface configuration;
4. load process: read said hosting Information, and be loaded into the target database of the text that is used for depositing extraction to the text that the extraction process derives.
2. a kind of CDC data distributing method according to claim 1 is characterized in that: said extraction process comprises that Mysql extracts, Sql server extracts, greenplum extracts, Oracle extracts, db2 extracts and/or group extracts.
3. a kind of CDC data distributing method according to claim 2 is characterized in that: wherein, said Mysql extracts and may further comprise the steps:
1. Mysql extracts process and initiates a connection to the Mysql database; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that Mysql connects, if 3. in the time window scope; Connect the Mysql database; Utilize the Mysql api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said Sql server extracts and may further comprise the steps:
1. Sql server extracts process and initiates a connection to Sql server database; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that Sql server connects, if 3. in the time window scope; Connect Sql server database; Utilize the freetds api interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said greenplum extracts and may further comprise the steps:
1. greenplum extracts process and initiates a connection to greenplum distributed data warehouse; 2. read the extraction information of configuration, judge that system time at that time is whether in the time window that greenplum connects, if 3. in the time window scope; Connect the greenplum database; Utilize the copy command interface that data derive are generated text,, directly finish extraction work if 4. not in the time window scope;
Said Oracle extracts and may further comprise the steps:
1. Oracle extracts process and initiates a connection to oracle database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that Oracle connects; If 3. in the time window scope, connect oracle database, through the data recorded block address; The extraction process extracts a data source table; Data derive to generate text the most at last, if 4. not in the time window scope, directly finish extraction work;
Said db2 extracts and may further comprise the steps:
1. db2 extracts process and initiates a connection to the db2 database, 2. reads the extraction information of configuration, judges that at that time system time is whether in the time window that db2 connects; If 3. in the time window scope; Connect the db2 database, in db2 multi partition data, open the subregion extraction process of a plurality of correspondences automatically at all subregions, distributed derived data generates text; If 4. not in the time window scope, directly finish extraction work;
Said group of extraction may further comprise the steps:
1. organize the identical or connection of data of different types storehouse initiation of extraction process, 2. read the extraction information of configuration, judge whether system time extracts in the time window that connects at that time under group extracts to two or more; If 3. in the time window scope, connect corresponding database, after group extracts all down extraction completion; Whole group is extracted end; Otherwise be failure,, directly finish extraction work if 4. not in the time window scope.
4. a kind of CDC data distributing method according to claim 1 is characterized in that: said loading data process comprises that Data Loading, db2 load, Oracle loads and/or greenplum loads.
5. a kind of CDC data distributing method according to claim 1; It is characterized in that: when in carrying out said loading process, reading the hosting Information of configuration; Extraction information according to correspondence; Be loaded into target database to the text that extracts,, read load configurations information again and load again if load failure.
6. a kind of CDC data distributing method according to claim 5, it is characterized in that: said Data Loading may further comprise the steps:
1. load process Mysql database and initiate a connection; 2. read the hosting Information of configuration, judge that system time at that time is whether in loading the time window that connects, if 3. in the time window scope; The load data that calls the Mysql database loads interface; Be loaded into file in the Mysql database,, directly finish extraction work if 4. not in the time window scope;
Said db2 loads and may further comprise the steps:
1. the db2 process of loading is initiated a connection to the db2 database; 2. read the hosting Information of configuration, judge that system time at that time is whether in db2 loads the time window that connects, if 3. in the time window scope; Call db2 and load interface; Be loaded into file in the db2 database,, directly finish extraction work if 4. not in the time window scope;
Said Oracle loads and may further comprise the steps:
1. the Oracle process of loading is initiated a connection to oracle database; 2. read the hosting Information of configuration, judge that system time at that time is whether in oracle database loads the time window that connects, if 3. in the time window scope; Call Oracle and load interface; Load interface interchange sql loader interface at Oracle and be loaded into text in the oracle database,, directly finish extraction work if 4. not in the time window scope;
Said greenplum loads and may further comprise the steps:
1. the greenplum process of loading is initiated a connection to the greenplum database; 2. read the hosting Information of configuration; Judge system time at that time whether in greenplum loads the time window that connects,, call greenplum and load interface and connect the greenplum database if 3. in the time window scope; 4. after connecting the greenplum database; The loading process is created external table and object table in the greenplum database, the process of 5. loading is carried out insert into select operation in greenplum distributed data warehouse, be loaded into file in the greenplum distributed data warehouse; If 6. not in the time window scope, directly finish extraction work.
7. a CDC data delivery device is characterized in that, this device comprises configuration extraction information module, abstraction module, and configuration hosting Information module, the loading data module, wherein:
Information module is extracted in configuration, is used for being used for through the configuration interface configuration extraction information of extracted data storehouse data;
Abstraction module is used to read the extraction information of configuration, from the database of the system relevant with business, extracts the generation text to the data in the source database;
Configuration hosting Information module is used for disposing the hosting Information that is used for loading data storehouse data through configuration interface;
Loading module is used to read said hosting Information, and is loaded into the target database of the text that is used for depositing extraction to the text that the extraction process derives.
8. a kind of CDC data delivery device according to claim 7 is characterized in that: said abstraction module comprises that Mysql extracts submodule, Sql server extracts submodule, greenplum extraction submodule, Oracle extraction submodule, db2 extraction submodule and/or group and extracts submodule.
9. a kind of CDC data delivery device according to claim 8 is characterized in that:
Said Mysql extracts submodule, is used for exporting to text to the SQL statement of Mysql data of database through appointment to data from the Mysql database;
Said Sql server extracts submodule, is used for data integration to the data warehouse platform;
Said greenplum extracts submodule, be used for coming out the data pick-up in greenplum distributed data warehouse to Sql server type of database, with data distribution in other applied environment;
Said Oracle extracts submodule, is used for utilizing the extraction process that oracle database Oracle data source table is extracted;
Said db2 extracts submodule, is used for to db2 multi partition data, from the parallel derived data of feasible multi partition database;
Said group is extracted submodule; Be used for the data of a table database from two or more similar and different types; But only be loaded into the situation in the table in the object library; Extract the data of database of two or more similar and different types, generate two or more texts.
10. a kind of CDC data delivery device according to claim 7 is characterized in that: said loading module comprises that Data Loading submodule, db2 load submodule, Oracle loads submodule and/or greenplum loads submodule.
11. a kind of CDC data delivery device according to claim 7; It is characterized in that: operation in the said loading module read the hosting Information of configuration the time; Extraction information according to correspondence; Be loaded into object library to the text that extracts,, read load configurations information again and load again if load failure.
12. a kind of CDC data delivery device according to claim 10 is characterized in that:
Said Data Loading submodule is used for after exporting to text to Data Warehouse, loads interface routine through Mysql, is distributed to text in the Mysql database;
Said db2 loads submodule, and being used for to the target data warehouse is the situation of db2 database, is loaded into each source data unification in the db2 database through this interface;
Said Oracle loads submodule, and being used for to the target data warehouse is the situation that oracle database perhaps will be loaded into Data Warehouse the Oracle background data base of other application, loads interface through Oracle and accomplishes the loading process;
Said greenplum loads submodule, is used for greenplum distributed data warehouse environment, all imports to data the data of various data sources in the greenplum distributed data warehouse through extracting interface through the greenplum data-interface.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100769289A CN102663020A (en) | 2012-03-21 | 2012-03-21 | CDC data distribution method and device thereof |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012100769289A CN102663020A (en) | 2012-03-21 | 2012-03-21 | CDC data distribution method and device thereof |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102663020A true CN102663020A (en) | 2012-09-12 |
Family
ID=46772511
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012100769289A Pending CN102663020A (en) | 2012-03-21 | 2012-03-21 | CDC data distribution method and device thereof |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102663020A (en) |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102737A (en) * | 2014-07-28 | 2014-10-15 | 中国农业银行股份有限公司 | Historical data storage method and system |
| CN105760485A (en) * | 2016-02-17 | 2016-07-13 | 上海携程商务有限公司 | Financial data extraction method and system |
| CN108255855A (en) * | 2016-12-29 | 2018-07-06 | 北京国双科技有限公司 | Date storage method and device |
| CN108268542A (en) * | 2016-12-31 | 2018-07-10 | 中国移动通信集团河北有限公司 | For the method and system of data-base cluster Data Migration |
| CN108446145A (en) * | 2018-03-21 | 2018-08-24 | 苏州提点信息科技有限公司 | A kind of distributed document loads MPP data base methods automatically |
| CN110019446A (en) * | 2017-09-12 | 2019-07-16 | 上海酷服信息科技有限公司 | ETL data processing system and method |
| CN110032559A (en) * | 2019-04-19 | 2019-07-19 | 成都四方伟业软件股份有限公司 | A kind of data pick-up method and device |
| CN110275913A (en) * | 2019-04-25 | 2019-09-24 | 深圳壹账通智能科技有限公司 | Data furnishing method, device and storage medium and electronic device |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1674018A (en) * | 2004-08-13 | 2005-09-28 | 上海宝信软件股份有限公司 | Data distributing central system and data exchanging method |
| CN101882165A (en) * | 2010-08-02 | 2010-11-10 | 山东中创软件工程股份有限公司 | Multithreading data processing method based on ETL (Extract Transform Loading) |
-
2012
- 2012-03-21 CN CN2012100769289A patent/CN102663020A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1674018A (en) * | 2004-08-13 | 2005-09-28 | 上海宝信软件股份有限公司 | Data distributing central system and data exchanging method |
| CN101882165A (en) * | 2010-08-02 | 2010-11-10 | 山东中创软件工程股份有限公司 | Multithreading data processing method based on ETL (Extract Transform Loading) |
Non-Patent Citations (2)
| Title |
|---|
| 赵俊: "etl在数据中心中的设计与实现", 《中国优秀硕士学位论文全文数据库》, 31 August 2011 (2011-08-31) * |
| 黄怀毅等: "一种轻量级架构的ETL系统设计与实现", 《计算机技术与发展》, vol. 18, no. 6, 30 June 2008 (2008-06-30) * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104102737A (en) * | 2014-07-28 | 2014-10-15 | 中国农业银行股份有限公司 | Historical data storage method and system |
| CN104102737B (en) * | 2014-07-28 | 2018-01-30 | 中国农业银行股份有限公司 | A kind of historical data storage method and system |
| CN105760485A (en) * | 2016-02-17 | 2016-07-13 | 上海携程商务有限公司 | Financial data extraction method and system |
| CN108255855A (en) * | 2016-12-29 | 2018-07-06 | 北京国双科技有限公司 | Date storage method and device |
| CN108255855B (en) * | 2016-12-29 | 2021-10-08 | 北京国双科技有限公司 | Data storage method and device |
| CN108268542A (en) * | 2016-12-31 | 2018-07-10 | 中国移动通信集团河北有限公司 | For the method and system of data-base cluster Data Migration |
| CN110019446A (en) * | 2017-09-12 | 2019-07-16 | 上海酷服信息科技有限公司 | ETL data processing system and method |
| CN108446145A (en) * | 2018-03-21 | 2018-08-24 | 苏州提点信息科技有限公司 | A kind of distributed document loads MPP data base methods automatically |
| CN110032559A (en) * | 2019-04-19 | 2019-07-19 | 成都四方伟业软件股份有限公司 | A kind of data pick-up method and device |
| CN110275913A (en) * | 2019-04-25 | 2019-09-24 | 深圳壹账通智能科技有限公司 | Data furnishing method, device and storage medium and electronic device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102663020A (en) | CDC data distribution method and device thereof | |
| CN102270225B (en) | Data change daily record method for supervising and data change daily record supervising device | |
| CN102752372A (en) | File based database synchronization method | |
| CN104102737B (en) | A kind of historical data storage method and system | |
| CN102236672A (en) | Method and device for importing data | |
| CN102375891A (en) | Implementation tool for unloading and loading incremental data | |
| CN103425551B (en) | Management method in database distributed backup set | |
| CN106126601A (en) | A kind of social security distributed preprocess method of big data and system | |
| CN104699723A (en) | Data exchange adapter and system and method for synchronizing data among heterogeneous systems | |
| CN103218402A (en) | General database data structure, data migratory system and method thereof | |
| RU2008126117A (en) | METHODS FOR LONG-TERM STORAGE OF A LOT OF SIMULTANEOUSLY WORKING WORK FLOWS | |
| CN103092980A (en) | Method and system of data automatic conversion and storage | |
| CN102810070A (en) | High-performance professional ability packaging process engine and process control method thereof | |
| CN105589968A (en) | Data summarization system and method | |
| CN105956123A (en) | Local updating software-based data processing method and apparatus | |
| CN105653554A (en) | File data comparison method and system | |
| CN104778175A (en) | Method and system for realizing data synchronization of heterogeneous database | |
| CN103309977B (en) | Heterogeneous data resource integration method | |
| CN107301214A (en) | Data migration method, device and terminal device in HIVE | |
| CN102184190A (en) | Data comparison method | |
| CN105740462A (en) | Method for supporting data migration between different environments | |
| CN104834860A (en) | Dynamic warehousing method for security events | |
| CN102467525A (en) | Document associating method and system | |
| CN104239580A (en) | General single-field split data extraction method and device based on value-column mapping | |
| CN103258047B (en) | A kind of data organization method towards medicine enterprise Activity-Based Cost Control data warehouse |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20160608 |
|
| C20 | Patent right or utility model deemed to be abandoned or is abandoned |