CN103488631A - Construction method for data warehouse based on reusable components - Google Patents
Construction method for data warehouse based on reusable components Download PDFInfo
- Publication number
- CN103488631A CN103488631A CN201210188408.7A CN201210188408A CN103488631A CN 103488631 A CN103488631 A CN 103488631A CN 201210188408 A CN201210188408 A CN 201210188408A CN 103488631 A CN103488631 A CN 103488631A
- Authority
- CN
- China
- Prior art keywords
- data warehouse
- data
- layer
- etl
- building method
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
- G06F16/212—Schema design and management with details for data modelling support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/254—Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention discloses a construction method for a data warehouse based on reusable components. According to the construction method, four levels including a basic service layer, an extraction layer, an integrated conversion layer and a special processing layer are included. A software reusing technology based on a component technology can sufficiently utilize knowledge and experiences accumulated in previous work and can apply the recognized components with the relative independent functions to the development of a new system so as to guarantee that key points can be concentrated to recognize and realize special composition components of an application system in the development process of the new system; and finally, the development period of the system is shortened and the quality of the system is improved.
Description
Technical field
The present invention relates to a kind of construction method of data warehouse, refer to especially a kind of Building Method of Data Warehouse based on reusable component.
Background technology
Data warehouse is the inevitable outcome that computing machine and database application develop into certain phase, is the core technology of supporting that business decision is analyzed.The purpose of data warehouse is to set up a kind of data storage environment of architecture, and the mass data that analysis decision is required is separated from traditional operating environment, makes dispersion, inconsistent service data be converted to integrated, unified information.Yet, this process not a duck soup, here so-called traditional operating environment, refer to the transacter that enterprise develops under different times, different background, the foundation of these systems is also often towards different application, completed by different developers, and the storage organization of its data, storage platform and system platform have very large isomerism.How the data of these isomeries being integrated in data warehouse effectively, is the difficult problem that enterprise faces.Enterprise needs a comprehensive solution to solve consistance and the integrated problem of data, make people can be from all traditional platforms and environment image data, and utilize a single solution to change efficiently it, this solution is exactly data pick-up, conversion and loading procedure ETL (Extract Transform Load).
ETL is extracted data from various isomery manipulation type data sources, and the data that are drawn into are carried out to conversion process, finally is loaded into the process in data warehouse.It is foundation stone and the soul of setting up data warehouse, is also the steps necessary of setting up data warehouse, in the process of construction of data warehouse, occupies and consequence.From whole angle, the Main Function of ETL is that it has shielded complicated service logic, thereby provides unified data-interface for the analysis and application of various Data Warehouse--baseds.Can say, ETL frame between traditional operation system and data warehouse has erected a bridge block, guarantees that new data can enter data warehouse continuously.
At present, both at home and abroad for how in class in similar or close data warehouse project the research of sharing E TL process less, hindered to a great extent the further raising of data warehouse project construction efficiency.
?
Summary of the invention
For shortening system development cycle, improve mass of system, the present invention has developed a kind of Building Method of Data Warehouse based on reusable component.
According to the present invention, native system comprises: infrastructure service layer, extraction layer, integrated conversion layer, four level of special processing layer;
Described infrastructure service layer comprises metadata management member, interlayer interface member, automatic test member three classes of KPI Key Performance Indicator;
Described metadata management member mainly completes the function of metadata management module in the ETL subsystem;
Described interlayer interface member is processed framework for each data warehouse project provides an identical ETL abstract aspect, for the various building blocks of functions of each level of ETL processing procedure provide interface, realizes the transparence of the concrete processing procedure of member to framework;
The automatic test member of described KPI Key Performance Indicator provides an automatic test member for every class KPI Key Performance Indicator;
Described extraction layer member is positioned at the bottom of ETL framework, directly, in the face of data source, completes the work in data pick-up stage, and the ETL member of this layer is widely different between the different pieces of information warehouse, and reusable degree is lower generally;
The data-switching that described integrated conversion layer member mainly will extract layer extraction becomes format specification, implication unification, the second best in quality data, and is integrated in data warehouse; Described integrated conversion layer is processed member for every class data object provides a class ETL, with relatively independent between layer member, by abstract each similar data warehouse project business rule, be encapsulated in member, while guaranteeing that the ETL framework is transplanted between similar data warehouse, as long as, by the configuration service rule, the ETL member can come into operation;
On the basis that described special processing layer is processed at integrated conversion layer member, the form that the responsible data reduction that will in data warehouse, press flowing water transaction form tissue becomes to press the KPI Key Performance Indicator tissue.
The accompanying drawing explanation
The schematic diagram that Fig. 1 is framework of the present invention.
Accompanying drawing described herein is used to provide a further understanding of the present invention, forms the application's a part, and schematic description and description of the present invention the present invention does not form inappropriate limitation of the present invention for explaining.
Embodiment
Embodiment 1.
1 couple of the present invention is described more fully with reference to the accompanying drawings, and exemplary embodiment of the present invention wherein is described.
Native system comprises: infrastructure service layer (a), extraction layer (b), an integrated conversion layer (c), four levels of special processing layer (d);
Described infrastructure service layer (a) comprises:
Metadata management member (a1)
Interlayer interface member (a2)
The automatic test member of KPI Key Performance Indicator (a3);
Described metadata management member (a1) mainly completes the function of metadata management module in the ETL subsystem;
Described interlayer interface member (a2) is processed framework for each data warehouse project provides an identical ETL abstract aspect, for the various building blocks of functions of each level of ETL processing procedure provide interface, realizes the transparence of the concrete processing procedure of member to framework;
The automatic test member of described KPI Key Performance Indicator (a3) provides an automatic test member for every class KPI Key Performance Indicator;
Described extraction layer member (b) is positioned at the bottom of ETL framework, directly, in the face of data source, completes the work in data pick-up stage, and the ETL member of this layer is widely different between the different pieces of information warehouse, and reusable degree is lower generally;
The data-switching that described integrated conversion layer member (c) mainly will extract layer extraction becomes format specification, implication unification, the second best in quality data, and is integrated in data warehouse; Described integrated conversion layer is processed member for every class data object provides a class ETL, with relatively independent between layer member, by abstract each similar data warehouse project business rule, be encapsulated in member, while guaranteeing that the ETL framework is transplanted between similar data warehouse, as long as, by the configuration service rule, the ETL member can come into operation;
On the basis that described special processing layer (d) is processed at integrated conversion layer member, the form that the responsible data reduction that will in data warehouse, press flowing water transaction form tissue becomes to press the KPI Key Performance Indicator tissue.
Embodiment 2.
A kind of Building Method of Data Warehouse based on reusable component comprises: module, the automatic maintenance module of data warehouse framework, ETL procedure definition module, ETL scheduler module, ETL member generation module are selected and imported to Metadata Extraction module, Reusable Components;
Described Metadata Extraction module mainly completes: extract business datum and dimension data metadata, and on this basis system is carried out to more precise definition;
The selection of described Reusable Components and importing module extract the full flowing water transaction data encapsulated and extract member, integrated translation building block, KPI translation building block, dimension class data integration translation building block, the automatic test class member of KPI from component base, and it is imported to the ETL procedure library;
The automatic maintenance module of described data warehouse framework is according to the information in metadatabase, for statistical analysis system completes establishment and the initial work of data warehouse, complete fact table, dimension table establishment, complete the work such as foundation of allocation list, middle table and the temporary table of each member needs;
Described ETL procedure definition module can define in component base does not visually have the ETL of Reusable Components process;
The ETL scheduler module can be carried out the ETL process in the ETL procedure library according to the scheduling setting of system, realizes extraction, conversion, loading, the conversion of data;
The Reusable Components generation module extracts corresponding ETL processing procedure and is packaged into Reusable Components from the ETL procedure library.
?description of the invention is in order to provide for the purpose of example and explanation, and is not exhaustively or limit the invention to disclosed form.Many modifications and variations are obvious for the ordinary skill in the art.Selecting and describing embodiment is for better explanation principle of the present invention and practical application, thereby and makes those of ordinary skill in the art can understand the various embodiment with various modifications that the present invention's design is suitable for special-purpose.
Claims (8)
1. the Building Method of Data Warehouse based on reusable component, is characterized in that, comprises 4 levels, is respectively: infrastructure service layer, extraction layer, integrated conversion layer, special processing layer.
2. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, is characterized in that, described infrastructure service layer comprises metadata management member, interlayer interface member, automatic test member three classes of KPI Key Performance Indicator.
3. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, is characterized in that, described metadata management member mainly completes the function of metadata management module in the ETL subsystem.
4. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, it is characterized in that, described interlayer interface member is processed framework for each data warehouse project provides an identical ETL abstract aspect, for the various building blocks of functions of each level of ETL processing procedure provide interface, realize the transparence of the concrete processing procedure of member to framework.
5. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, is characterized in that, the automatic test member of described KPI Key Performance Indicator provides an automatic test member for every class KPI Key Performance Indicator.
6. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, it is characterized in that, described extraction layer member is positioned at the bottom of ETL framework, directly in the face of data source, complete the work in data pick-up stage, the ETL member of this layer is widely different between the different pieces of information warehouse, and reusable degree is lower generally.
7. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, it is characterized in that, the data-switching that described integrated conversion layer member mainly will extract layer extraction becomes format specification, implication unification, the second best in quality data, and be integrated in data warehouse, described integrated conversion layer is processed member for every class data object provides a class ETL, with relatively independent between layer member, by abstract each similar data warehouse project business rule, be encapsulated in member.
8. a kind of Building Method of Data Warehouse based on reusable component as claimed in claim 1, it is characterized in that, on the basis that described special processing layer is processed at integrated conversion layer member, the form that the responsible data reduction that will in data warehouse, press flowing water transaction form tissue becomes to press the KPI Key Performance Indicator tissue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210188408.7A CN103488631A (en) | 2012-06-11 | 2012-06-11 | Construction method for data warehouse based on reusable components |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201210188408.7A CN103488631A (en) | 2012-06-11 | 2012-06-11 | Construction method for data warehouse based on reusable components |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103488631A true CN103488631A (en) | 2014-01-01 |
Family
ID=49828873
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201210188408.7A Pending CN103488631A (en) | 2012-06-11 | 2012-06-11 | Construction method for data warehouse based on reusable components |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103488631A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109376143A (en) * | 2018-09-19 | 2019-02-22 | 中建材信息技术股份有限公司 | A kind of design method of data warehouse of effective agility |
| CN110442562A (en) * | 2019-06-28 | 2019-11-12 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus of building advantage performance data warehouse |
| CN110750259A (en) * | 2018-07-23 | 2020-02-04 | 北京奇虎科技有限公司 | Method and device for treating a component |
-
2012
- 2012-06-11 CN CN201210188408.7A patent/CN103488631A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110750259A (en) * | 2018-07-23 | 2020-02-04 | 北京奇虎科技有限公司 | Method and device for treating a component |
| CN110750259B (en) * | 2018-07-23 | 2024-04-05 | 三六零科技集团有限公司 | Method and device for processing component |
| CN109376143A (en) * | 2018-09-19 | 2019-02-22 | 中建材信息技术股份有限公司 | A kind of design method of data warehouse of effective agility |
| CN110442562A (en) * | 2019-06-28 | 2019-11-12 | 苏州浪潮智能科技有限公司 | A kind of method and apparatus of building advantage performance data warehouse |
| CN110442562B (en) * | 2019-06-28 | 2022-02-18 | 苏州浪潮智能科技有限公司 | Method and device for constructing dominant performance data warehouse |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103218360B (en) | RTDB in Industry Control uses the method that memory pool technique realizes dynamic memory management | |
| CN101017457A (en) | Automatically testing method of computer software | |
| CN105574082A (en) | Storm based stream processing method and system | |
| CN102521024B (en) | Job scheduling method based on bioinformation cloud platform | |
| CN101860752B (en) | Video code stream parallelization method for embedded multi-core system | |
| CN104036365A (en) | Method for constructing enterprise-level data service platform | |
| CN104571026A (en) | Platform and method for establishing whole-process metallurgical manufacturing execution system | |
| CN102222105A (en) | Method for generating real-time statistical report | |
| CN103473642A (en) | Method for rule engine for production dispatching | |
| CN103294599A (en) | Cloud-based method for cross test of embedded software | |
| CN103279416A (en) | Storage software automated testing system and method | |
| CN105718601A (en) | Business dynamic integration model and application method thereof | |
| CN103488631A (en) | Construction method for data warehouse based on reusable components | |
| CN103685564A (en) | Plug-in application ability layer introduced industry application online operation cloud platform architecture | |
| CN106055325A (en) | Establishing method of service for supporting concurrent operation of multiple systems | |
| CN106088598A (en) | A kind of BIM technology is used to carry out the method led the way of model of constructing | |
| CN110515995A (en) | Quickly generate the ETL operational method and device of big data platform | |
| CN103235978A (en) | Disaster monitoring and early warning system and method for establishing disaster monitoring and early warning system | |
| CN104461832B (en) | A kind of method and device for monitoring application server resource | |
| CN105653334B (en) | MIS system rapid development framework based on SAAS mode | |
| CN103914304B (en) | Method for converting different structure type parameters on basis of SAP (service access point) platforms | |
| Damgrave et al. | Rationalizing virtual reality based on manufacturing paradigms | |
| CN109376143A (en) | A kind of design method of data warehouse of effective agility | |
| Xue | A task parallel processing technology for robot process automation | |
| CN103198380A (en) | Method for supporting Saas applications by utilizing workflow engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20140101 |