CN109376143A - A kind of design method of data warehouse of effective agility - Google Patents
A kind of design method of data warehouse of effective agility Download PDFInfo
- Publication number
- CN109376143A CN109376143A CN201811090917.XA CN201811090917A CN109376143A CN 109376143 A CN109376143 A CN 109376143A CN 201811090917 A CN201811090917 A CN 201811090917A CN 109376143 A CN109376143 A CN 109376143A
- Authority
- CN
- China
- Prior art keywords
- component
- layer
- data
- etl
- data warehouse
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention provides a kind of design method of data warehouse of effective agility, comprising: basal layer, abstraction, layer, integrated four conversion layer, specially treated layer level.The knowledge and experience accumulated in the work of can making full use of over of the software reuse technology of Component- Based Development technology, the recognized component with relatively independent function is applied to the exploitation of new system, during guaranteeing new system exploitation, emphasis can be concentrated on to identification and realize the distinctive constituent of application system, it is final to shorten system development cycle, the quality of raising system, it is the configuration of the present invention is simple, easy to use, practical.
Description
Technical field
The invention belongs to computer field, in particular to a kind of design method of data warehouse of effective agility.
Background technique
Data warehouse is the inevitable outcome of computer and database application development to certain phase, is to support business decision point
The core technology of analysis.The purpose of data warehouse is to establish a kind of data storage environment of architecture, will be needed for analysis decision
Mass data separated from traditional operating environment, be converted to dispersion, inconsistent operation data integrated, unified
Information.But the transacter that developed under different times, different background in traditional operating environment, enterprise, this
The foundation of a little systems also tend to be completed towards different applications, by different developers, the storage organization of data, storage
Platform and system platform have very big isomerism.How the data of these isomeries to be effectively integrated in data warehouse, is
The problem that developer is faced.Developer needs a comprehensive solution to solve the consistency of data and integrated
Problem enables people to acquire data from all traditional platform and environment, and using a single solution to it
It is efficiently converted, this solution is exactly data pick-up, conversion and loading procedure ETL.
ETL is data to be extracted from various isomery manipulation type data sources, and carry out conversion process to the data being drawn into, most
The process being loaded into data warehouse afterwards.It is the steps necessary established the foundation stone of data warehouse, and establish data warehouse,
Occupy in the process of construction of data warehouse and its consequence.From the point of view of whole angle, the main function of ETL is its shielding
Complicated service logic, to for various analyses based on data warehouse and apply and provide unified data-interface.It can be with
It says, ETL is erected between traditional operation system and data warehouse and played a bridge block, it is ensured that new data can be continually
Ground enters data warehouse.Fudan University's Master's thesis in 21012 " research and design of General ETL Tool " devises one and leads to
With the design scheme and system architecture of ETL tool, it supports a variety of isomeric data platforms, in conversion links, provides a large amount of thin
The transition components of granularity complete complicated affairs in such a way that component combines, to support the business demand of multiple fields.However
Its ETL carries out conversion or quality testing to the data of memory are loaded into one by one, and transfer efficiency is relatively low, when being transferred to other necks
When domain, it is also difficult to handle in face of new service logic.
Then both at home and abroad for how class is similar or similar data warehouse project in share the research of ETL process compared with
It is few, largely hinder further increasing for data warehouse project construction efficiency.
Summary of the invention
Background technique there are aiming at the problem that, the present invention provides a kind of design method of data warehouse of effective agility.
In order to solve the above technical problems, the present invention adopts the following technical scheme:
A kind of design method of data warehouse of effective agility, comprising: basal layer, abstraction, layer, integrated conversion layer, specially treated
Four level of layer；
The basal layer includes metadata management component, interlayer interface component, the automatic test member three of KPI Key Performance Indicator
Class；
The metadata management component mainly completes the function of metadata management module in ETL subsystem；
The interlayer interface component provides an identical ETL processing block from abstract level for each data warehouse project
Frame provides interface for the various building blocks of function of each level of ETL treatment process, realizes component concrete processing procedure to the transparent of framework
Change；
The automatic test member of KPI Key Performance Indicator provides an automatic test member for every class KPI Key Performance Indicator；
The abstraction, layer component is located at the bottom of ETL framework, directly facing data source, completes the work in data pick-up stage
Make, the ETL component of this layer is widely different between different data warehouse, and reusable degree is generally relatively low；
The data conversion that the integrated conversion layer component mainly extracts abstraction, layer is at format specification, meaning unification, quality
Good data, and be integrated into data warehouse；The integrated conversion layer provides a kind of ETL for every class data object and handles structure
Part, it is relatively independent between same layer component, by being abstracted each set of metadata of similar data repository entry business rule, it is encapsulated in component
It is interior, when guaranteeing that ETL framework is transplanted between set of metadata of similar data warehouse, as long as ETL component, which can be put into, to be made by configuration service rule
With；
On the basis of integrated conversion layer component is handled, flowing water will be pressed by being responsible in data warehouse trades the specially treated layer
The data reduction of form tissue is at the form for pressing KPI Key Performance Indicator tissue.
Detailed description of the invention
Fig. 1 the structural representation of present invention.
Specific embodiment
The invention will be further described for embodiment shown in reference to the accompanying drawing.
As shown in Fig. 1, the present invention includes basal layer (1), abstraction, layer (2), integrated conversion layer (3), specially treated layer
(4)；The basal layer (1) includes: metadata management component (1-1)；Interlayer interface component (1-2)；KPI Key Performance Indicator is automatic
Test member (1-3)；Metadata management component (1-1) mainly completes the function of metadata management module in ETL subsystem；Interlayer
Interface component (1-2) provides an identical ETL from abstract level for each data warehouse project and handles frame, handles for ETL
The various building blocks of function of each level of process provide interface, realize component concrete processing procedure to the transparence of framework；Key Performance
Index automatic test component (1-3) provides an automatic test member for every class KPI Key Performance Indicator；Abstraction, layer component (2) is located at
The bottom of ETL framework completes the work in data pick-up stage, the ETL component of this layer is in different data directly facing data source
Widely different between warehouse, reusable degree is generally relatively low；
The data conversion that integrated conversion layer component (3) mainly extract abstraction, layer is good at format specification, meaning unification, quality
Good data, and be integrated into data warehouse；The integrated conversion layer provides a kind of ETL for every class data object and handles component,
It is relatively independent between same layer component, by being abstracted each set of metadata of similar data repository entry business rule, it is encapsulated in component, protects
When card ETL framework is transplanted between set of metadata of similar data warehouse, as long as by configuration service rule, ETL component can come into operation；
Different process layer (4) on the basis of integrated conversion layer component is handled is responsible for that flowing water transaction shape will be pressed in data warehouse
The data reduction of formula tissue is at the form for pressing KPI Key Performance Indicator tissue.
Embodiment 2.
A kind of design method of data warehouse of effective agility includes: metadata extraction module, Reusable Components selection and leads
Enter the automatic maintenance module of module, data warehouse schema, ETL process definition module, ETL scheduler module, ETL component generation module；
Metadata extraction module is mainly completed: being extracted business datum and dimension data metadata, and is carried out more to system on this basis
Precise definition；The selection of Reusable Components and import modul extract encapsulated full flowing water transaction data from component base and take out
Component, integrated translation building block, KPI translation building block, dimension class data integration translation building block, the automatic test class component of KPI are taken, by it
It imported into ETL procedure library；
The automatic maintenance module of data warehouse schema completes data bins according to the information in metadatabase, for statistical analysis system
True table, dimension table creation are completed in the creation and initial work in library, complete allocation list, middle table that each component needs and
The work such as the foundation of interim table；
ETL process definition module can visually define the ETL process for not having Reusable Components in component base；
ETL scheduler module can be arranged according to the scheduling of system, execute the ETL process in ETL procedure library, realize data
It extracts, conversion, load, conversion；
Reusable Components generation module extracts corresponding ETL treatment process from ETL procedure library and is packaged into Reusable Components.It is real
It applies and tests the effect of the invention with protrusion in example there are also comparative example.
Protection scope of the present invention is not limited to the above embodiments, it is clear that those skilled in the art can be to the present invention
Various changes and deformation are carried out without departing from scope and spirit of the present invention.If these changes and deformation belong to right of the present invention
It is required that and its equivalent technologies range, then the intent of the present invention also include these change and deformation including.
Claims (4)
1. a kind of design method of data warehouse of effective agility, it is characterised in that: including basal layer, abstraction, layer, integrated conversion layer,
Specially treated layer；The basal layer includes metadata management component, interlayer interface component, the automatic test member of KPI Key Performance Indicator
Three classes；The metadata management component mainly completes the function of metadata management module in ETL subsystem；The interlayer interface structure
Part provides an identical ETL from abstract level for each data warehouse project and handles frame, is each level of ETL treatment process
Various building blocks of function provide interface, realize component concrete processing procedure to the transparence of framework；The KPI Key Performance Indicator is automatic
Test member provides an automatic test member for every class KPI Key Performance Indicator.
2. a kind of design method of data warehouse of effective agility according to claim 1, it is characterised in that: the abstraction, layer
Component is located at the bottom of ETL framework, directly facing data source, completes the work in data pick-up stage, the ETL component of this layer exists
Widely different between different data warehouse, reusable degree is generally relatively low.
3. a kind of design method of data warehouse of effective agility according to claim 1, it is characterised in that: described integrated turn
Layer component is changed mainly by the data conversion of abstraction, layer extraction at format specification, meaning unification, the second best in quality data, and is integrated into
In data warehouse, the integrated conversion layer provides a kind of ETL for every class data object and handles component, between same layer component relatively solely
It is vertical, by being abstracted each set of metadata of similar data repository entry business rule, it is encapsulated in component.
4. a kind of design method of data warehouse of effective agility according to claim 1, it is characterised in that: the special place
Layer is managed on the basis of integrated conversion layer component is handled, is responsible for that the data reduction of flowing water transaction form tissue will be pressed in data warehouse
At the form for pressing KPI Key Performance Indicator tissue.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811090917.XA CN109376143A (en) | 2018-09-19 | 2018-09-19 | A kind of design method of data warehouse of effective agility |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811090917.XA CN109376143A (en) | 2018-09-19 | 2018-09-19 | A kind of design method of data warehouse of effective agility |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109376143A true CN109376143A (en) | 2019-02-22 |
Family
ID=65405639
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811090917.XA Pending CN109376143A (en) | 2018-09-19 | 2018-09-19 | A kind of design method of data warehouse of effective agility |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109376143A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110297820A (en) * | 2019-06-28 | 2019-10-01 | 京东数字科技控股有限公司 | A kind of data processing method, device, equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6167405A (en) * | 1998-04-27 | 2000-12-26 | Bull Hn Information Systems Inc. | Method and apparatus for automatically populating a data warehouse system |
| US6366905B1 (en) * | 1999-06-22 | 2002-04-02 | Microsoft Corporation | Aggregations design in database services |
| CN101452485A (en) * | 2008-12-31 | 2009-06-10 | 中国建设银行股份有限公司 | Method and device for generating multidimensional cubic based on relational database |
| CN101477572A (en) * | 2009-01-12 | 2009-07-08 | 深圳市里王智通软件有限公司 | Method and system of dynamic data base based on TDS transition data storage technology |
| CN103488631A (en) * | 2012-06-11 | 2014-01-01 | 上海博路信息技术有限公司 | Construction method for data warehouse based on reusable components |
-
2018
- 2018-09-19 CN CN201811090917.XA patent/CN109376143A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US6167405A (en) * | 1998-04-27 | 2000-12-26 | Bull Hn Information Systems Inc. | Method and apparatus for automatically populating a data warehouse system |
| US6366905B1 (en) * | 1999-06-22 | 2002-04-02 | Microsoft Corporation | Aggregations design in database services |
| CN101452485A (en) * | 2008-12-31 | 2009-06-10 | 中国建设银行股份有限公司 | Method and device for generating multidimensional cubic based on relational database |
| CN101477572A (en) * | 2009-01-12 | 2009-07-08 | 深圳市里王智通软件有限公司 | Method and system of dynamic data base based on TDS transition data storage technology |
| CN103488631A (en) * | 2012-06-11 | 2014-01-01 | 上海博路信息技术有限公司 | Construction method for data warehouse based on reusable components |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110297820A (en) * | 2019-06-28 | 2019-10-01 | 京东数字科技控股有限公司 | A kind of data processing method, device, equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106354833A (en) | Platform for achieving data management and sharing exchange on basis of B/S framework | |
| US8930918B2 (en) | System and method for SQL performance assurance services | |
| CN111176867B (en) | Data sharing exchange and open application platform | |
| CN101539855A (en) | Service basic software platform | |
| CN101908015A (en) | Device and method for creating test case based on components | |
| CN106055325B (en) | A kind of service construction method that support multisystem is run simultaneously | |
| CN105338045A (en) | Cloud computing resource processing device, method and cloud computing system | |
| CN102508919A (en) | Data processing method and system | |
| Zhang et al. | Towards building a multi‐datacenter infrastructure for massive remote sensing image processing | |
| CN108369675A (en) | Technology for case distribution | |
| CN103218360A (en) | Method of industrial real-time database for realizing dynamic memory management by adopting memory pool technology | |
| CN109150964B (en) | Migratable data management method and service migration method | |
| Henry et al. | Migrating to microservices | |
| CN107977773A (en) | A kind of method for the entry resource amount for managing multiple cloud platforms | |
| CN105718601A (en) | Dynamic business integrating model and application method thereof | |
| US7877355B2 (en) | Job scheduling for automatic movement of multidimensional data between live datacubes | |
| CN102722368B (en) | Plug-in software designing method based on document tree and message pump | |
| CN108133005A (en) | A kind of environmental model analogy method, terminal device and storage medium based on memory database | |
| CN109376143A (en) | A kind of design method of data warehouse of effective agility | |
| CN104461832B (en) | A kind of method and device for monitoring application server resource | |
| CN106802928A (en) | Power network historical data management method and its system | |
| CN116301760B (en) | Application Design System for Software Development | |
| CN103559574A (en) | Method and system for operating workflow | |
| CN103488631A (en) | Construction method for data warehouse based on reusable components | |
| CN106383893A (en) | Time sequence data management method and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20190606 Address after: Room 1801, Floor 18, Building 4, Subject Business Center, 9 Shoubei South Road, Haidian District, Beijing Applicant after: China Building Materials Information Technology Limited by Share Ltd Applicant after: China Building Materials Xinyun Zhilian Technology Co., Ltd. Address before: Room 1801, Floor 18, Building 4, Subject Business Center, 9 Shoubei South Road, Haidian District, Beijing, 100098 Applicant before: China Building Materials Information Technology Limited by Share Ltd |
|
| TA01 | Transfer of patent application right | ||
| CB02 | Change of applicant information |
Address after: Room 01, 2 / F, 101-1-11 / F, building 9, area 2, 186 South Fourth Ring Road West, Fengtai District, Beijing 100160 Applicant after: CNBM TECHNOLOGY Corp.,Ltd. Applicant after: China Building Materials Xinyun Zhilian Technology Co.,Ltd. Address before: Room 1801, Floor 18, Building 4, Subject Business Center, 9 Shoubei South Road, Haidian District, Beijing Applicant before: CNBM TECHNOLOGY Corp.,Ltd. Applicant before: China Building Materials Xinyun Zhilian Technology Co.,Ltd. |
|
| CB02 | Change of applicant information | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20190222 |
|
| WD01 | Invention patent application deemed withdrawn after publication |