CN113946632A

CN113946632A - Agile data warehouse architecture and construction method and application thereof

Info

Publication number: CN113946632A
Application number: CN202111323604.6A
Authority: CN
Inventors: 王洋; 吴振刚; 丁毅
Original assignee: China Building Materials Xinyun Zhilian Technology Co ltd; Cnbm Technology Corp ltd
Current assignee: China Building Materials Xinyun Zhilian Technology Co ltd; Cnbm Technology Corp ltd
Priority date: 2021-11-10
Filing date: 2021-11-10
Publication date: 2022-01-18

Abstract

The invention relates to the technical field of data warehouses, in particular to an agile data warehouse architecture and a construction method and application thereof. The data warehouse system comprises a data source, a storage layer, a data warehouse, an analysis tool and an application layer, wherein the data warehouse mainly comprises two items, namely a data warehouse model and a data warehouse tool, the data warehouse model is used for solving the problem that data are available, and the data warehouse tool is used for solving the problem that the data are easy to use. The framework designed by the invention can realize the consistency of the data calibers and determine the meaning and the correctness of the data by reconciling the data from different sources; the construction method of the system is used for realizing the compact combination of concept analysis, logic design and physics, reasonably designing a data storage and management mode, recombining data, strengthening and optimizing the process of quality control of a data warehouse and improving the practicability of the warehouse counting; the application of the method can provide efficient and detailed data, is suitable for a large number of application scenes and users, meets the data requirements of different levels, and is suitable for the continuous change process of business and products of enterprises.

Description

Agile data warehouse architecture and construction method and application thereof

Technical Field

The invention relates to the technical field of data warehouses, in particular to an agile data warehouse architecture and a construction method and application thereof.

Background

A data warehouse is a strategic collection that provides all types of data support for all levels of decision making processes of an enterprise. The data warehouse contains not only the data required for analysis but also the applications required for processing the data, including applications that transfer data from external media to the data warehouse, and applications that analyze and present the data to the user. Data warehouses have five basic characteristics of theme-oriented, integrated, stable, time-varying, and support for management decisions. With the rapid development of the internet, the business and products of enterprises are undergoing a continuous rapid development process, and how a data warehouse adapts to changes becomes one of the problems of restricting development; a decision maker pursues refined operation, and requires a data warehouse to have the capability of providing high-efficiency detailed data, and how the data warehouse meets the problem that the development of data requirements of different levels is restricted; data finally reach the hands of a data user after ETL, the requirements for extracting data and proposing data are from different departments and different purposes, the data apertures are inconsistent, the data meaning is fuzzy, even the data correctness is difficult to check, and how to ensure the data aperture consistency and the data path traceability of a data warehouse becomes the third problem which restricts the development. However, there is no architecture, construction method and application of agile data warehouse that can solve the above problems.

Disclosure of Invention

The invention aims to provide an agile data warehouse architecture, a construction method and application thereof, so as to solve the problems in the background technology. To achieve the above technical problem, one of the objects of the present invention is to provide an agile data warehouse architecture, comprising: a data source: the method mainly comprises two sources of a traditional database DB and a log cluster file, and is used for acquiring operation type and application related service data: a storage layer: the ETL system mainly comprises a computing engine, a development tool, a platform and other data warehouse tools and is used for solving the ETL problem, realizing stable and correct transmission of data and providing a reliable storage computing environment; a data warehouse: the system mainly comprises two items, namely a data warehouse model and a data warehouse tool, wherein the data warehouse model is used for solving the problem of data availability, and the data warehouse tool is used for solving the problem of data usability; an analysis tool: data analysis is mainly carried out through an interest-bearing query tool, a multi-dimensional analysis tool, a search analysis tool, a report system and other tools, and the problem of how users with different roles use a data warehouse is solved; an application layer: and the method is used for analyzing the result according to the requirements of the user and applying the constructed data warehouse architecture to different service scenes and users. As a further improvement of the present technical solution, in the data warehouse: the data warehouse model mainly comprises a near source data layer, a data wide table and a basic index layer; the near source data layer is a packaging intermediate layer and is used for merging different service data, shielding dirty data, merging redundant fields and the like, and the near source data layer basically keeps a field structure consistent with a data source, and the method specifically comprises the following steps: a traditional database DB in a data source is imported into a data warehouse according to the structure of the service, and a log file is mapped into a Hive table to the data warehouse according to fields; the data wide table is used for extracting enough frequently-used fields from the near-source data layer; the basic index layer is used for extracting an intuitive basic index table from the wide table; the data warehouse tool mainly comprises but is not limited to a data dictionary, a data map, authority management and the like. As a further improvement of the technical solution, the near-source data layer mainly includes a reconciliation data layer, a derivative data layer, a global data warehouse, and metadata, and specifically includes: and (3) reconciling the data layers: forming a reconciliation layer by the data belonging to different data sources, and putting all the data into a global data warehouse for consistent reconciliation treatment; derivative data layer: filtering and summarizing the harmonic data into each data mart for decision support to form derivative data, wherein the data mart is used for managing data, modes and applications for each department for decision support; global data warehouse: for managing data, schema, and applications related to reconciling data; metadata: for providing relevant information about data sources, modes and interactions. The situation that the same question is answered differently can be avoided by reconciling all the data. The second objective of the present invention is to provide a method for constructing an agile data warehouse architecture, which is used for constructing the agile data warehouse architecture, and comprises the following steps: s1, concept analysis: analyzing requirements are carried out by taking data as a center, and a data storage and management mode is designed according to clear/potential analysis requirements; s2, logic design: according to the business model of the enterprise and the basic theory of the data warehouse, the data in the source system is recombined and designed, and the design work of the data warehouse is completed; s3, physical implementation: a developer extracts data from a source database by means of a certain tool or a certain programming language to realize the tasks of the data warehouse and the client database described by the logic design; s4, data warehouse quality control: and respectively strengthening and optimizing the control of the quality of the data warehouse according to the quality categories of the data warehouse divided by the user. In S1, the explicit analysis requirement (or called daily analysis requirement) refers to information that is frequently queried by business decision makers of clients in daily decision-making, and generally, the explicit requirement description can be given by summarizing and summarizing historical management activities of clients, and the final result is expressed as support for reporting tray; the potential analysis requirement refers to the requirement that a client cannot clearly explain in advance and professionals conduct regular exploration and knowledge discovery on historical data of enterprises. As a further improvement of the present technical solution, in S1, the specific method for concept analysis includes the following steps: s1.1, knowing the business process of an enterprise and the reaction thereof on data, including but not limited to various data, tables, classification rules and the like related to various businesses of the enterprise; s1.2, learning the original IT system of an enterprise and the owned data thereof, and determining the corresponding relation between the data stream and the service stream; and S1.3, respectively knowing the analysis requirements of the end users according to the end user categories of the data warehouse. In S1.2, the correspondence between the data stream and the service stream is determined, so that the foundation for understanding the source database is also the guarantee of data accuracy and consistency. In S1.3, the terminal users of the general data warehouse may be divided into two types, namely, report users and professional analysis users; usually, the requirements of report users are very clear, and report samples can be provided accurately; the requirements of the two professional analysis users are fuzzy. As a further improvement of the present technical solution, in S2, the specific method for logic design includes the following steps: s2.1, application system function design: including but not limited to the design and planning of system functions, the design of input and output interfaces of specific functions, the implementation logic of specific input and output interfaces, etc.; s2.2, designing a data warehouse: including but not limited to design of data warehouse storage structures, dimensional modeling, determination of data granularity, source data table planning, and the like; s2.3, ETL design: the system related personnel determine the logic and updating frequency of data extraction according to the understanding of the data in the original system and the design of a data warehouse, and the analysis system selects the required data from the data warehouse according to the set logic and carries out necessary aggregation to establish a data mart for a specific analysis application, and the like. In S2.3, ETL refers to a process of data extraction, conversion, cleaning, and loading, ETL is an important loop for constructing a data warehouse, and a user extracts required data from a data source, and loads the data into the data warehouse according to a predefined data warehouse model after data cleaning. As a further improvement of the technical solution, in S3, the specific method for physical implementation includes the following steps: s3.1, selecting a development tool and a platform according to the requirements of the project and the characteristics of tool software; and S3.2, developing a program specific to a specific application by a developer through a development tool and a platform so as to realize main projects of the data warehouse. In S3.1, the principles that can be referred to when selecting among a plurality of pieces of tool software meeting the requirements include, but are not limited to: software familiar to developers is selected as much as possible, software with high compatibility is selected as much as possible, and development tools of a data warehouse preferably have high analysis and processing capacity. As a further improvement of the technical solution, in S3.2, the main items and programs of the data warehouse are mainly classified into five categories, including: ETL procedure: a process for completing the reading of data from a data source into a data warehouse: a data processing program: a process for performing the necessary data aggregation and modeling; a data interface program: for providing an interface for other systems or subsequent applications to access the data repository: the application program comprises the following steps: for completing existing application functions in the detailed design; metadata management program: for performing metadata generation and access functions. As a further improvement of the technical solution, in S4, the specific method for controlling the quality of the data warehouse includes the following steps: s4.1, dividing the quality of a data warehouse into four categories of design and management quality, application quality, data use quality and data quality according to the identity of a user; s4.2, starting from concept analysis, sorting data logic based on the business process of an enterprise, constructing a data model, and ensuring the design and management quality of a data warehouse; s4.3, controlling and improving the application quality of the data warehouse by using a scientific software project management method through reasonable division of labor in projects, reasonable design of development flows and rich programming experience; s4.4, by constructing an operable project prototype basically having a data warehouse structure through a prototype development method, the project prototype is used for finding that detailed data and business understanding are not enough to be adjusted and modified in time, so that a basic report and analysis function is realized, customers can feed back requirements and check effects, and the use quality of the data is continuously improved; and S4.5, effectively utilizing metadata management, and improving the data quality through operations such as classification, storage, maintenance, updating, integration, exchange and the like of the metadata. In S4.1, the design and management quality of the data warehouse may be divided into two or more quality dimensions, one of which is a data model and a data quality dimension, and represents the capability of the data model and the data to effectively represent the real information; the second is quality metadata evolution dimension, namely the evolution condition of a data model in the using process of a data warehouse; the data model and data quality dimension can be further divided into aspects of correctness, completeness, interpretability of the tool, redundancy minimization, traceability and the like. The application quality of the data warehouse is mainly directed at software on the physical layer of the data warehouse and mainly represents the functionality (suitability, correctness, coupling, consistency and safety), reliability (maturity, fault tolerance and recoverability), applicability (intelligibility, learnability and operability), software effectiveness (time complexity and resource complexity), maintainability (analyzability, variability, stability and testability), portability (adaptability, installability, consistency and replaceability) of an application program. The quality of use of data warehouse data includes accessibility (system availability, transaction availability, and data security) and data usefulness (responsiveness, data interpretability, and data timeliness), among others. The data quality refers to the attribute of the data, is a core problem of the data warehouse quality, is different from the use quality of the data, and mainly comprises the integrity, the credibility, the correctness, the consistency and the uniqueness of the data. In S4.5, the metadata defines the role of the data warehouse, indicates the content and location of the total information of the data warehouse, describes the extraction and transformation rules of the data, accesses the subject and related information of the data warehouse, monitors the whole process of data flow in the data warehouse, and is an important basis for managing data of the data warehouse; the main purpose of metadata management is to reduce the workload of data warehouse system management to the maximum extent and improve the data extraction quality of the data warehouse to the maximum extent. The third objective of the present invention is to provide an application of an agile data warehouse architecture, including the agile data warehouse architecture constructed by the above-mentioned construction method of the agile data warehouse architecture, the data warehouse has incomparable lateral expansion and iterative computation capability of other data platforms, and can directly or indirectly provide data services for users, including: service scene: enterprise information management, product operation, financial wind control, digital engineering construction, communication industry management, information retrieval and the like; the user: data analysts, enterprise decision makers, and the like. The fourth objective of the present invention is to provide an apparatus for constructing an agile data warehouse architecture, which includes a processor, a memory, and a computer program stored in the memory and running on the processor, where the processor is configured to implement the steps of the agile data warehouse architecture, the constructing method thereof, and the application when executing the computer program. It is a fifth object of the present invention to provide a computer-readable storage medium, which stores a computer program, and the computer program realizes the steps of the agile data warehouse architecture, the construction method thereof and the application thereof when being executed by a processor. Compared with the prior art, the invention has the beneficial effects that: the agile data warehouse architecture is characterized in that a data warehouse architecture consisting of a data warehouse model and a data warehouse tool is designed, data from different sources are harmonized, derived data are extracted from harmonized data, all data are placed in a global data warehouse for consistency harmonization processing, the same problem can be effectively avoided from being answered differently, the consistent calibers of the data are realized, the meaning and the correctness of the data are determined, and the data are convenient to trace; the construction method of the agile data warehouse architecture starts from a data warehouse system structure with three layers of concept, logic and physics, adopts a data warehouse construction method which closely combines concept analysis, logic design and physics, quickly, comprehensively and perfectly constructs the agile data warehouse, reasonably designs a data storage and management mode, recombines data, and respectively strengthens and optimizes the quality control process of the data warehouse according to the quality category of the data warehouse, thereby improving the practicability of the data warehouse; the application of the agile data warehouse architecture can provide efficient and detailed data by constructing the agile and high-quality data warehouse, so that the agile data warehouse architecture can be suitable for a large number of application scenes and users, meets the data requirements of different levels, and is suitable for the continuous change process of business and products of enterprises. Description of the drawingsfigure 1 is an overall architectural block diagram of the present invention; FIG. 2 is a block diagram of a partial architecture of the present invention; FIG. 3 is a flow chart of the overall construction method of the present invention; FIG. 4 is a flow chart of a partial construction method according to the present invention; FIG. 5 is a second flowchart of a partial construction method according to the present invention; FIG. 6 is a third flowchart of a partial construction method according to the present invention; FIG. 7 is a fourth flowchart of the local building method of the present invention. Detailed description of the inventiontechnical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention. Embodiment 1 as shown in fig. 1-2, this embodiment provides an agile data warehouse architecture, including: a data source: the method mainly comprises two sources of a traditional database DB and a log cluster file, and is used for acquiring operation type and application related service data: a storage layer: the ETL system mainly comprises a computing engine, a development tool, a platform and other data warehouse tools and is used for solving the ETL problem, realizing stable and correct transmission of data and providing a reliable storage computing environment; a data warehouse: the system mainly comprises two items, namely a data warehouse model and a data warehouse tool, wherein the data warehouse model is used for solving the problem of data availability, and the data warehouse tool is used for solving the problem of data usability; an analysis tool: data analysis is mainly carried out through an interest-bearing query tool, a multi-dimensional analysis tool, a search analysis tool, a report system and other tools, and the problem of how users with different roles use a data warehouse is solved; an application layer: and the method is used for analyzing the result according to the requirements of the user and applying the constructed data warehouse architecture to different service scenes and users. In this embodiment, in the data warehouse: the data warehouse model mainly comprises a near source data layer, a data wide table and a basic index layer; the near source data layer is a packaging intermediate layer and is used for merging different service data, shielding dirty data, merging redundant fields and the like, and the near source data layer basically keeps a field structure consistent with a data source, and the method specifically comprises the following steps: a traditional database DB in a data source is imported into a data warehouse according to the structure of the service, and a log file is mapped into a Hive table to the data warehouse according to fields; the data wide table is used for extracting enough frequently-used fields from the near-source data layer; the basic index layer is used for extracting an intuitive basic index table from the wide table; the data warehouse tool mainly comprises but is not limited to a data dictionary, a data map, authority management and the like. In this embodiment, the near-source data layer mainly includes a reconciliation data layer, a derivative data layer, a global data warehouse, and metadata, and specifically includes: and (3) reconciling the data layers: forming a reconciliation layer by the data belonging to different data sources, and putting all the data into a global data warehouse for consistent reconciliation treatment; derivative data layer: filtering and summarizing the harmonic data into each data mart for decision support to form derivative data, wherein the data mart is used for managing data, modes and applications for each department for decision support; global data warehouse: for managing data, schema, and applications related to reconciling data; metadata: for providing relevant information about data sources, modes and interactions. The situation that the same question is answered differently can be avoided by reconciling all the data. As shown in fig. 3 to fig. 7, the present embodiment provides a method for constructing an agile data warehouse architecture, which is used for constructing the agile data warehouse architecture, and includes the following steps: s1, concept analysis: analyzing requirements are carried out by taking data as a center, and a data storage and management mode is designed according to clear/potential analysis requirements; s2, logic design: according to the business model of the enterprise and the basic theory of the data warehouse, the data in the source system is recombined and designed, and the design work of the data warehouse is completed; s3, physical implementation: a developer extracts data from a source database by means of a certain tool or a certain programming language to realize the tasks of the data warehouse and the client database described by the logic design; s4, data warehouse quality control: and respectively strengthening and optimizing the control of the quality of the data warehouse according to the quality categories of the data warehouse divided by the user. In S1, the explicit analysis requirement (or called daily analysis requirement) refers to information that is frequently queried by business decision makers of clients for daily decision-making, and generally, the explicit requirement description can be given by summarizing and summarizing historical management activities of clients, and the final result is expressed as support for the tray of the report; the potential analysis requirement refers to the requirement that a client cannot clearly explain in advance and professionals conduct regular exploration and knowledge discovery on historical data of enterprises. In this embodiment, in S1, the specific method for concept analysis includes the following steps: s1.1, knowing the business process of an enterprise and the reaction thereof on data, including but not limited to various data, tables, classification rules and the like related to various businesses of the enterprise; s1.2, learning the original IT system of an enterprise and the owned data thereof, and determining the corresponding relation between the data stream and the service stream; and S1.3, respectively knowing the analysis requirements of the end users according to the end user categories of the data warehouse. In S1.2, the correspondence between the data stream and the service stream is determined, so that the foundation of understanding the source database is also the guarantee of data accuracy and consistency. In S1.3, terminal users of a general data warehouse can be divided into two types, namely report users and professional analysis users; usually, the requirements of report users are very clear, and report samples can be provided accurately; the requirements of the two professional analysis users are fuzzy. In this embodiment, in S2, the specific method for logic design includes the following steps: s2.1, application system function design: including but not limited to the design and planning of system functions, the design of input and output interfaces of specific functions, the implementation logic of specific input and output interfaces, etc.; s2.2, designing a data warehouse: including but not limited to design of data warehouse storage structures, dimensional modeling, determination of data granularity, source data table planning, and the like; s2.3, ETL design: the system related personnel determine the logic and updating frequency of data extraction according to the understanding of the data in the original system and the design of a data warehouse, and the analysis system selects the required data from the data warehouse according to the set logic and carries out necessary aggregation to establish a data mart for a specific analysis application, and the like. In S2.3, ETL refers to the processes of data extraction, conversion, cleaning and loading, ETL is an important ring for constructing a data warehouse, and a user extracts required data from a data source, and finally loads the data into the data warehouse according to a predefined data warehouse model after data cleaning. In this embodiment, in S3, the specific method for physical implementation includes the following steps: s3.1, selecting a development tool and a platform according to the requirements of the project and the characteristics of tool software; and S3.2, developing a program specific to a specific application by a developer through a development tool and a platform so as to realize main projects of the data warehouse. In S3.1, the principles that can be referred to when selecting among a plurality of pieces of tool software meeting the requirements include, but are not limited to: software familiar to developers is selected as much as possible, software with high compatibility is selected as much as possible, and development tools of a data warehouse preferably have high analysis and processing capacity. Specifically, in S3.2, the main items and programs of the data warehouse are mainly classified into five categories, including: ETL procedure: a process for completing the reading of data from a data source into a data warehouse: a data processing program: a process for performing the necessary data aggregation and modeling; a data interface program: for providing an interface for other systems or subsequent applications to access the data repository: the application program comprises the following steps: for completing existing application functions in the detailed design; metadata management program: for performing metadata generation and access functions. In this embodiment, in S4, the specific method for controlling quality of the data warehouse includes the following steps: s4.1, dividing the quality of a data warehouse into four categories of design and management quality, application quality, data use quality and data quality according to the identity of a user; s4.2, starting from concept analysis, sorting data logic based on the business process of an enterprise, constructing a data model, and ensuring the design and management quality of a data warehouse; s4.3, controlling and improving the application quality of the data warehouse by using a scientific software project management method through reasonable division of labor in projects, reasonable design of development flows and rich programming experience; s4.4, by constructing an operable project prototype basically having a data warehouse structure through a prototype development method, the project prototype is used for finding that detailed data and business understanding are not enough to be adjusted and modified in time, so that a basic report and analysis function is realized, customers can feed back requirements and check effects, and the use quality of the data is continuously improved; and S4.5, effectively utilizing metadata management, and improving the data quality through operations such as classification, storage, maintenance, updating, integration, exchange and the like of the metadata. In S4.1, the design and management quality of the data warehouse can be divided into two types and a plurality of quality dimensions, wherein one type is a data model and a data quality dimension and represents the capability of the data model and the data to effectively express the real information; the second is quality metadata evolution dimension, namely the evolution condition of a data model in the using process of a data warehouse; the data model and data quality dimension can be further divided into aspects of correctness, completeness, interpretability of the tool, redundancy minimization, traceability and the like. The application quality of the data warehouse is mainly directed at software on the physical layer of the data warehouse and mainly represents the functionality (suitability, correctness, coupling, consistency and safety), reliability (maturity, fault tolerance and recoverability), applicability (intelligibility, learnability and operability), software effectiveness (time complexity and resource complexity), maintainability (analyzability, variability, stability and testability), portability (adaptability, installability, consistency and replaceability) of an application program. The quality of use of data warehouse data includes accessibility (system availability, transaction availability, and data security) and data usefulness (responsiveness, data interpretability, and data timeliness), among others. The data quality refers to the attribute of the data, is a core problem of the data warehouse quality, is different from the use quality of the data, and mainly comprises the integrity, the credibility, the correctness, the consistency and the uniqueness of the data. In S4.5, metadata defines the function of the data warehouse, indicates the content and the position of the total information of the data warehouse, describes the extraction and conversion rules of data, accesses the subject and the related information of the data warehouse, monitors the whole process of data flow in the data warehouse, and is an important basis for managing data of the data warehouse; the main purpose of metadata management is to reduce the workload of data warehouse system management to the maximum extent and improve the data extraction quality of the data warehouse to the maximum extent. The embodiment also provides an application of the agile data warehouse architecture, including the agile data warehouse architecture constructed by the construction method of the agile data warehouse architecture, the data warehouse has incomparable lateral extension and iterative computing capability of other data platforms, and can directly or indirectly provide data services for users, including: service scene: enterprise information management, product operation, financial wind control, digital engineering construction, communication industry management, information retrieval and the like; the user: data analysts, enterprise decision makers, and the like. The present embodiments also provide an apparatus for building an agile data warehouse architecture comprising a processor, a memory, and a computer program stored in the memory and running on the processor. The processor comprises one or more processing cores, the processor is connected with the memory through the bus, the memory is used for storing program instructions, and the agile data warehouse architecture, the construction method and the application thereof are realized when the processor executes the program instructions in the memory. Alternatively, the memory may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks. In addition, the present invention also provides a computer readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method implements the agile data warehouse architecture, and the steps of the method and the application for constructing the agile data warehouse architecture. Optionally, the present invention also provides a computer program product containing instructions which, when run on a computer, cause the computer to perform the steps of the above-described aspects of agile data warehouse architecture, method of construction and applications thereof. It will be understood by those skilled in the art that all or part of the steps of implementing the above embodiments may be implemented by hardware, or may be implemented by hardware related to instructions of a program, which may be stored in a computer-readable storage medium, such as a read-only memory, a magnetic or optical disk, and the like. The foregoing shows and describes the general principles, essential features, and advantages of the invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, and the preferred embodiments of the present invention are described in the above embodiments and the description, and are not intended to limit the present invention. The scope of the invention is defined by the appended claims and equivalents thereof.

Claims

1. An agile data warehouse architecture, comprising: the method comprises the following steps: a data source: the method mainly comprises two sources of a traditional database DB and a log cluster file, and is used for acquiring operation type and application related service data: a storage layer: the ETL system mainly comprises a computing engine, a development tool, a platform and other data warehouse tools and is used for solving the ETL problem, realizing stable and correct transmission of data and providing a reliable storage computing environment; a data warehouse: the system mainly comprises two items, namely a data warehouse model and a data warehouse tool, wherein the data warehouse model is used for solving the problem of data availability, and the data warehouse tool is used for solving the problem of data usability; an analysis tool: data analysis is mainly carried out through an interest-bearing query tool, a multi-dimensional analysis tool, a search analysis tool, a report system and other tools, and the problem of how users with different roles use a data warehouse is solved; an application layer: and the method is used for analyzing the result according to the requirements of the user and applying the constructed data warehouse architecture to different service scenes and users.

2. The agile data repository architecture of claim 1, wherein: in the data repository: the data warehouse model mainly comprises a near source data layer, a data wide table and a basic index layer; the near source data layer is a packaging intermediate layer and is used for merging different service data, shielding dirty data, merging redundant fields and the like, and the near source data layer basically keeps a field structure consistent with a data source, and specifically comprises the following steps: a traditional database DB in a data source is imported into a data warehouse according to the structure of the service, and a log file is mapped into a Hive table to the data warehouse according to fields; the data wide table is used for extracting enough frequently-used fields from the near-source data layer; the basic index layer is used for extracting an intuitive basic index table from the wide table; the data warehouse tool mainly comprises but is not limited to a data dictionary, a data map, authority management and the like.

3. The agile data repository architecture of claim 2, wherein: the near-source data layer mainly comprises a harmonic data layer, a derivative data layer, a global data warehouse and metadata, and specifically comprises the following steps: and (3) reconciling the data layers: forming a reconciliation layer by the data belonging to different data sources, and putting all the data into a global data warehouse for consistent reconciliation treatment; the derivative data layer of claim 2: filtering and summarizing the harmonic data into each data mart for decision support to form derivative data, wherein the data mart is used for managing data, modes and applications for each department for decision support; global data warehouse: for managing data, schema, and applications related to reconciling data; metadata: for providing relevant information about data sources, modes and interactions.

4. A method of constructing an agile data warehouse schema for constructing an agile data warehouse schema as claimed in claim 3, wherein: the method comprises the following steps: s1, concept analysis: analyzing requirements are carried out by taking data as a center, and a data storage and management mode is designed according to clear/potential analysis requirements; s2, logic design: according to the business model of the enterprise and the basic theory of the data warehouse, the data in the source system is recombined and designed, and the design work of the data warehouse is completed; s3, physical implementation: a developer extracts data from a source database by means of a certain tool or a certain programming language to realize the tasks of the data warehouse and the client database described by the logic design; s4, data warehouse quality control: and respectively strengthening and optimizing the control of the quality of the data warehouse according to the quality categories of the data warehouse divided by the user.

5. The method of building an agile data warehouse architecture according to claim 4, wherein: in S1, the method for concept analysis includes the following steps: s1.1, knowing the business process of an enterprise and the reaction thereof on data, including but not limited to various data, tables, classification rules and the like related to various businesses of the enterprise; s1.2, learning the original IT system of an enterprise and the owned data thereof, and determining the corresponding relation between the data stream and the service stream; and S1.3, respectively knowing the analysis requirements of the end users according to the end user categories of the data warehouse.

6. The method of building an agile data warehouse architecture according to claim 5, wherein: in S2, the specific method of logic design includes the following steps: s2.1, application system function design: including but not limited to the design and planning of system functions, the design of input/output interfaces of claim 3, the implementation logic of specific input/output interfaces, etc.; s2.2, designing a data warehouse: including but not limited to design of data warehouse storage structures, dimensional modeling, determination of data granularity, source data table planning, and the like; s2.3, ETL design: the system related personnel determine the logic and updating frequency of data extraction according to the understanding of the data in the original system and the design of a data warehouse, and the analysis system selects the required data from the data warehouse according to the set logic and carries out necessary aggregation to establish a data mart for a specific analysis application, and the like.

7. The method of building an agile data warehouse architecture according to claim 6, wherein: in S3, the specific method for physical implementation includes the following steps: s3.1, selecting a development tool and a platform according to the requirements of the project and the characteristics of tool software; and S3.2, developing a program specific to a specific application by a developer through a development tool and a platform so as to realize main projects of the data warehouse.

8. The method of building an agile data warehouse architecture according to claim 7, wherein: in S3.2, the main items and programs of the data warehouse are mainly divided into five categories, including: ETL procedure: a process for completing the reading of data from a data source into a data warehouse: a data processing program: a process for performing the necessary data aggregation and modeling; a data interface program: for providing an interface for other systems or subsequent applications to access the data repository: the application program comprises the following steps: for completing existing application functions in the detailed design; metadata management program: for performing metadata generation and access functions.

9. The method of building an agile data warehouse architecture according to claim 7, wherein: in S4, the method for controlling quality of a data warehouse includes the following steps: s4.1, dividing the quality of the data warehouse into four categories of design and management quality, application quality, data use quality and data quality according to the identity of a user; s4.2, starting from concept analysis, sorting data logic based on the business process of an enterprise, constructing a data model, and ensuring the design and management quality of a data warehouse; claim 4S 4.3, using scientific software project management method, controlling and improving the application quality of the data warehouse by reasonable division of labor, reasonable design of development flow and rich programming experience in the project; s4.4, by constructing an operable project prototype basically having a data warehouse structure through a prototype development method, the project prototype is used for finding that detailed data and business understanding are insufficient to adjust and modify in time, so that basic report forms and analysis functions are realized, customers can feed back requirements and check effects, and the use quality of data is continuously improved; and S4.5, effectively utilizing metadata management, and improving the data quality through operations such as classification, storage, maintenance, updating, integration, exchange and the like of the metadata.

10. An application of agile data warehouse architecture, comprising the agile data warehouse architecture constructed by the construction method of agile data warehouse architecture of claim 9, characterized in that: the data warehouse has incomparable lateral expansion and iterative computing capacity compared with other data platforms, and can directly or indirectly provide data services for users, and the data services comprise: service scene: enterprise information management, product operation, financial management, digital engineering construction, communication industry management, information retrieval and the like; the user: data analysts, enterprise decision makers, and the like.