CN114357088B - Nuclear power industry data warehouse system - Google Patents
Nuclear power industry data warehouse system Download PDFInfo
- Publication number
- CN114357088B CN114357088B CN202111526722.7A CN202111526722A CN114357088B CN 114357088 B CN114357088 B CN 114357088B CN 202111526722 A CN202111526722 A CN 202111526722A CN 114357088 B CN114357088 B CN 114357088B
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- nuclear power
- service
- catalog
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 239000010410 layer Substances 0.000 claims abstract description 110
- 238000012545 processing Methods 0.000 claims abstract description 52
- 238000013507 mapping Methods 0.000 claims abstract description 18
- 238000000034 method Methods 0.000 claims abstract description 13
- 238000012384 transportation and delivery Methods 0.000 claims abstract description 12
- 238000006243 chemical reaction Methods 0.000 claims abstract description 9
- 238000004140 cleaning Methods 0.000 claims abstract description 8
- 239000002344 surface layer Substances 0.000 claims abstract description 6
- 238000013468 resource allocation Methods 0.000 claims abstract description 5
- 238000011161 development Methods 0.000 claims description 57
- 238000004519 manufacturing process Methods 0.000 claims description 55
- 238000013499 data model Methods 0.000 claims description 15
- 238000012360 testing method Methods 0.000 claims description 10
- 230000005012 migration Effects 0.000 claims description 9
- 238000013508 migration Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 6
- 230000001360 synchronised effect Effects 0.000 claims description 6
- 238000009826 distribution Methods 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims description 2
- 238000012795 verification Methods 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 5
- 238000007726 management method Methods 0.000 description 6
- 238000004458 analytical method Methods 0.000 description 4
- 238000013075 data extraction Methods 0.000 description 4
- 230000008676 import Effects 0.000 description 4
- 238000013461 design Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000012432 intermediate storage Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000000975 co-precipitation Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The disclosure belongs to the technical field of nuclear power, and particularly relates to a nuclear power industrial data warehouse system. The disclosed nuclear power industrial data warehouse system includes: the data exchange layer stores metadata related to data resource allocation; the original data layer is used for storing service system source data; the data detail layer carries out data cleaning and format conversion according to field standards, the data standard layer processes and gathers the processed data in the data detail layer according to the data standards in the nuclear power field, and the fields and the attributes related to the service are associated to form unified nuclear power standard data; the data summarizing layer takes the service as a guide, develops a model according to the service requirement, and outputs a data delivery object; the data mart layer performs row-column level processing on the data of the data detail layer, the data standard layer and the data summarization layer and then provides application to the outside; the common dimension surface layer stores common dictionary coding tables and common dimension table mapping information used by each business and is also used for dimension association of other layered data.
Description
Technical Field
The invention belongs to the technical field of nuclear power, and particularly relates to a nuclear power industrial data warehouse system.
Background
The system comprises a China nuclear power large nuclear source platform (DHP) platform, a series of innovative nuclear power industrial applications oriented to a nuclear power full-industry chain, and safe, reliable and efficient operation of an operation unit, wherein the DHP platform is used as a supporting platform and a neural center of digital nuclear power, integrates data of a China nuclear power massive industrial system and equipment, builds an extensible open nuclear power industrial Internet platform, synchronously develops a nuclear power industrial application development ecological system oriented to various scenes and reusability, improves the use efficiency and sharing range of nuclear power plant hardware, service and data, realizes intelligent management and operation optimization of China nuclear power business and resources, and drives a series of innovative nuclear power industrial applications oriented to the nuclear power full-industry chain.
The DHP data platform needs to access the relational data from different service systems of different power plants, and how to process the accessed data becomes a problem to be solved.
Disclosure of Invention
In order to overcome the problems in the related art, a nuclear power industrial data warehouse system is provided.
According to an aspect of the disclosed embodiments, there is provided a nuclear power industrial data warehouse system, the nuclear power industrial data warehouse system comprising:
data exchange layer: for storing metadata related to the data resource allocation;
the original data layer is used for storing the source data of the service system and keeping synchronous updating with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to field standards, and integrating data from different power plants in a service system into corresponding data tables according to service fields and topic field classification;
data standard layer: the method comprises the steps of processing and summarizing processed data in a data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
data summarization layer: the system is used for taking the service as a guide, developing a model according to the service requirement and outputting a data delivery object;
data mart layer: the data processing module is used for performing row-column level processing on the data in the data detail layer, the data standard layer and the data summarization layer and then providing application to the outside;
and the public dimension surface layer is used for storing a public dictionary coding table and public dimension table mapping information used by each service and dimension association of other layered data.
In one possible implementation manner, the nuclear power industrial data warehouse system allocates a development environment catalog and a production environment catalog for users under a metadata catalog level, wherein the development environment catalog and the production environment catalog are respectively provided with the data detail layer, the data standard layer, the data summarization layer, the data mart layer and the public dimension surface layer, and the development environment catalog also comprises a development and processing task catalog and a development application delivery data catalog; the production environment catalogue also comprises a production processing task catalogue and a production application delivery data catalogue; the nuclear power industrial data warehouse system acquires external requirements, extracts data from a source database to an original data layer, completes operation development under a development environment catalog, and issues the data outwards under a production environment catalog after the data passes the test.
In one possible implementation, the development environment catalog is used for running the workflow of data migration, processing and auditing and verifying whether the result data accords with expectations; the production environment catalog is used for migrating the data model and the processing task which pass the verification in the development environment catalog to the production environment catalog and providing services to the outside.
In one possible implementation, the development environment catalog and the production environment catalog are data isolated using different databases.
In one possible implementation, the data warehouse is built based on Hive databases, with different number of bins being layered to correspond to different Hive databases.
In one possible implementation, the metadata directory of the data exchange layer includes:
the data source connection is used for establishing access connection configuration information of various types of data sources according to the characteristics of access data of the service system, the provided access authority and account types, maintaining the connection between the DHP data platform and the service database, the message queue or the server, and storing proxy information configured for data processing and service distribution;
the data source metadata information is used for storing the metadata information in the scanned source system;
and the data warehouse resource is used for storing the connection information stored in the data warehouse and providing storage resources for each layer of physical model and service data floor.
In one possible implementation, the metadata directory of the original data layer includes:
the business system database instance or Schema is used for establishing a corresponding catalog according to the source database instance and Schema of different business systems of different power plants, and establishing a corresponding physical model according to the table information in the source database;
a manually imported database instance or Schema for storing manually imported data from the service system database instance or Schema; and establishing a corresponding catalog according to the database instance of the imported data source and the Schema name, and storing the imported manual land table under the catalog.
In one possible implementation, the data detail layer includes:
data formatting processing layer: the method comprises the steps of respectively carrying out data cleaning, format conversion and null value removal operation on data of different service systems of each power plant in a data exchange layer according to field constraint so as to form formatted data;
data integration layer: and according to the classification of the nuclear power theme zone and the business sub-zone, carrying out data field modeling, and integrating the data of different power plants into a unified data table.
In one possible implementation, the data model of the data summary layer includes one or more combinations of topic tables, multi-level summary tables, application wide tables, and public index libraries.
The beneficial effects of the present disclosure are: the nuclear power industrial data warehouse system disclosed by the invention adopts a mode of layering and dividing the subject domain to construct a data center so as to access the relational data of different service systems from different power plants, and forms data sets of different layers through layering processing, integration and conversion according to the nuclear power data standard and application requirements, so that the data sets are provided for external application, the development and analysis of data are facilitated, the blood edges of the data are tracked, and the reusability of a data model is improved.
Drawings
FIG. 1 is a schematic diagram of a nuclear power industrial data warehouse system, according to an example embodiment.
FIG. 2 is a diagram illustrating a nuclear power industry data warehouse that allocates development environment directories and production environment directories to users at a metadata directory level according to an example embodiment.
Detailed Description
The invention will be described in further detail with reference to the accompanying drawings and specific examples.
The present disclosure provides a nuclear power industry data warehouse system, the nuclear power industry data warehouse system comprising:
data exchange layer: for storing metadata related to the data resource allocation.
The original data layer is used for storing the source data of the service system and keeping synchronous updating with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to field standards, and integrating data from different power plants in a service system into corresponding data tables according to service fields and topic field classification;
data standard layer: the method comprises the steps of processing and summarizing processed data in a data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
data summarization layer: the system is used for taking the service as a guide, developing a model according to the service requirement and outputting a data delivery object;
data mart layer: the method is used for providing application for the outside after row-column level processing of data in DWD (Data Warehouse Detail), STD (Standard) and DWS (Data Warehouse Summary) layers;
a common dimension skin: DIM (Dimension), configured to store mapping information such as a public dictionary coding table and a public dimension table used by each service, and use the mapping information in dimension association of other hierarchical data, so that service personnel can query meaning of a data field conveniently.
In this embodiment, the DHP data warehouse is divided into 7 levels as follows:
1) Data exchange layer: exchange, store metadata related to data resource allocation. The metadata directory of this layer mainly includes:
a) Data source connection: and saving information such as database resources, proxy resources and the like. And establishing access connection configuration information of various types of data sources according to the characteristics of access data of the service system, the provided access authority and the account number type, and maintaining the connection between the DHP data platform and the service database, the message queue or the server. Proxy information configured for data processing and service distribution may also be saved.
b) Data source metadata information: storing metadata information in the scanned source system, such as: data table, view, stored procedure, function.
c) Data warehouse resources: and storing the connection information stored in the data warehouse, and providing storage resources for each layer of physical model and service data floor.
2) Original data layer: ODS (Operational Data Store), also called as source data layer, is used for storing source data of service system, extracting original data of service system to ODS layer for storage, and keeping synchronous update with source database of service system as basis for processing and analyzing data of each layer. The metadata directory of this layer mainly includes: a) Business system database instance or Schema: establishing a corresponding catalog according to the source database examples and Schema of different service systems of different power plants; and establishing a corresponding physical model according to the table information in the source database, wherein the physical model table and the data table in the source database keep the same table structure, table logic relation and field information.
b) Manually imported database instances or schemas: data manually imported from business system database instances or schemas is stored. And establishing a corresponding catalog according to the database instance of the imported data source and the Schema name, and storing the imported manual land table under the catalog.
3) Data detail layer: and DWD, performing data cleaning and format conversion according to field standards, classifying according to the service field and the theme field, and integrating data from different power plants in the service system into corresponding data tables. The field number of the DWD layer data table is expanded according to different power plant business conditions, and the types of the fields are consistent with those of the ODS layer. The DWD layer is used for permanently storing source system data in the data platform, and solves the problems of data quality, data integrity and non-uniform data format of different power plants.
The layering mainly comprises the following processing flows:
a) Data formatting: and respectively performing operations such as data cleaning, format conversion, null value removal and the like on the data of different service systems of each power plant in the ODS layer according to field constraint to form formatted data.
b) Data integration: and according to the classification of the nuclear power theme zone and the business sub-zone, carrying out data field modeling, integrating the data of different power plants into a unified data table, and expanding the field number according to the condition of a business system. And the slight processing treatment of the service system data is realized, and a plurality of data tables with the same service functions of different power plants are fused, so that the basic data for subsequent analysis is obtained.
The processing flows can land the processed intermediate data on physical storage according to factors such as service data conditions, processing efficiency and the like; the final integrated data can also be directly generated after field association and scheduling of the processing logic and the workflow without landing. And establishing a corresponding metadata catalogue according to the processing flow and the theme zone.
4) Data standard layer: and the STD processes and gathers the processed data in the DWD layer according to the data standard in the nuclear power field, and associates fields and attributes related to the service to form unified nuclear power standard data.
5) Data summarization layer: DWS takes business as guide, develops a model according to business requirements, outputs data delivery objects, and binds business to all the summary layer data tables so as to meet the aim of business tasks. The construction of the DWS layer is based on the standardized data to be summarized or processed in multiple models to form a service data model, and common information is subjected to coprecipitation and processing. The fields in the DWD and STD layers can be summarized according to service requirements to form a theme table and a wide table, so that complex logic relations of a service system are simplified, and a data structure which is easy to understand by service personnel is formed.
The layer data model comprises: topic tables, multi-level summary tables, application wide tables, public index libraries, and the like. This layer of data may support cross-domain enterprise-level data analysis, data mining, ad hoc queries.
6) Data mart layer: DM (Data Mart), application-oriented, can service the Data in DWD, STD, DWS layer after rank-level processing. Providing data distribution service for external applications by establishing a data asset view and providing a data access interface, and providing data for each application to support the data requirements of the service; the method can also carry out filtering or logic processing on a data model related to a specific or common report, set input parameters and provide report data for the application; personalized statistical index data in the data product may also be generated.
7) Public dimension surface layer: DIM (Dimension), storing mapping information such as a common dictionary coding table, a common dimension table and the like used by each business, and being used for dimension association of other layered data, so that business personnel can conveniently inquire the meaning of a data field.
As shown in fig. 2, the DHP data repository allocates development environment directories and production environment directories to users at a metadata directory level; the development environment catalogue and the production environment catalogue are respectively provided with a data standard layer, a data summarizing layer, a data mart layer and a public dimension surface layer. And the development environment catalogue and the production environment catalogue share a data exchange layer and an original data layer.
And each user isolates the production environment/development environment through the catalogue, and authorizes the corresponding environment and catalogue resources according to the organization structure of each user, so that the data isolation is ensured. The submission from the development environment to the production environment is required to be operated according to a specified flow, and in the process of developing the data resource, the data resource is strictly required to be released under the production environment after the test under the relevant catalog under the development environment passes. The direct modification of the workflow in the production environment is strictly prohibited, and all the workflow in the production environment must pass the test and confirm that the test passes.
Metadata directories are created to better manage the data model, mapping information, flow, etc. resources in the project. According to the project situation, the DHP data warehouse disclosed by the invention distributes corresponding development and production environment catalogs for the project under the existing metadata catalogs of the DHP center tenant by the DHP manager. In order to better implement the above operation, in this embodiment, the resource directory hierarchy of the DHP center tenant is as follows:
data exchange layer (EXCH): and storing metadata such as connection configuration of data resources, source table structure information and the like.
1) The resource: including database resources, proxy resources, FTP resources, and Kafka resources. Database resources: and storing connection information of a source system database, and storing connection information of resources such as Hive memory bank, MPP database, RDS and the like used in a data center station. In order to avoid excessive impact of data extraction on the data source service library, only one connection to a certain source service library is usually maintained in the tenant of the DHP center, and unified management is performed under the catalog.
Proxy resources: data center for managing workflow, auditing task and data service
Proxy service resources responsible for scheduling execution of flows and services.
FTP resource: and managing FTP server resources used when the file mode carries out data import and export.
Kafka resource: and managing Kafka resources used in real-time transmission, calculation and analysis.
2) And (3) a data acquisition source: the data table information scanned from the source system database is stored so as to establish a destination table with the same field structure according to the source data table information during data extraction.
Original data layer (ODS): and extracting the original data of the service system to the ODS layer for storage, and keeping synchronous updating with the source database of the service system to serve as a basis for subsequent data processing and analysis. In order to ensure the consistency of data and avoid data redundancy, a DHP platform manager performs unified extraction and management on the endogenous data of the DHP center tenant, provides the data for corresponding production and development users according to project requirements, and grants the users access rights to the data table in the data according to project management requirements.
1) Table: the ODS layer has no logic model, only establishes a physical table consistent with the data source table, and respectively performs data extraction and update according to the frequency of data update.
2) Processing tasks: storing migration mapping, processing mapping, workflow and auditing tasks for realizing ODS layer data extraction.
Migration mapping: data migration between homologous or heterogeneous databases may be implemented.
And (3) processing and mapping: and realizing data processing among homologous data tables. The data in the source list is processed and then falls to the destination list.
The workflow: the mapping is loaded and the executed job tasks are scheduled, and a plurality of processing nodes scheduled in the execution process can be added in the workflow.
Auditing tasks: and scheduling the job task for executing the auditing rules, and checking the data in the table appointed with the auditing rules.
The development environment, the catalog stores metadata for development and testing, data tables related to result data and processing tasks, can be used for verifying the reasonability of the data table structure and the validity of processing logic, and generates temporary data for some tests.
The following subdirectories are divided:
1) Development data detail layer (development DWD): and according to the topic domain classification, integrating the data from different power plants in the ODS layer into a corresponding data table.
2) Development data standard layer (development STD): according to the topic domain classification, processing and forming unified nuclear power standard data according to the data standard of the nuclear power domain.
3) Development data summary layer (development DWS): and according to the topic domain classification, summarizing and processing the lower data to form a multi-level summary table, a public index library and a tag library.
4) Development data mart layer (development DM): and after the lower-layer data are subjected to row-column level processing, establishing a data asset view, and serving the outside.
5) Development of common dimension skins (development of DIM): and storing mapping information such as a data dictionary, a coding table, a dimension table and the like used by the service according to the topic domain classification, and using the mapping information for dimension association of other layered data.
6) Development application test: the processed result data for testing is imported into a designated intermediate storage, typically a database with higher query performance, such as MPP and MySQL, and provided to an application, and whether the result data meets expectations or not is tested.
7) Development and processing tasks: and storing migration mapping, processing mapping, workflow and auditing tasks for data import, processing and quality inspection.
And the production environment catalog stores metadata provided for production and use, data sheets related to data delivery and processing tasks, and the generated result data supports business applications.
In order to avoid the influence of data change and human misoperation on the normal operation of the production application, a data model and a processing task of the development environment are not directly built in the production environment, but are migrated to the production environment after the development and testing in the development environment are completed. And any changes in the production environment require approval for auditing and recording.
The directory structure design in the production environment is basically the same as the directory structure design in the development environment, and comprises the following steps: a production data detail layer (production DWD), a production data standard layer (production STD), a production data summarization layer (production DWS), a production data mart layer (production DM), a production public dimension layer (production DIM) and a production processing task catalog.
Production application delivery data: the catalog is used for importing processed result data for production operation into a designated intermediate storage, and real-time query is provided for the application.
And the data migration, processing and auditing workflow is operated in the development environment, and the result data is verified to accord with the expectation, so that metadata such as data models, mapping, auditing and workflow in the development environment can be imported into the production environment. Comprising the following steps:
data model import: the logic designer of the development environment is exported and then imported into the production environment. And generating a physical designer and a physical model through physical materialization of the logic designer.
Map import: the migration map and the processing map of the development environment are exported and then imported into the production environment, and renaming is needed when importing, because the uniqueness of the names of the maps in one tenant must be guaranteed, the renaming of the maps can lead to the fact that when the flow is scheduled, two maps with identical names exist, and the execution of which map cannot be distinguished.
And (3) auditing rule importing: and exporting auditing rules of the development environment in batches, and importing the auditing rules into a production environment related catalog.
Workflow importation: and exporting the workflow of the development environment in batches, and importing the workflow of the development environment into a production environment related directory. Auditing tasks may be configured to schedule execution in a workflow, thus in a development environment
After the workflow in the production environment is scheduled to be executed, the generated result data can be provided to the application in a data service mode. The delivery data in the data warehouse is synchronized to the MPP database by using migration mapping, and the data is provided to the application by means of the data service interface API. The data may also be distributed to designated intermediate stores, such as RDS, redis, kafka, providing the data to the application.
Meanwhile, as a preferred scheme, in the embodiment, the data warehouse uses the topic domains to carry out classification management on the data models, and the topic domains are divided to better maintain data from the aspects of data asset management, identification and classification, so that the data models among different topic domains are separated from each other, and no intersection exists on the service use level, and the high cohesion and low coupling are realized; and splitting and correlating the contained data model in the topic domain.
The foregoing description of the embodiments of the present disclosure has been presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the various embodiments described. The terminology used herein was chosen in order to best explain the principles of the embodiments, the practical application, or the technical improvement of the technology in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (7)
1. A nuclear power industrial data warehouse system, the nuclear power industrial data warehouse system comprising:
data exchange layer: for storing metadata related to the data resource allocation;
the original data layer is used for storing the source data of the service system and keeping synchronous updating with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to field standards, and integrating data from different power plants in a service system into corresponding data tables according to service fields and topic field classification;
data standard layer: the method comprises the steps of processing and summarizing processed data in a data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
data summarization layer: the system is used for taking the service as a guide, developing a model according to the service requirement and outputting a data delivery object;
data mart layer: the data processing module is used for performing row-column level processing on the data in the data detail layer, the data standard layer and the data summarization layer and then providing application to the outside;
the public dimension surface layer is used for storing a public dictionary coding table and public dimension table mapping information used by each service and dimension association of other layered data;
the metadata directory of the data exchange layer includes:
the data source connection is used for establishing access connection configuration information of various types of data sources according to the characteristics of access data of the service system, the provided access authority and account types, maintaining the connection between the DHP data platform and the service database, the message queue or the server, and storing proxy information configured for data processing and service distribution;
the data source metadata information is used for storing the metadata information in the scanned source system;
the data warehouse resource is used for storing connection information stored in the data warehouse and providing storage resources for each layer of physical model and service data floor;
the metadata directory of the original data layer includes:
the business system database instance or Schema is used for establishing a corresponding catalog according to the source database instance and Schema of different business systems of different power plants, and establishing a corresponding physical model according to the table information in the source database;
a manually imported database instance or Schema for storing manually imported data from the service system database instance or Schema; and establishing a corresponding catalog according to the database instance of the imported data source and the Schema name, and storing the imported manual land table under the catalog.
2. The nuclear power industry data warehouse system of claim 1, wherein the nuclear power industry data warehouse system allocates a development environment catalog and a production environment catalog to users at a metadata catalog level, wherein the development environment catalog and the production environment catalog are provided with the data detail layer, the data standard layer, the data summarization layer, the data mart layer and the public dimension layer, and the development environment catalog further comprises a development processing task catalog and a development application delivery data catalog; the production environment catalogue also comprises a production processing task catalogue and a production application delivery data catalogue; the nuclear power industrial data warehouse system acquires external requirements, extracts data from a source database to an original data layer, completes operation development under a development environment catalog, and issues the data outwards under a production environment catalog after the data passes the test.
3. The nuclear power industry data warehouse system of claim 2, wherein the development environment inventory is configured to run workflow for data migration, processing, auditing and to verify that the result data meets expectations; the production environment catalog is used for migrating the data model and the processing task which pass the verification in the development environment catalog to the production environment catalog and providing services to the outside.
4. The nuclear power industry data warehouse system of claim 2, wherein the development environment inventory and the production environment inventory are data isolated using different databases.
5. The nuclear power industry data warehouse system of claim 4, wherein the data warehouse is constructed based on Hive databases, different bins being layered to correspond to different Hive databases.
6. The nuclear power industrial data warehouse system of claim 1, wherein the data detail layer comprises:
data formatting processing layer: the method comprises the steps of respectively carrying out data cleaning, format conversion and null value removal operation on data of different service systems of each power plant in a data exchange layer according to field constraint so as to form formatted data;
data integration layer: and according to the classification of the nuclear power theme zone and the business sub-zone, carrying out data field modeling, and integrating the data of different power plants into a unified data table.
7. The nuclear power industrial data warehouse system of claim 1, wherein the data model of the data summary layer comprises one or more combinations of a topic table, a multi-level summary table, an application wide table, a common index library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111526722.7A CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111526722.7A CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114357088A CN114357088A (en) | 2022-04-15 |
| CN114357088B true CN114357088B (en) | 2024-02-27 |
Family
ID=81098683
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111526722.7A Active CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114357088B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115098477A (en) * | 2022-06-23 | 2022-09-23 | 中核核电运行管理有限公司 | Data standard management method and device for production data of nuclear power station |
| CN115510021B (en) * | 2022-06-29 | 2023-12-22 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing data warehouse standard layer |
| CN115033646B (en) * | 2022-08-11 | 2023-01-13 | 深圳联友科技有限公司 | Method for constructing real-time warehouse system based on Flink and Doris |
| CN115794929B (en) * | 2023-01-13 | 2023-05-23 | 中企云链（北京）金融信息服务有限公司 | Data management system and data management method for data marts |
| CN116401254A (en) * | 2023-04-17 | 2023-07-07 | 广东数果科技有限公司 | Unified storage method and device for index result data |
| CN117390054A (en) * | 2023-12-08 | 2024-01-12 | 先进操作系统创新中心(天津)有限公司 | Data management method and system for domestic ecological adaptation data |
| CN117407456B (en) * | 2023-12-11 | 2024-03-19 | 中核武汉核电运行技术股份有限公司 | Structured data sharing system for nuclear power service |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019462A (en) * | 2017-11-14 | 2019-07-16 | 南方电网科学研究院有限责任公司 | Electric power research creation data analysis method, device, system and storage medium |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN112182091A (en) * | 2020-12-03 | 2021-01-05 | 光大科技有限公司 | Multi-source data integration method, system, storage medium and electronic device |
| CN112181940A (en) * | 2020-08-25 | 2021-01-05 | 天津农学院 | Method for constructing national industrial and commercial big data processing system |
| CN112232854A (en) * | 2020-09-25 | 2021-01-15 | 北京三快在线科技有限公司 | Service processing method, device, equipment and storage medium |
| CN113254544A (en) * | 2021-04-29 | 2021-08-13 | 西安交通大学 | Data processing device and method based on dimension modeling |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030204487A1 (en) * | 2002-04-26 | 2003-10-30 | Sssv Muni Kumar | A System of reusable components for implementing data warehousing and business intelligence solutions |
| US10235439B2 (en) * | 2010-07-09 | 2019-03-19 | State Street Corporation | Systems and methods for data warehousing in private cloud environment |
-
2021
- 2021-12-14 CN CN202111526722.7A patent/CN114357088B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110019462A (en) * | 2017-11-14 | 2019-07-16 | 南方电网科学研究院有限责任公司 | Electric power research creation data analysis method, device, system and storage medium |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN112181940A (en) * | 2020-08-25 | 2021-01-05 | 天津农学院 | Method for constructing national industrial and commercial big data processing system |
| CN112232854A (en) * | 2020-09-25 | 2021-01-15 | 北京三快在线科技有限公司 | Service processing method, device, equipment and storage medium |
| CN112182091A (en) * | 2020-12-03 | 2021-01-05 | 光大科技有限公司 | Multi-source data integration method, system, storage medium and electronic device |
| CN113254544A (en) * | 2021-04-29 | 2021-08-13 | 西安交通大学 | Data processing device and method based on dimension modeling |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114357088A (en) | 2022-04-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114357088B (en) | Nuclear power industry data warehouse system | |
| CN112685385B (en) | Big data platform for smart city construction | |
| US11360950B2 (en) | System for analysing data relationships to support data query execution | |
| US10878064B2 (en) | Clinical data management system | |
| CN110781236A (en) | Method for constructing government affair big data management system | |
| CN107506462B (en) | Data processing method and system for enterprise-level data, electronic equipment and storage medium | |
| CN113392227B (en) | Metadata knowledge graph engine system oriented to rail transit field | |
| CN111651431B (en) | Database service-oriented management flow standardization method | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| CN105589874A (en) | ETL task dependence relationship detecting method and device and ETL tool | |
| CN111858615B (en) | Database table generation method, system, computer system and readable storage medium | |
| US20110153562A1 (en) | Error prevention for data replication | |
| WO2012130489A1 (en) | Method, system, and computer program product for maintaining data consistency between two databases | |
| CN102663008A (en) | Government integrated business platform business library and construction method of base library | |
| CN109918426A (en) | Eat the building method of drug administration data management platform | |
| CN111177847A (en) | Civil aircraft configuration management method and device | |
| CN115599840A (en) | Complex service data management method and system | |
| CN115617776A (en) | Data management system and method | |
| US20190065535A1 (en) | Mining of policy data source description based on file, storage and application meta-data | |
| CN115080765A (en) | Aerospace quality knowledge map construction method, system, medium and equipment | |
| US20080294673A1 (en) | Data transfer and storage based on meta-data | |
| Al-Barak et al. | Database design debts through examining schema evolution | |
| CN109150964A (en) | A kind of transportable data managing method and services migrating method | |
| Glava et al. | Information Systems Reengineering Approach Based on the Model of Information Systems Domains | |
| CN112084177B (en) | Data pool application method and device based on data acquisition treatment and mining analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |