CN114357088A - Nuclear power industry data warehouse system - Google Patents
Nuclear power industry data warehouse system Download PDFInfo
- Publication number
- CN114357088A CN114357088A CN202111526722.7A CN202111526722A CN114357088A CN 114357088 A CN114357088 A CN 114357088A CN 202111526722 A CN202111526722 A CN 202111526722A CN 114357088 A CN114357088 A CN 114357088A
- Authority
- CN
- China
- Prior art keywords
- data
- layer
- nuclear power
- directory
- service
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012545 processing Methods 0.000 claims abstract description 63
- 238000013507 mapping Methods 0.000 claims abstract description 23
- 238000012384 transportation and delivery Methods 0.000 claims abstract description 12
- 238000006243 chemical reaction Methods 0.000 claims abstract description 9
- 238000004140 cleaning Methods 0.000 claims abstract description 8
- 238000011161 development Methods 0.000 claims description 56
- 238000004519 manufacturing process Methods 0.000 claims description 56
- 238000013499 data model Methods 0.000 claims description 16
- 238000012360 testing method Methods 0.000 claims description 10
- 238000013508 migration Methods 0.000 claims description 9
- 230000005012 migration Effects 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 7
- 238000009826 distribution Methods 0.000 claims description 5
- 230000001360 synchronised effect Effects 0.000 claims description 5
- 239000000284 extract Substances 0.000 claims description 4
- 230000010354 integration Effects 0.000 claims description 4
- 238000003672 processing method Methods 0.000 claims description 2
- 238000012795 verification Methods 0.000 claims description 2
- 238000000034 method Methods 0.000 abstract description 12
- 230000008569 process Effects 0.000 abstract description 7
- 239000010410 layer Substances 0.000 description 89
- 238000012550 audit Methods 0.000 description 8
- 238000007726 management method Methods 0.000 description 7
- 238000004458 analytical method Methods 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 230000008676 import Effects 0.000 description 3
- 238000012432 intermediate storage Methods 0.000 description 3
- 238000013075 data extraction Methods 0.000 description 2
- 238000000605 extraction Methods 0.000 description 2
- 239000002344 surface layer Substances 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000007418 data mining Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000002955 isolation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000001556 precipitation Methods 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Images
Abstract
The disclosure belongs to the technical field of nuclear power, and particularly relates to a nuclear power industry data warehouse system. The nuclear power industry data warehouse system of the present disclosure includes: the data exchange layer is used for storing metadata related to data resource configuration; the original data layer is used for storing service system source data; the data detail layer performs data cleaning and format conversion according to the field standard, the data standard layer processes and summarizes the processed data in the data detail layer according to the data standard of the nuclear power field, and the fields and attributes related to the service are associated to form unified nuclear power standard data; the data summarization layer takes the business as guidance, develops a model according to business requirements and outputs data deliveries; the data market layer performs row-column level processing on data in the data detail layer, the data standard layer and the data summary layer and provides application for the outside; the public dimension layer stores a public dictionary coding table and public dimension table mapping information used by each service and is also used for dimension association of other hierarchical data.
Description
Technical Field
The invention belongs to the technical field of nuclear power, and particularly relates to a nuclear power industry data warehouse system.
Background
A Chinese nuclear power big nuclear source platform (DHP) serves as a digital nuclear power supporting platform and a nerve center, integrates data of Chinese nuclear power massive industrial systems and equipment, constructs an extensible open nuclear power industrial internet platform, synchronously develops application and development ecosystems for various scenes and reusable nuclear power industry, improves the use efficiency and sharing range of hardware, service and data of a nuclear power plant, realizes intelligent management and operation optimization of Chinese nuclear power business and resources, drives a series of innovative nuclear power industrial applications for a nuclear power whole industrial chain, and ensures safe, reliable and efficient operation of an operating unit.
The DHP data platform needs to access relational data from different service systems of different power plants, and how to process the accessed data becomes a problem to be solved urgently.
Disclosure of Invention
In order to overcome the problems in the related art, a nuclear power industry data warehouse system is provided.
According to an aspect of the disclosed embodiments, there is provided a nuclear power industry data warehouse system, comprising:
a data exchange layer: the system is used for storing metadata related to data resource configuration;
the original data layer is used for storing the source data of the service system and keeping synchronous update with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to the field standard, classifying according to the service field and the subject field, and integrating data from different power plants in the service system into corresponding data tables;
data standard layer: the system comprises a data detail layer, a data processing layer and a data processing layer, wherein the data processing layer is used for processing and summarizing processed data in the data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
and a data summarization layer: the system is used for developing a model according to business requirements by taking business as guidance and outputting data deliveries;
data mart layer: the data processing device is used for performing row-column level processing on data in the data detail layer, the data standard layer and the data summary layer and then providing application for the outside;
and the public dimension layer is used for storing a public dictionary coding table and public dimension table mapping information used by each service and also used for dimension association of other hierarchical data.
In a possible implementation manner, the nuclear power industry data warehouse system allocates a development environment directory and a production environment directory for users under a metadata directory level, the development environment directory and the production environment directory are respectively provided with the data detail layer, the data standard layer, the data summary layer, the data mart layer and the public dimension surface layer, and the development environment directory further comprises a development and processing task directory and a development and application delivery data directory; the production environment catalog also comprises a production and processing task catalog and a production application delivery data catalog; the nuclear power industry data warehouse system acquires external requirements, extracts data from a source database to an original data layer, completes operation development under a development environment directory, and releases the data to the outside under a production environment directory after the data passes a test.
In one possible implementation manner, the development environment directory is used for passing the workflow of data migration, processing and auditing and verifying whether the result data meets expectations; the production environment catalog is used for transferring the data model and the processing task which pass the verification in the development environment catalog to the production environment catalog and providing services for the outside.
In one possible implementation, the development environment catalog and the production environment catalog are data isolated using different databases.
In one possible implementation, the data warehouse is constructed based on Hive databases, and different levels of the data warehouse correspond to different Hive databases.
In one possible implementation, the metadata directory of the data exchange layer includes:
the data source connection is used for establishing access connection configuration information of various types of data sources according to the characteristics of service system access data, provided access authority and account number types, maintaining the connection between the DHP data platform and a service database, a message queue or a server, and storing agent information configured for data processing and service distribution;
the data source metadata information is used for storing the scanned metadata information in the source system;
and the data warehouse resources are used for storing the connection information stored in the data warehouse and providing storage resources for the physical models and the service data of each layer.
In one possible implementation, the metadata directory of the original data layer includes:
the service system database instance or Schema is used for establishing a corresponding directory according to the accessed source database instances and schemas of different service systems of different power plants and establishing a corresponding physical model according to the table information in the source database;
the database instance or Schema imported manually is used for storing data imported from the database instance or Schema of the manual business system; and establishing a corresponding directory according to the database instance of the imported data source and the Schema name, and storing the imported manual ground surface under the directory.
In one possible implementation, the data detail layer includes:
data formatting layer: the data processing method comprises the steps of respectively carrying out data cleaning, format conversion and null value removing operations on data of different service systems of each power plant in a data exchange layer according to field constraints to form formatted data;
and a data integration layer: and modeling the data field according to the classification of the nuclear power theme field and the business subdomain, and integrating the data of different power plants into a unified data table.
In one possible implementation manner, the data model of the data summarization layer comprises one or more combinations of a theme table, a multi-level summary table, an application wide table and a public index library.
The beneficial effect of this disclosure lies in: the nuclear power industry data warehouse system disclosed by the invention adopts a layered theme domain-based mode to construct a data center so as to access relational data from different business systems of different power plants, forms data sets of different layers through layered processing, integration and conversion according to nuclear power data standards and application requirements, provides the data sets for external application, facilitates development and analysis of data, tracks data bloodiness and improves reusability of a data model.
Drawings
FIG. 1 is a schematic diagram illustrating a nuclear power industry data warehouse system in accordance with an exemplary embodiment.
FIG. 2 illustrates a nuclear power industry data warehouse with distribution of development environment directories and production environment directories to users at a metadata directory level in accordance with an exemplary embodiment.
Detailed Description
The invention is described in further detail below with reference to the figures and the embodiments.
The present disclosure provides a nuclear power industry data warehouse system, which includes:
a data exchange layer: for storing metadata related to the data resource configuration.
The original data layer is used for storing the source data of the service system and keeping synchronous update with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to the field standard, classifying according to the service field and the subject field, and integrating data from different power plants in the service system into corresponding data tables;
data standard layer: the system comprises a data detail layer, a data processing layer and a data processing layer, wherein the data processing layer is used for processing and summarizing processed data in the data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
and a data summarization layer: the system is used for developing a model according to business requirements by taking business as guidance and outputting data deliveries;
data mart layer: used for providing application for external after row-column level processing is carried out on data in DWD (data Warehouse detail), STD (Standard), DWS (data Warehouse summary) layers;
and a common dimension layer: DIM (dimension) used for storing mapping information such as a public dictionary code table and a public dimension table used by each service, and used for dimension association of other hierarchical data, thereby facilitating service personnel to inquire the meaning of the data field.
In this embodiment, the DHP data warehouse is divided into the following 7 levels:
1) a data exchange layer: exchange, storing metadata related to data resource configuration. The metadata directory of this layer mainly includes:
a) data source connection: and storing information such as database resources, proxy resources and the like. According to the characteristics of the service system access data, the provided access authority and the account number type, establishing access connection configuration information of various types of data sources, and maintaining the connection between the DHP data platform and a service database, a message queue or a server. Agent information configured for data processing and service distribution may also be stored.
b) Data source metadata information: and storing metadata information in the scanned source system, such as: data tables, views, stored procedures, functions.
c) Data warehouse resources: and storing the connection information stored in the data warehouse, and providing storage resources for each layer of physical model and service data.
2) Original data layer: the ODS (operational Data store), also called a source Data layer, is used to store the source Data of the service system, extract the original Data of the service system to the ODS layer for storage, and keep updating synchronously with the source database of the service system, which is used as the basis for processing and analyzing the Data of each subsequent layer. The metadata directory of this layer mainly includes: a) service system database instance or Schema: establishing a corresponding directory according to source database examples and schemas of different service systems of different power plants; and establishing a corresponding physical model according to the table information in the source database, wherein the physical model table and the data table in the source database keep the same table structure, table logic relationship and field information.
b) Database instance or Schema imported by hand: and storing data manually imported from a business system database instance or Schema. And establishing a corresponding directory according to the database example of the imported data source and the Schema name, and storing the imported manual earth falling surface under the directory.
3) Data detail layer: and DWD, according to the field standard, performing data cleaning and format conversion, classifying according to the service field and the subject field, and integrating the data from different power plants in the service system into the corresponding data table. The number of fields of the DWD layer data table is expanded according to different power plant service conditions, and the types of the fields are consistent with the ODS layer. The DWD layer is used for permanently storing source system data on a data platform, and solves the problems of data quality of different power plants, data integrity and non-uniform data formats.
The layering mainly comprises the following processing flows:
a) data formatting treatment: and respectively carrying out operations such as data cleaning, format conversion, null value removal and the like on the data of different service systems of each power plant in the ODS layer according to field constraints to form formatted data.
b) Data integration: according to the classification of the nuclear power theme domain and the business sub-domain, data domain modeling is carried out, data of different power plants are integrated into a unified data table, and the number of fields is expanded according to the business system condition. The method has the advantages that mild processing of the data of the service system is achieved, and a plurality of data tables with the same service functions of different power plants are fused, so that basic data for subsequent analysis are obtained.
The above processing flows can be used for landing the processed intermediate data to physical storage according to the factors such as the service data condition, the processing efficiency and the like; or the processing logic and the workflow are subjected to field association, scheduling and arrangement without falling to the ground, and then the final integrated data is directly generated. And establishing a corresponding metadata directory according to the processing flow and the subject domain.
4) Data standard layer: and the STD processes and summarizes the processed data in the DWD layer according to the data standard in the nuclear power field, and associates fields and attributes related to services to form uniform nuclear power standard data.
5) And a data summarization layer: DWS, which takes business as guidance, develops a model according to business requirements, outputs data deliveries, and binds all summary data tables with business so as to meet business tasks. And the DWS layer is constructed based on the data subjected to standardization processing to perform summary or multi-model processing to form a service data model, and common information is subjected to common precipitation and processing. Fields in DWD and STD layers can be gathered to form a subject table and a wide table according to service requirements, so that the complex logic relationship of a service system is simplified, and a data structure which is easy to understand by service personnel is formed.
The layer data model comprises: subject tables, multi-level summary tables, application wide tables, public index libraries, and the like. The layer data may support cross-domain enterprise-level data analysis, data mining, ad hoc queries.
6) Data mart layer: DM (data mart) is oriented to application, and can serve the data in DWD, STD and DWS layers after row-column level processing. The method comprises the steps of providing data distribution service for external applications by establishing a data asset view and providing a data access interface, providing data for each application and supporting data requirements of services; or filtering or logically processing the data model related to a specific or common report, setting input parameters and providing report data for the application; personalized statistical indicator data in the data product may also be generated.
7) Public dimension layer: DIM (dimension) stores mapping information such as a public dictionary code table and a public dimension table used by each service, is used for dimension association of other hierarchical data, and is convenient for service personnel to inquire data field meanings.
As shown in fig. 2, the DHP data warehouse allocates a development environment directory and a production environment directory to users at a metadata directory level; the development environment catalog and the production environment catalog are respectively provided with a data detail layer data standard layer, a data summarization layer, a data mart layer and a public dimension surface layer. And the development environment directory and the production environment directory share a data exchange layer and an original data layer.
And each user isolates the production environment/development environment through the directory, authorizes the corresponding environment and directory resources according to the organizational structure of each user, and ensures data isolation. The submission from the development environment to the production environment needs to be operated according to a specified flow, and the data resource is strictly required to be released in the production environment directory after the test passes under the relevant directory under the development environment in the data resource development process. The operation flow is strictly prohibited from being modified directly in the production environment, and all the operation flows on the production environment line must be tested and the test is confirmed to pass.
The metadata catalog is created to better manage resources such as data models, mapping information, processes, etc. in the project. According to the project situation, the DHP management party distributes corresponding development and production environment catalogues for the project under the existing metadata catalog level of the DHP center tenant. In order to better implement the above operation, in this embodiment, the resource directory hierarchy of the DHP center tenant is as follows:
data exchange layer (EXCH): and storing metadata such as connection configuration of data resources, source table structure information and the like.
1) Resource: including database resources, proxy resources, FTP resources, and Kafka resources. Database resources: and storing the connection information of the source system database and the connection information of resources such as a Hive storage library, an MPP database, RDS and the like used in the data center station. In order to avoid excessive influence of data extraction on the data source service library, only one connection to a certain source service library is usually maintained in the DHP central tenant, and is managed under the directory in a unified manner.
Proxy resources: data middleboxes for managing workflow, auditing tasks, and for use in data services
And the proxy service resource is responsible for scheduling and executing the flow and the service.
FTP resources: the management file system is used for importing and exporting data.
Kafka resources: managing Kafka resources used in real-time transmission, calculation, and analysis.
2) A data acquisition source: and storing the scanned data table information from the database of the source system so as to establish a destination table with the same field structure according to the source data table information during data extraction.
Original data layer (ODS): and extracting the original data of the service system to an ODS layer for storage, and keeping synchronous update with a source database of the service system to serve as a basis for subsequent data processing and analysis. In order to ensure the consistency of data and avoid data redundancy, a DHP platform manager performs unified extraction and management on the endogenous data of a DHP center tenant, provides the endogenous data for corresponding production and development users according to project requirements, and grants the access rights of the users to a data table in the DHP center tenant according to project management requirements.
1) Table: the ODS layer has no logical model, only establishes a physical table consistent with the data source table field, and respectively extracts and updates data according to the frequency of data updating and divided into days, weeks and months.
2) And (3) processing tasks: and storing migration mapping, processing mapping, workflow and audit tasks for realizing extraction of ODS layer data.
Migration mapping: data migration between homogeneous or heterogeneous databases may be implemented.
Processing and mapping: and data processing between homologous data tables is realized. And processing the data in the source table and then falling to the destination table.
Workflow: the mapping is loaded and the executed job tasks are scheduled, and a plurality of processing nodes scheduled in the execution process can be added in the workflow.
Auditing tasks: and scheduling the job task for executing the audit rule, and checking data in the table with the audit rule.
The development environment is used for storing metadata for development and testing, data tables related to result data and processing tasks, verifying the reasonability of the data table structure and the effectiveness of processing logic and generating some temporary data for testing.
The following subdirectories are divided:
1) development data detail layer (development DWD): and summarizing and integrating data from different power plants in the ODS layer into corresponding data tables according to the classification of the subject domain.
2) Development data standard layer (development STD): and processing to form uniform nuclear power standard data according to the classification of the subject domain and the data standard of the nuclear power field.
3) Development data summary layer (development DWS): and (4) according to the classification of the subject domains, summarizing and processing the data in the lower layer to form a multi-level summary table, a public index library and a label library.
4) Development data mart layer (development DM): and after row-column-level processing is carried out on the data in the lower layer, a data asset view is established, and external service is carried out.
5) Development of common dimension layer (development DIM): and according to the classification of the subject domain, storing mapping information such as a data dictionary, a coding table, a dimension table and the like used by the service for the dimension association of other hierarchical data.
6) Developing and applying tests: and importing the processed result data for testing into a specified intermediate storage, usually querying a database with higher performance, such as MPP and MySQL, and providing the database for application to test whether the result data meets expectations.
7) Developing and processing tasks: and storing migration mapping, processing mapping, workflow and audit tasks for data import, processing and quality inspection.
And the production environment directory stores metadata for production use, data tables related to data delivery and processing tasks, and the generated result data supports business applications.
In order to avoid the influence of data change and manual misoperation on the normal operation of production application, a data model and a processing task are not directly newly built in a production environment, but the data model and the processing task of the development environment are migrated to the production environment after the development and the test in the development environment are completed. And any changes in the production environment require review approval and recording.
The design of the directory structure in the production environment is basically the same as that in the development environment, and both the design of the directory structure in the production environment and the design of the directory structure in the development environment comprise: the production data detail layer (production DWD), the production data standard layer (production STD), the production data summary layer (production DWS), the production data market layer (production DM), the production public dimension layer (production DIM) and the production and processing task directory.
Production application delivery data: the catalog is used for importing processed result data for production operation into a designated intermediate storage, and provides real-time query for applications.
The data migration, processing and auditing workflows are run through in the development environment, and the result data are verified to be in accordance with expectations, so that the metadata of the data models, the mapping, the auditing, the workflow and the like in the development environment can be imported into the production environment. The method comprises the following steps:
importing a data model: the logic designer for the development environment is exported and then imported into the production environment. Physical designers and physical models are generated through logical designer materialization.
Mapping import: the migration mapping and the processing mapping of the development environment are exported and then imported into the production environment, and renaming is needed during import, because the mapping name in one tenant needs to ensure uniqueness, and the mapping name is duplicated, two names have the same mapping when the process is scheduled, and which mapping can not be executed can not be distinguished.
Importing audit rules: exporting the audit rules of the development environment in batches, and then importing the audit rules into a relevant directory of the production environment.
And (3) workflow importing: and exporting the workflow of the development environment in batches, and then importing the workflow into a related directory of the production environment. Audit tasks can be configured to be scheduled for execution in a workflow, and thus in a development environment
After the workflow scheduling in the production environment is executed, the generated result data can be provided to the application in a data service mode. Delivery data in a data warehouse is synchronized to an MPP database by using migration mapping, and the data is provided for an application in a data service interface API mode. Data may also be distributed to designated intermediate storage, such as RDS, Redis, Kafka, to provide data to applications.
Meanwhile, as a preferred scheme, in this embodiment, a data warehouse uses a topic domain to perform classification management on data models, the topic domain is divided for better maintaining data from the perspective of data asset management, identification and classification, data models between different topic domains are separated from each other, there is no intersection on a service usage level, and high cohesion and low coupling are provided; and splitting and associating the contained data model inside the subject domain.
Having described embodiments of the present disclosure, the foregoing description is intended to be exemplary, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein were chosen in order to best explain the principles of the embodiments, the practical application, or technical improvements to the techniques in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.
Claims (9)
1. A nuclear power industry data warehouse system, comprising:
a data exchange layer: the system is used for storing metadata related to data resource configuration;
the original data layer is used for storing the source data of the service system and keeping synchronous update with the source database of the service system;
data detail layer: the system is used for carrying out data cleaning and format conversion according to the field standard, classifying according to the service field and the subject field, and integrating data from different power plants in the service system into corresponding data tables;
data standard layer: the system comprises a data detail layer, a data processing layer and a data processing layer, wherein the data processing layer is used for processing and summarizing processed data in the data detail layer according to a data standard in the nuclear power field, and associating fields and attributes related to services to form unified nuclear power standard data;
and a data summarization layer: the system is used for developing a model according to business requirements by taking business as guidance and outputting data deliveries;
data mart layer: the data processing device is used for performing row-column level processing on data in the data detail layer, the data standard layer and the data summary layer and then providing application for the outside;
and the public dimension layer is used for storing a public dictionary coding table and public dimension table mapping information used by each service and also used for dimension association of other hierarchical data.
2. The nuclear power industry data warehouse system of claim 1, wherein the nuclear power industry data warehouse system allocates a development environment directory and a production environment directory to users at a metadata directory level, the development environment directory and the production environment directory are each provided with the data detail layer, the data standard layer, the data summary layer, the data mart layer and the public dimension layer, and the development environment directory further includes a development and processing task directory and a development and application delivery data directory; the production environment catalog also comprises a production and processing task catalog and a production application delivery data catalog; the nuclear power industry data warehouse system acquires external requirements, extracts data from a source database to an original data layer, completes operation development under a development environment directory, and releases the data to the outside under a production environment directory after the data passes a test.
3. The nuclear power industry data warehouse system of claim 2, wherein the development environment directory is configured to run data migration, processing, and auditing workflows and verify whether the resulting data meets expectations; the production environment catalog is used for transferring the data model and the processing task which pass the verification in the development environment catalog to the production environment catalog and providing services for the outside.
4. The nuclear power industry data warehouse system of claim 1, wherein the development environment catalog and the production environment catalog are data isolated using different databases.
5. The nuclear power industry data warehouse system of claim 4, wherein the data warehouse is constructed based on Hive databases, and different levels of the bins correspond to different Hive databases.
6. The nuclear power industry data warehouse system of claim 2, wherein the metadata directory of the data exchange layer comprises:
the data source connection is used for establishing access connection configuration information of various types of data sources according to the characteristics of service system access data, provided access authority and account number types, maintaining the connection between the DHP data platform and a service database, a message queue or a server, and storing agent information configured for data processing and service distribution;
the data source metadata information is used for storing the scanned metadata information in the source system;
and the data warehouse resources are used for storing the connection information stored in the data warehouse and providing storage resources for the physical models and the service data of each layer.
7. The nuclear power industry data warehouse system of claim 2, wherein the metadata directory of the raw data layer comprises:
the service system database instance or Schema is used for establishing a corresponding directory according to the accessed source database instances and schemas of different service systems of different power plants and establishing a corresponding physical model according to the table information in the source database;
the database instance or Schema imported manually is used for storing data imported from the database instance or Schema of the manual business system; and establishing a corresponding directory according to the database instance of the imported data source and the Schema name, and storing the imported manual ground surface under the directory.
8. The nuclear power industry data warehouse system of claim 1, wherein the data detail layer includes:
data formatting layer: the data processing method comprises the steps of respectively carrying out data cleaning, format conversion and null value removing operations on data of different service systems of each power plant in a data exchange layer according to field constraints to form formatted data;
and a data integration layer: and modeling the data field according to the classification of the nuclear power theme field and the business subdomain, and integrating the data of different power plants into a unified data table.
9. The nuclear power industry data warehouse system of claim 1, wherein the data models of the data summarization layer include one or more combinations of a subject table, a multi-level summary table, an application wide table, and a common index library.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111526722.7A CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111526722.7A CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114357088A true CN114357088A (en) | 2022-04-15 |
| CN114357088B CN114357088B (en) | 2024-02-27 |
Family
ID=81098683
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111526722.7A Active CN114357088B (en) | 2021-12-14 | 2021-12-14 | Nuclear power industry data warehouse system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114357088B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115033646A (en) * | 2022-08-11 | 2022-09-09 | 深圳联友科技有限公司 | Method for constructing real-time warehouse system based on Flink and Doris |
| CN115510021A (en) * | 2022-06-29 | 2022-12-23 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing standard layer of data warehouse |
| CN115794929A (en) * | 2023-01-13 | 2023-03-14 | 中企云链（北京）金融信息服务有限公司 | Data management system and data management method for data mart |
| CN116401254A (en) * | 2023-04-17 | 2023-07-07 | 广东数果科技有限公司 | Unified storage method and device for index result data |
| CN117390054A (en) * | 2023-12-08 | 2024-01-12 | 先进操作系统创新中心(天津)有限公司 | Data management method and system for domestic ecological adaptation data |
| CN117407456A (en) * | 2023-12-11 | 2024-01-16 | 中核武汉核电运行技术股份有限公司 | Structured data sharing system for nuclear power service |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030204487A1 (en) * | 2002-04-26 | 2003-10-30 | Sssv Muni Kumar | A System of reusable components for implementing data warehousing and business intelligence solutions |
| US20150347542A1 (en) * | 2010-07-09 | 2015-12-03 | State Street Corporation | Systems and Methods for Data Warehousing in Private Cloud Environment |
| CN110019462A (en) * | 2017-11-14 | 2019-07-16 | 南方电网科学研究院有限责任公司 | Electric power research creation data analysis method, device, system and storage medium |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN112181940A (en) * | 2020-08-25 | 2021-01-05 | 天津农学院 | Method for constructing national industrial and commercial big data processing system |
| CN112182091A (en) * | 2020-12-03 | 2021-01-05 | 光大科技有限公司 | Multi-source data integration method, system, storage medium and electronic device |
| CN112232854A (en) * | 2020-09-25 | 2021-01-15 | 北京三快在线科技有限公司 | Service processing method, device, equipment and storage medium |
| CN113254544A (en) * | 2021-04-29 | 2021-08-13 | 西安交通大学 | Data processing device and method based on dimension modeling |
-
2021
- 2021-12-14 CN CN202111526722.7A patent/CN114357088B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20030204487A1 (en) * | 2002-04-26 | 2003-10-30 | Sssv Muni Kumar | A System of reusable components for implementing data warehousing and business intelligence solutions |
| US20150347542A1 (en) * | 2010-07-09 | 2015-12-03 | State Street Corporation | Systems and Methods for Data Warehousing in Private Cloud Environment |
| CN110019462A (en) * | 2017-11-14 | 2019-07-16 | 南方电网科学研究院有限责任公司 | Electric power research creation data analysis method, device, system and storage medium |
| CN110489459A (en) * | 2019-08-07 | 2019-11-22 | 国网安徽省电力有限公司 | A kind of enterprise-level industry number fused data analysis system based on big data platform |
| CN112181940A (en) * | 2020-08-25 | 2021-01-05 | 天津农学院 | Method for constructing national industrial and commercial big data processing system |
| CN112232854A (en) * | 2020-09-25 | 2021-01-15 | 北京三快在线科技有限公司 | Service processing method, device, equipment and storage medium |
| CN112182091A (en) * | 2020-12-03 | 2021-01-05 | 光大科技有限公司 | Multi-source data integration method, system, storage medium and electronic device |
| CN113254544A (en) * | 2021-04-29 | 2021-08-13 | 西安交通大学 | Data processing device and method based on dimension modeling |
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115510021A (en) * | 2022-06-29 | 2022-12-23 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing standard layer of data warehouse |
| CN115510021B (en) * | 2022-06-29 | 2023-12-22 | 江苏昆山农村商业银行股份有限公司 | Method and system for constructing data warehouse standard layer |
| CN115033646A (en) * | 2022-08-11 | 2022-09-09 | 深圳联友科技有限公司 | Method for constructing real-time warehouse system based on Flink and Doris |
| CN115033646B (en) * | 2022-08-11 | 2023-01-13 | 深圳联友科技有限公司 | Method for constructing real-time warehouse system based on Flink and Doris |
| CN115794929A (en) * | 2023-01-13 | 2023-03-14 | 中企云链（北京）金融信息服务有限公司 | Data management system and data management method for data mart |
| CN116401254A (en) * | 2023-04-17 | 2023-07-07 | 广东数果科技有限公司 | Unified storage method and device for index result data |
| CN117390054A (en) * | 2023-12-08 | 2024-01-12 | 先进操作系统创新中心(天津)有限公司 | Data management method and system for domestic ecological adaptation data |
| CN117407456A (en) * | 2023-12-11 | 2024-01-16 | 中核武汉核电运行技术股份有限公司 | Structured data sharing system for nuclear power service |
| CN117407456B (en) * | 2023-12-11 | 2024-03-19 | 中核武汉核电运行技术股份有限公司 | Structured data sharing system for nuclear power service |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114357088B (en) | 2024-02-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114357088A (en) | Nuclear power industry data warehouse system | |
| CN112685385B (en) | Big data platform for smart city construction | |
| US11360950B2 (en) | System for analysing data relationships to support data query execution | |
| CN110300963B (en) | Data management system in a large-scale data repository | |
| US10878064B2 (en) | Clinical data management system | |
| US11003645B1 (en) | Column lineage for resource dependency system and graphical user interface | |
| CN110781236A (en) | Method for constructing government affair big data management system | |
| US11341171B2 (en) | Method and apparatus for implementing a set of integrated data systems | |
| US8484255B2 (en) | Automatic conversion of multidimentional schema entities | |
| CN112199433A (en) | Data management system for city-level data middling station | |
| US20110153562A1 (en) | Error prevention for data replication | |
| WO2012130489A1 (en) | Method, system, and computer program product for maintaining data consistency between two databases | |
| US20080294673A1 (en) | Data transfer and storage based on meta-data | |
| CN115617776A (en) | Data management system and method | |
| CN103069382A (en) | Migrating artifacts between service-oriented architecture repositories | |
| US10430413B2 (en) | Data information framework | |
| US10146809B2 (en) | Mining of policy data source description based on file, storage and application meta-data | |
| US20140143248A1 (en) | Integration to central analytics systems | |
| Schulz et al. | Read Code quality assurance: from simple syntax to semantic stability | |
| US20150363711A1 (en) | Device for rapid operational visibility and analytics automation | |
| CN115599840A (en) | Complex service data management method and system | |
| KR100796905B1 (en) | System for Quality Control of DataBase | |
| CN115952160B (en) | Data checking method | |
| CN116089413A (en) | Data management method and device | |
| CN105183756A (en) | Method for integrating NHIBERNATE in WCF RIA Services |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |