WO2021135727A1 - 一种能源数据仓库系统 - Google Patents

一种能源数据仓库系统 Download PDF

Info

Publication number
WO2021135727A1
WO2021135727A1 PCT/CN2020/130942 CN2020130942W WO2021135727A1 WO 2021135727 A1 WO2021135727 A1 WO 2021135727A1 CN 2020130942 W CN2020130942 W CN 2020130942W WO 2021135727 A1 WO2021135727 A1 WO 2021135727A1
Authority
WO
WIPO (PCT)
Prior art keywords
data
energy
processing
basic
equipment
Prior art date
Application number
PCT/CN2020/130942
Other languages
English (en)
French (fr)
Inventor
徐锡明
黄博淘
吴建波
Original Assignee
新奥数能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 新奥数能科技有限公司 filed Critical 新奥数能科技有限公司
Publication of WO2021135727A1 publication Critical patent/WO2021135727A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/21—Design, administration or maintenance of databases
    • G06F16/214—Database migration support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/23—Updating
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/27—Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/283—Multi-dimensional databases or data warehouses, e.g. MOLAP or ROLAP
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/28—Databases characterised by their database models, e.g. relational or object models
    • G06F16/284—Relational databases
    • G06F16/285—Clustering or classification

Definitions

  • the invention belongs to the technical field of energy data processing, and in particular relates to an energy data warehouse system.
  • the data warehouse is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is used to support management decision-making.
  • the enterprise data warehouse architecture proposed by Bill Inmon, the father of data warehouses, and the dimensional data warehouse architecture proposed by Ralph Kimball are two mainstream data warehouse construction methods.
  • Teradata has its own FS-LDM (Teradata Financial Services Logical Data Model) model
  • IBM has its own BDWM (Banking Data Warehouse Model) model
  • Teradata has its own CLDM (Teradata Communications Logical) model
  • Data Model and IBM has TDWM (Telecom Data Warehouse Model).
  • the purpose of the embodiments of the present invention is to provide an energy data warehouse system to solve the technical problem that there is no data warehouse system for the energy industry in the prior art.
  • the embodiment of the present invention provides an energy data warehouse system, including:
  • the operation data layer includes a detailed data table, and the energy data of the data source is written into the detailed data table after the first data processing;
  • the basic data layer includes a basic data table corresponding to each energy equipment type, and the data of the detailed data table is written into the basic data table after the second data processing;
  • the general data layer includes the data warehouse theme corresponding to the business analysis needs, and the data of the basic data table is written into the data warehouse theme after the third data processing;
  • the application data layer includes a data mart corresponding to the business unit, and the data of the data warehouse theme is written into the data mart after the fourth data processing.
  • the energy data of the data source includes at least energy equipment operating data, energy system configuration data, business data during business development, network data obtained through the network, and third-party data provided by a third-party system.
  • energy equipment operating data energy equipment operating data
  • energy system configuration data business data during business development
  • network data obtained through the network
  • third-party data provided by a third-party system.
  • the operation data layer further includes an original data table
  • the first data processing includes at least:
  • the newly loaded data is partitioned according to the type of energy equipment and the acquisition time.
  • the detailed data table includes at least one of site information, device type, device identification, measurement attribute, measurement time, and measurement value.
  • the basic data layer includes at least one of a gas-steam boiler basic data table, an air compressor basic data table, an energy meter basic data table, a business data basic data table, and an environmental data basic data table.
  • the second data processing includes at least:
  • the second data is written into the basic data table corresponding to each energy equipment type.
  • the general data layer includes at least the topic of energy enterprise capacity analysis, the topic of energy consumption analysis of energy-using enterprises, the topic of enterprise energy efficiency analysis, the topic of equipment state operation trend analysis, the topic of equipment predictive maintenance analysis, and enterprise data access One of the themes of quality analysis.
  • the third data processing at least includes:
  • the application data layer includes at least one of an energy consumption data mart, an energy supply data mart, an operation data mart, and an artificial intelligence data mart.
  • the fourth data processing at least includes:
  • the energy data warehouse system further includes:
  • Dimensional data includes at least one of time dimension, geographic dimension, user dimension, campus dimension, system dimension, and device dimension.
  • the time dimension provides dimensional definitions of year, quarter, month, day, hour, and minute;
  • the geographic dimensions provide definitions of the dimensions of the country, regions, provinces, cities, districts and counties, and parks;
  • the user dimension provides dimensional definitions of the user's industry, user level, and user category;
  • the energy equipment in the equipment dimension includes at least air compressors, refrigerators, air conditioners, gas steam boilers, gas hot water boilers, transformers, steam meters, electricity meters, energy meters, thermometers, pressure meters, gas flow meters, liquid flow meters, A kind of differential pressure gauge.
  • the energy data warehouse system further includes:
  • Management tools including metadata management tools, workflow management tools, data collection tools, data processing tools, and data migration tools.
  • the metadata management tool is used to provide a data map
  • the data collection tool is used to provide data access services to collect energy data of the data source to the operation data layer;
  • the data processing tool is used to provide tool support for data processing
  • the data migration tool is used to migrate data from the data mart to an external system to provide tool support;
  • the workflow management tool is used to provide workflow support for data collection, data processing and data migration.
  • the energy data warehouse system for the energy industry provided by the embodiments of the present invention includes an operation data layer, a basic data layer, a general data layer, and an application data layer, which effectively solves the problem
  • the problem that the general data warehouse cannot be applied to the energy field is helpful to form a unified and standard data system, speed up the energy industry's data processing and data analysis of energy equipment, and facilitate data analysts and data scientists based on high-quality mass data Perform real-time and effective analysis.
  • Figure 1 is a schematic diagram of an energy data warehouse system provided by an embodiment of the present invention.
  • FIG. 2 is an implementation flowchart of a workflow from raw data to basic data in the energy data warehouse system provided by an embodiment of the present invention
  • FIG. 3 is an implementation flowchart of a workflow for generating an energy consumption report based on basic data in an energy data warehouse system provided by an embodiment of the present invention
  • FIG. 4 is a schematic diagram of data collection in an energy data warehouse system provided by an embodiment of the present invention.
  • FIG. 5 is a schematic diagram of an implementation process of a method for constructing an energy data warehouse system provided by an embodiment of the present invention
  • FIG. 6 is a schematic diagram of an implementation process of constructing an operation data layer in a method for constructing an energy data warehouse system provided by an embodiment of the present invention
  • FIG. 7 is a schematic diagram of the implementation process of constructing a basic data layer in the method for constructing an energy data warehouse system provided by an embodiment of the present invention.
  • FIG. 8 is a schematic diagram of the implementation process of constructing a general data layer in the method for constructing an energy data warehouse system provided by an embodiment of the present invention.
  • Fig. 9 is a schematic diagram of an implementation process of constructing an application data layer in a construction method of an energy data warehouse system provided by an embodiment of the present invention.
  • the data warehouse is a subject-oriented, integrated, relatively stable data collection that reflects historical changes and is used to support management decision-making.
  • Traditional data warehouse vendors have relatively mature data warehouse products.
  • the existing general data warehouse solutions are very difficult to implement when applied to energy data warehouse systems. If it is a data warehouse without an industry model, it needs to be applied to the energy industry from Designing data models and data processing logic from scratch, and deploying them on general tools, is a huge workload and requires a very professional industry knowledge background. Otherwise, the established data model is difficult to meet the needs of energy data analysis, and it is currently oriented to the banking industry and The data model of traditional industries such as the telecommunications industry is different from that of the energy industry, and thus cannot be applied to the energy industry.
  • This embodiment provides an energy data warehouse system for the energy industry, which combines the business characteristics of the energy industry, is integrated based on the business data of the energy industry, is oriented to energy analysis topics, is relatively stable, and reflects historical changes. Energy data collection can be used for data analysis of energy companies to support management decisions of energy companies.
  • Fig. 1 is a schematic diagram of the energy data warehouse system provided by this embodiment.
  • the energy data warehouse system includes an operating data layer 11, a basic data layer 12, a general data layer 13, and an application data layer 14 which are sequentially connected.
  • the operation data layer 11 includes a detailed data table 111, and the energy data of the data source 10 is written into the detailed data table 111 after first data processing.
  • the basic data layer 12 includes basic data tables corresponding to each energy equipment type, and the data of the detailed data table 111 is written into the basic data table after the second data processing.
  • the general data layer 13 includes the data warehouse theme corresponding to the business analysis needs, and the data of the basic data table is written into the data warehouse theme after the third data processing.
  • the application data layer 14 includes a data mart corresponding to the business unit, and the data of the data warehouse theme is written into the data mart after the fourth data processing.
  • the energy data warehouse system for the energy industry provided by the embodiments of the present invention includes an operation data layer, a basic data layer, a general data layer, and an application data layer. It effectively solves the problem that the general data warehouse cannot be used in the energy field, helps to form a unified and standard data system, accelerates the speed of data processing and data analysis of energy equipment in the energy industry, and facilitates data analysts and data scientists based on high-quality Mass data for real-time and effective analysis.
  • the operating data layer 11 serves as a buffer layer, and incrementally stores newly generated or updated data between each data collection interval.
  • the operation data layer 11 it is necessary to perform data processing on the obtained energy data, and write the processed energy data correspondingly into the detailed data table, so as to realize the construction of the operation data layer 11.
  • Data source 10 is the data source of the energy data warehouse system.
  • the energy data sources of the energy data warehouse system mainly include energy equipment operating data 101, energy system configuration data 102, business data 103, Internet data 104, and third-party data 105, etc. Types of data are collected in different ways to build a data source.
  • Energy equipment operation data 101 Energy equipment operation data is the main data source of the energy data warehouse system. A large amount of equipment operation data is collected and uploaded to the message bus through the Internet of Things. The data collection program of the energy data warehouse system consumes data in real time from the message bus. Stored in the original data layer. The energy equipment operating data received from the message bus can be data in the standard json format collected by the Internet of Things. The main information includes information about the equipment, measurement attributes, measurement time, and measurement values.
  • Energy system configuration data 102 The configuration information about the energy system is the key information of the energy system data model and the main source of dimensional information in energy data analysis, such as the structure of the energy system, park information, system information, equipment attributes and relationship information . This part of the data is collected into the data source of the energy data warehouse system by synchronizing from the configuration database.
  • Business data 103 is data related to personnel, organization, and processes in the company's business development process, including employee information, department information, product information, purchasing information, sales information, project information, etc. The business data is synchronized through the business database and collected into the data source of the energy data warehouse system.
  • Internet data 104 In the process of energy data analysis, some external data is needed, such as weather data (temperature, humidity, wind direction, wind force, etc.) of the operating environment of the device, price data of different types of energy in different regions, etc. Internet data is collected into the data source of the data warehouse system through the Internet data crawling program.
  • weather data temperature, humidity, wind direction, wind force, etc.
  • Third-party data 105 During data analysis, in addition to the energy equipment operating data collected through the Internet of Things, a large number of third-party manufacturers have already collected and stored data in the third-party system.
  • the third-party data mainly includes equipment information data, equipment operation data, etc. This part of the data is collected into the data source of the energy data warehouse system through the third-party data interface service.
  • the energy data source of the energy data warehouse system may also be other, and it is not limited to the above-mentioned situation, and there is no limitation here.
  • the operating data layer 11 further includes an original data table 110. All collected energy data will enter the message queue.
  • the message program obtains energy data from the message queue and stores it in the distributed file system.
  • the batch program periodically loads the energy data from the distributed file system to the original data table through an external table.
  • the structure of the original data table is consistent with the format of the energy data received by the distributed file system.
  • performing the first data processing on the data at least includes:
  • the energy data has a time stamp when it is loaded into the original data table, which can facilitate subsequent data processing. For example, when loading energy data, you can determine which energy data is newly loaded data and which data is not newly loaded data according to the processing time of energy data. Only the newly loaded data requires further processing instead of newly loaded data. It can be written into the detailed data table after the corresponding processing has been carried out in the previous process.
  • the newly loaded data is partitioned according to the type of energy equipment and the acquisition time. Since the amount of energy data is usually relatively large, in the design of the detailed data table, combining the characteristics of energy data, partitioning the data acquisition time dimension and device type dimension can improve the efficiency of energy data storage and subsequent processing.
  • the structure of the detailed data sheet is based on time sequence. The structure mainly includes the following information: site information, device type, device identification (device ID), measurement attributes, measurement time, and measurement values.
  • the corresponding basic data table is mainly established with energy equipment as the unit.
  • the type of the basic data table in the basic data layer 12 can be set as required, for example, it can include the gas steam boiler basic data table 121, the air compressor basic data table 122, the energy meter basic data table 123, and business data.
  • the basic data table 124 and the environmental data basic data table 125 may also include other types of basic data tables, which are not limited to the above-mentioned situations.
  • the main things that need to be done include: classifying energy data according to the type of energy equipment, flattening the narrow table into a wide table, data standardization, and loading the basic data of the corresponding energy equipment Table.
  • performing the second data processing on the data at least includes:
  • the table name is FDM-TRAN, and its basic data table structure definition and description are as shown in Table 1:
  • the description information, status information and measurement information of the transformer are all concentrated in this basic data table, whether it is the energy data collected by the Internet of Things or the energy data accessed from a third-party interface.
  • Standardization is the definition model, and subsequent transformer-based analysis can extract data from it.
  • the basic data layer defines the basic data table structure for more than 100 types of commonly used energy equipment in the energy industry, which can support most of the data storage and analysis needs of the energy industry, and due to the weak correlation between the types of energy equipment It is very convenient to expand new types of energy equipment.
  • the basic data layer also includes environmental data obtained through data crawling programs and business data synchronized from business systems. These data are also standardized before entering the basic data layer for subsequent analysis. Provide a unified view.
  • the general data layer is business-oriented, and the data warehouse theme is designed from top to bottom according to the needs of business analysis.
  • the third data processing includes reading the data in the basic data table of each energy equipment type, aggregating the data, and so on.
  • the third data processing includes at least:
  • the general data layer 13 includes one of a data quality theme 131, a device status theme 132, a device energy consumption theme 133, and an enterprise energy use theme 134.
  • the general data layer 13 may also include other analysis topics, such as energy enterprise capacity analysis topics, enterprise energy efficiency analysis topics, equipment status operation trend analysis topics, equipment predictive maintenance analysis topics, etc., and are not limited to the above Situation.
  • the business analysis here will be carried out at different levels. Taking the topic of enterprise energy consumption analysis as an example, it will be in different time dimensions (hour level, day level, month level, grade), and different energy use unit dimensions (department, production line).
  • the calculation of the general data layer is a process of gradual aggregation from basic data.
  • the low-dimensional data is first calculated, and then the low-dimensional calculation results are aggregated into high-dimensional data. Take the calculation of electricity consumption and electricity costs as an example.
  • the electricity at the minute level is combined with the electricity price strategy (the electricity prices at different periods of peak and valley) to calculate the hourly electricity and electricity costs, and then the hourly electricity and the electricity fee are calculated.
  • the electricity fee is calculated by calculating the electricity and electricity charges at the day-level (or group period) level, and continues to be aggregated into monthly electricity and electricity charges, and finally aggregated into quarterly and annual electricity and electricity charges.
  • the application data layer 14 is a data layer that provides external application access to the calculation results of the energy data warehouse system. Applications include front-end products, report systems, algorithm platforms, operation analysis, etc., because the energy data warehouse system is mainly calculated and stored in large
  • the data platform integrates all energy data inside and outside the enterprise.
  • the application data layer 14 may include one or more of an energy consumption data mart 141, an energy supply data mart 142, an operation data mart 143, and an artificial intelligence data mart 144.
  • the fourth data processing includes at least:
  • the types of data marts include energy consumption data mart, energy supply data mart, operation data mart, artificial intelligence data mart, etc., of course, can also include For other types of markets, there are no restrictions here.
  • the energy data warehouse system provided in this embodiment also includes dimension data 15.
  • the dimension data 15 may include time dimension 151, geographic dimension 152, and user One or more of dimension 153, campus dimension 154, system dimension 155, and equipment dimension 156.
  • Dimensional data can provide references for data processing (including data association, data aggregation, data aggregation, etc.) in the general data layer, and can also provide dimensional information for data processing and data migration in the application data layer.
  • the dimensional data includes at least one of time dimension, geographic dimension, user dimension, campus dimension, system dimension, and device dimension. These dimensional data are the basis for data cleaning and data processing, as well as subsequent multi-dimensional modeling and data analysis. Basically, only when the data is cleaned and processed according to the standard, can the consistency and accuracy of the subsequent analysis be guaranteed.
  • the time dimension provides year (including natural year and corporate custom fiscal year), quarter, month (including natural month and custom calculation and settlement month), day (including natural day and custom team period), Dimension definitions at different levels such as hours and minutes; geographic dimensions provide definitions of different geographic dimensions including the country, regions, provinces (cities), cities (districts), districts, counties, and parks; according to user characteristics, provide the user’s industry and user Dimensional definitions such as level and user category; each energy system belongs to a park, and the corresponding dimensions are established based on the park and system information when the system is modeled, and the energy type to which the system belongs; the energy system’s capacity and energy-using equipment are based on equipment Categories, equipment categories, equipment manufacturers and other dimensions provide standards, such as the following common equipment types: air compressors, refrigerators, air conditioners, gas steam boilers, gas hot water boilers, transformers, steam meters, electricity meters, energy meters, thermometers, Pressure gauges, gas flow meters, liquid flow meters, differential pressure meters, etc.
  • common equipment types air compressors, refrigerator
  • the energy data warehouse system provided in this embodiment also includes a management tool 16, which includes a metadata management tool 161, Stream management tool 162, data collection tool 163, data processing tool 164, and data migration tool 165.
  • the metadata management tool 161 is used to provide a data map
  • the data collection tool 163 is used to provide data access services to collect the energy data of the data source to the operation data layer 11
  • the data processing tool 164 is used to Data processing provides tool support
  • the data migration tool 165 is used to migrate data from the data mart to an external system to provide tool support
  • the workflow management tool 162 is used to provide workflow support for data collection, data processing, and data migration.
  • Metadata management tool 161 There are various intricate energy data in the energy data warehouse system. In order to enable users to have a clear understanding of energy data, metadata management tools are provided to solve three data problems: data in each link What is it, where does the data from each link come from, and where does the data from each link go. Among them, what is the data of each link, you can search and view the name, type, length, business meaning, etc. of various data through tools, and you can view the meaning of the code for the coded data; where the data of each link comes from, to solve the problem of data traceability The problem is that all data in the energy data warehouse system has upstream data.
  • Workflow management tool 162 Because there are a large number of data collection, processing, and migration tasks in the energy data warehouse system, and there are dependencies between these tasks, in order to support the automatic periodic and orderly execution of the energy data warehouse system, workflow support is required.
  • One of the largest workflows in the energy data warehouse system is to start from incremental data collection to the file system, and perform the following tasks in sequence:
  • the workflow management tool in the energy data warehouse system provides task scheduling, workflow or task dependency management, topological relationship management, task execution strategy, execution result management, workflow and task re-execution support, etc.
  • the workflow system drives the entire energy data
  • the warehouse system operates in an orderly manner.
  • the above steps (1) to (3) constitute a workflow from the original data to the basic data, as shown in Figure 2.
  • the workflow from raw data to basic data runs once an hour, and is responsible for processing the data that arrived in the last hour into the basic data layer. With the help of workflow management tools, these tasks are organized together, and the scheduling period is set to every hour Executed at the 5th minute (it can also be executed at other times, there is no restriction here).
  • the first step of the workflow is to start multiple parallel tasks, and load the integrated station data, energy domain data, gas exchange station data, photovoltaic station data, heating station data, etc. into the original table respectively. When all these are executed in parallel After all tasks are completed, start the second task; write the newly loaded data into the detailed data table.
  • the basis for judging the newly loaded data is that the data processing time will be increased when the data is loaded in the first step, and whether it is judged according to the processing time.
  • Newly loaded data in the process of writing the data into the detail table, it is necessary to judge the data format.
  • For abnormal data write it into the abnormal log. Only the correct data will be written into the detail data table. At the same time, it will be based on the energy equipment to which the data belongs.
  • Types are partitioned to facilitate subsequent data processing based on the type of energy equipment; the third task is based on the type of energy equipment, and the data is processed and loaded into the basic data table based on the type of energy equipment (ie FDM table), one for each energy equipment type Separate tasks, these tasks are executed in parallel, each task mainly deals with two aspects of logic: one is to align the time of all data, because the data uploaded by the Internet of Things may have a time difference, the measurement value of the same device at the same time It may be uploaded at different time points. In order to facilitate subsequent analysis, it is necessary to align these data with the smallest time granularity (such as minutes), and process the second-level differences in the time information to the same minute, and put them into the same time dimension. In the same row; the second is to flatten the data, and put all the measurement information in the same time dimension into one row, which is convenient for subsequent indicator analysis and data comparison. After all the basic data of all energy equipment types are processed, the workflow ends.
  • FIG. 3 shows the data processing of the enterprise energy consumption report based on basic data. flow. There are many analysis based on basic data according to the analysis theme, such as transformer analysis, air compressor equipment analysis, gas boiler equipment analysis, enterprise energy structure analysis, capacity energy efficiency analysis, etc.
  • the various energy structures including electricity, heat, natural gas, steam, etc. used by the enterprise will be comprehensively analyzed, first based on the bottom numbers of various energy sources, that is, the meter of various energy measurements.
  • the workflow Based on the uploaded data, calculate the various energy consumption in different periods, and calculate the cost of different periods according to the pricing strategy (such as the peak and valley period electricity price of electric energy or the use step price, the step price of natural gas, etc.), and then calculate all costs according to the enterprise (Accounting unit) summarizes and generates aggregated data with the lowest time granularity (hour-level) (including the consumption and cost of each energy type), and then aggregates the aggregated data of the higher time granularity (day-level), and then summarizes higher Aggregate data of the first-level time granularity (month and grade), and then generate data marts for the companies that need reports, and finally export the data marts of each company to the reporting system for report generation. Marked by the task of exporting energy data to the reporting system, the workflow ends.
  • the pricing strategy such as the peak and valley period electricity price of electric energy or the use step price, the step price of natural gas, etc.
  • Data collection tool 163 The data collection tool mainly facilitates the access of various data sources to the energy data warehouse system.
  • the main functions of data collection are shown in Figure 4 and provide access services for various energy data.
  • IOT device data is directly collected from the message queue and pushed to the distributed file system of the data warehouse; business and configuration data are regularly synchronized from the business system; third-party data is regularly pulled from the third-party system through the interface call method, interface During the calling process, permission authentication and data acquisition are required.
  • data mapping is performed, and then pushed to the energy data warehouse system; Internet data is obtained through program crawling, and the crawled data is analyzed and pushed to the energy data warehouse system .
  • Data collection tasks need to be executed periodically, and task scheduling is supported by the aforementioned workflow management tools.
  • Data processing tool 164 For a large number of data processing tasks in the energy data warehouse system, the circulation, classification, cleaning, aggregation, and calculation of data among various layers are all data processing tasks. Such a large number of processing tasks need to be provided. Tool support, support interface configuration processing logic. In the process of data processing, it is necessary to distinguish between the business time and processing time of the data.
  • Business time is the actual time when the data occurs in the business process. For example, the bottom number of the metering device is the corresponding value at a specific moment. The time corresponding to the value is the business time; and the processing time refers to the system time when the processing program is executed. This time is generally later than the business time.
  • variables $ ⁇ biztime ⁇ , $ ⁇ bizdate ⁇ , $ ⁇ systime ⁇ and $ ⁇ sysdate ⁇ can be provided to represent business time, business date, processing time and processing date respectively, which can be called during data processing.
  • many built-in variables unique to the energy system have been added, such as the park $ ⁇ parkid ⁇ to which the data belongs, the system $ ⁇ systemid ⁇ to which the data belongs, and the site $ ⁇ stationid ⁇ to which the data belongs.
  • These are also available as built-in variables for data processing.
  • the processing logic is not completed in one step. There will be multiple modifications and adjustments. Therefore, data processing supports debugging. Each debugging task will check the grammar and generate the task execution plan. Through debugging The information confirms whether the task logic is correct. Data processing tasks need to be executed periodically, and task scheduling and dependencies are supported by the aforementioned workflow management tools.
  • Data Migration Tool 165 For data migration between the energy data warehouse system and external systems, a tool-based support is needed, which can regularly migrate the data of the original system to the energy data warehouse system, or migrate the result data of the energy data warehouse system processing To data application systems, such as report systems, etc.
  • the data migration tool supports data migration based on the database level and table level, sets the mapping relationship between the source table and the destination table, sets the incremental mode (incremental migration or full migration), sets the target data coverage strategy, etc., combined Task scheduling allows migration tasks to be executed periodically or triggered when upstream processing tasks are completed. Data migration tasks need to be executed periodically, and task scheduling and dependencies are supported by the aforementioned workflow management tools.
  • the energy data warehouse system includes an operation data layer, a basic data layer, a general data layer, and an application data layer, which effectively solves the problem that the general data warehouse cannot be applied to the energy field, and helps to form a unified, standard
  • the data system accelerates the speed of data processing and data analysis of energy equipment in the energy industry, facilitating data analysts and data scientists to conduct real-time and effective analysis based on high-quality massive data.
  • the energy data warehouse system of this embodiment absorbs the advantages of different construction methods at different levels, thereby helping to improve the performance of the energy data warehouse system.
  • Bill Inmon's enterprise data warehouse architecture method is used when building the basic data layer.
  • the dimensional data warehouse construction method proposed by Ralph Kimball is used to provide good support for the flexible and changeable analysis of the upper layer.
  • the energy data warehouse system of this embodiment includes dimensional data, which can provide standard support and a unified perspective for data processing and data multi-dimensional analysis, ensuring that data processing processes such as data cleaning and processing can be unified in accordance with standards, thereby ensuring Consistency and accuracy of subsequent analysis.
  • the energy data warehouse system of this embodiment includes management tools.
  • the management tools include metadata management tools, workflow management tools, data collection tools, data processing tools, and data migration tools, which can support data in the energy data warehouse system. Task-related configuration, operation, and monitoring, etc., help maintain the effective operation of the energy data warehouse system.
  • the energy data warehouse system provided in this embodiment can be constructed by the following energy data warehouse system construction method:
  • Step S21 Perform first data processing on the energy data table of the data source to obtain a detailed data table corresponding to the energy data to construct an operation data layer.
  • Step S22 Perform second data processing on the detailed data table according to the type of energy equipment to obtain a basic data table to construct a basic data layer.
  • Step S23 Perform third data processing on the basic data table according to the business analysis needs to obtain the data warehouse theme to construct a general data layer.
  • Step S24 Perform fourth data processing on the data warehouse theme according to the business unit, and obtain the data mart corresponding to the business unit to construct an application data layer.
  • Step S25 Construct dimensional data according to the energy data analysis dimension.
  • the dimensional data includes at least one of time dimension, geographic dimension, user dimension, campus dimension, system dimension, and device dimension. These dimensional data are the basis for data cleaning and data processing, as well as subsequent multi-dimensional modeling and data analysis. basis.
  • Step S26 Build a management tool, the management tool includes at least one of a metadata management tool, a workflow management tool, a data collection tool, a data processing tool, and a data migration tool.
  • step S21 includes the following steps:
  • Step S211 Load the energy data in the data source to the original data table.
  • Step S212 Analyze the original data table according to the processing time of the energy data, and determine the newly loaded data in the original data table.
  • Step S213 Determine whether the newly loaded data is abnormal data
  • Step S214 Perform format conversion on the newly loaded data to obtain intermediate data
  • Step S215 Add the newly loaded data to the data exception log
  • Step S216 partition the intermediate data according to the type of energy equipment and the acquisition time, and write the intermediate data into the detailed data table to construct an operation data layer.
  • step S22 includes the following steps:
  • Step S2221 Classify the data in the detailed data table according to the type of energy equipment to obtain data corresponding to each type of energy equipment.
  • Step S222 Perform time alignment on the data corresponding to the energy equipment according to the minimum time granularity to obtain the first data.
  • Step S223 Perform data flattening processing on the first data to obtain second data.
  • Step S224 Write the second data into the basic data table corresponding to each energy equipment type to construct a basic data layer.
  • step S23 includes the following steps:
  • Step S231 Determine the theme of the data warehouse according to the needs of business analysis.
  • Step S232 According to the theme of the data warehouse, read the data in the basic data table corresponding to each energy equipment type.
  • Step S233 Aggregate data corresponding to each energy equipment type with reference to the dimensional data to obtain aggregated data.
  • Step S234 Write the aggregated data into the data warehouse theme to construct a general data layer.
  • step S24 includes the following steps:
  • Step S241 Determine the data mart according to the business unit.
  • the types of the data mart include energy consumption data mart, energy supply data mart, operation data mart, artificial intelligence data mart, etc., of course, can also include other types of mart City, there is no restriction here.
  • Step S242 According to the data mart, classify the data in the data warehouse theme.
  • Step S243 Write the classified data in the data warehouse theme into the corresponding data mart, and determine the access authority to build an application data layer.
  • step S143 it may further include:
  • Step S244 Migrate the data mart to the report system to generate a report. That is, some businesses or reporting systems put data marts into their own databases through data migration to improve data access efficiency.

Landscapes

  • Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

一种能源数据仓库系统，包括：操作数据层(11)，包括细节数据表(111)，数据源的能源数据经第一数据处理后写入细节数据表(111)；基础数据层(12)，包括各能源设备类型对应的基础数据表，细节数据表(111)的数据经第二数据处理后写入基础数据表；通用数据层(13)，包括与业务分析需要对应的数据仓库主题，基础数据表的数据经第三数据处理后写入数据仓库主题；应用数据层(14)，包括与业务单元对应的数据集市，数据仓库主题的数据经第四数据处理后写入数据集市；有效解决了通用数据仓库无法应用于能源领域的问题，有助于形成统一、标准的数据体系，方便数据分析师和数据科学家基于高质量的海量数据进行实时有效分析。

Description

一种能源数据仓库系统 技术领域
本发明属于能源数据处理技术领域，尤其涉及一种能源数据仓库系统。
背景技术
数据仓库是一个面向主题的、集成的、相对稳定的、反映历史变化的数据集合，用于支持管理决策。数据仓库之父Bill Inmon提出的企业数据仓库架构和Ralph Kimball提出的维度数据仓库架构是两个主流数据仓库构建方法。
传统数据仓库厂商都有比较成熟的数据仓库产品，也有针对某些行业的数据模型。例如，在银行业，Teradata有自己的FS-LDM(Teradata Financial Services Logical Data Model)模型，而IBM有自己的BDWM(Banking Data Warehouse Model)模型；在电信业，Teradata有自己的CLDM(Teradata Communications Logical Data Model)模型，而IBM有TDWM(Telecom Data Warehouse Model)。然而，这些模型主要针对传统行业，并没有针对能源行业的数据模型，无法适应能源行业特点；并且这些数据模型都是基于传统关系数据库构建的，不能适应当前海量数据的实时和准实时的分析需求，也无法根据分析主题的变化而进行灵活处理。
技术问题
本发明实施例的目的在于提供一种能源数据仓库系统，以解决现有技术中没有针对能源行业的数据仓库系统的技术问题。
技术解决方案
本发明实施例提供了一种能源数据仓库系统，包括：
操作数据层，包括细节数据表，数据源的能源数据经第一数据处理后写入所述细节数据表；
基础数据层，包括各能源设备类型对应的基础数据表，所述细节数据表的数据经第二数据处理后写入所述基础数据表；
通用数据层，包括与业务分析需要对应的数据仓库主题，所述基础数据表的数据经第三数据处理后写入所述数据仓库主题；
应用数据层，包括与业务单元对应的数据集市，所述数据仓库主题的数据经第四数据处理后写入所述数据集市。
在一个实施例中，所述数据源的能源数据至少包括能源设备运行数据、能源系统配置数据、业务开展过程中的业务数据、通过网络获取的网络数据以及第三方系统提供的第三方数据中的一种。
在一个实施例中，所述操作数据层还包括原始数据表；
所述第一数据处理至少包括：
将所述能源数据加载至所述原始数据表；
对所述能源数据进行解析，以确定所述原始数据表中的新加载数据；
去除出所述新加载数据中的异常数据；
根据能源设备类型和获取时间对所述新加载数据进行分区。
在一个实施例中，所述细节数据表中至少包括站点信息、设备类型、设备标识、量测属性、量测时间以及量测值中的一项。
在一个实施例中，所述基础数据层至少包括燃气蒸汽锅炉基础数据表、空压机基础数据表、能量表基础数据表、业务数据基础数据表以及环境数据基础数据表中的一种。
在一个实施例中，所述第二数据处理至少包括：
根据能源设备类型对所述细节数据表中的数据进行分类，以获取每类能源设备对应的数据；
根据最小时间粒度对所述能源设备对应的数据进行时间对齐，以获取第一数据；
对所述第一数据进行数据扁平化处理，获得第二数据；
将所述第二数据写入各能源设备类型对应的基础数据表中。
在一个实施例中，所述通用数据层至少包括能源企业产能分析主题、用能企业用能分析主题、企业能效分析主题、设备状态运行趋势分析主题、设备预测性维护分析主题、企业数据接入质量分析主题中的一种。
在一个实施例中，所述第三数据处理至少包括：
根据确定的数据仓库主题，读取各能源设备类型对应的基础数据表中的数据；
对各能源设备类型对应的数据进行聚合，以获取聚合数据；
将所述聚合数据写入所述数据仓库主题。
在一个实施例中，所述应用数据层中至少包括用能数据集市、供能数据集市、运营数据集市以及人工智能数据集市中的一种。
在一个实施例中，所述第四数据处理至少包括：
根据确定的数据集市，对所述数据仓库主题中的数据进行分类；
将经过分类的所述数据仓库主题中的数据写入对应的数据集市中，并确定访问权限。
在一个实施例中，所述能源数据仓库系统还包括：
维度数据，至少包括时间维度、地理维度、用户维度、园区维度、系统维度、设备维度中的一种。
在一个实施例中，所述时间维度提供年、季度、月份、天、小时以及分钟的维度定义；
所述地理维度提供全国、大区、省、市、区县以及园区的维度定义；
所述用户维度提供用户所属行业、用户等级以及用户类别的维度定义；
所述设备维度中能源设备至少包括空气压缩机、制冷机、空调、燃气蒸汽锅炉、燃气热水锅炉、变压器、蒸汽表、电表、能量表、温度计、压力计、燃气流量表、液体流量表、压差计中的一种。
在一个实施例中，所述能源数据仓库系统还包括：
管理工具，包括元数据管理工具、工作流管理工具、数据采集工具、数据加工工具以及数据迁移工具。
在一个实施例中，所述元数据管理工具用于提供数据地图；
所述数据采集工具用于提供数据接入服务，以将所述数据源的能源数据采集至所述操作数据层；
所述数据加工工具用于对数据处理提供工具支持；
所述数据迁移工具用于将数据集市的数据迁移至外部系统提供工具支持；
所述工作流管理工具用于对数据采集、数据处理以及数据迁移提供工作流支持。
有益效果
本发明实施例提供的能源数据仓库系统的有益效果至少在于：本发明实施例提供的针对能源行业的能源数据仓库系统包括含操作数据层、基础数据层、通用数据层以及应用数据层，有效解决了通用数据仓库无法应用于能源领域的问题，有助于形成统一、标准的数据体系，加快能源行业对能源设备数据加工和数据分析的速度，方便数据分析师和数据科学家基于高质量的海量数据进行实时有效分析。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其它的附图。
图1是本发明实施例提供的能源数据仓库系统的示意图；
图2是本发明实施例提供的能源数据仓库系统中从原始数据到基础数据的工作流的实现流程图；
图3是本发明实施例提供的能源数据仓库系统中基于基础数据生成用能报表的工作流的实现流程图；
图4是本发明实施例提供的能源数据仓库系统中数据采集的示意图；
图5是本发明实施例提供的能源数据仓库系统构建方法的实现流程示意图；
图6是本发明实施例提供的能源数据仓库系统构建方法中构建操作数据层的实现流程示意图；
图7是本发明实施例提供的能源数据仓库系统构建方法中构建基础数据层的实现流程示意图；
图8是本发明实施例提供的能源数据仓库系统构建方法中构建通用数据层的实现流程示意图；
图9是本发明实施例提供的能源数据仓库系统构建方法中构建应用数据层的实现流程示意图。
本发明的实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、技术之类的具体细节，以便透彻理解本发明实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本发明。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本发明的描述。
为了说明本发明所述的技术方案，下面通过具体实施例来进行说明。
数据仓库是一个面向主题的、集成的、相对稳定的、反映历史变化的数据集合，用于支持管理决策。传统数据仓库厂商都有比较成熟的数据仓库产品，然而目前已有的通用数据仓库方案应用到能源数据仓库系统时实施难度巨大，如果是一个不带行业模型的数据仓库，应用到能源行业需要从零开始设计数据模型以及数据加工逻辑，并在通用工具上部署，工作量巨大，并且需要非常专业的行业知识背景，否则建立的数据模型很难满足能源数据分析的需求，而目前面向银行业和电信业等传统行业的数据模型由于与能源行业存在差异，从而无法应用到能源行业。
本实施例则提供了一种针对能源行业的能源数据仓库系统，其结合能源行业的业务特征，是基于能源行业的业务数据而集成的、面向能源分析主题的、相对稳定的、反映历史变化的能源数据集合，可以用于能源企业的数据分析，支撑能源企业的管理决策。
图1是本实施例提供的能源数据仓库系统的示意图。能源数据仓库系统包括依次连接的操作数据层11、基础数据层12、通用数据层13以及应用数据层14。其中，操作数据层11包括细节数据表111，数据源10的能源数据经第一数据处理后写入所述细节数据表111。基础数据层12包括各能源设备类型对应的基础数据表，细节数据表111的数据经第二数据处理后写入基础数据表。通用数据层13包括与业务分析需要对应的数据仓库主题，基础数据表的数据经第三数据处理后写入数据仓库主题。应用数据层14包括与业务单元对应的数据集市，数据仓库主题的数据经第四数据处理后写入数据集市。
本发明实施例提供的能源数据仓库系统构建方法的有益效果至少在于：本发明实施例提供的针对能源行业的能源数据仓库系统包括含操作数据层、基础数据层、通用数据层以及应用数据层，有效解决了通用数据仓库无法应用于能源领域的问题，有助于形成统一、标准的数据体系，加快能源行业对能源设备数据加工和数据分析的速度，方便数据分析师和数据科学家基于高质量的海量数据进行实时有效分析。
在本实施例中，操作数据层11作为一个缓冲层，增量存储每次数据采集间隔之间新产生或更新的数据。为了构建操作数据层11，需要对获取的能源数据进行数据处理，并将处理后的能源数据对应写入细节数据表中，从而实现对操作数据层11的构建。
数据源10是能源数据仓库系统的数据来源，能源数据仓库系统的能源数据来源主要包括能源设备运行数据101、能源系统配置数据102、业务数据103、 互联网数据104以及第三方数据105等，每种类型的数据通过不同的方式进行采集，从而构建数据源。
能源设备运行数据101：能源设备运行数据是能源数据仓库系统的主要数据来源，大量的设备运行数据通过物联网采集、上传到消息总线，能源数据仓库系统的数据采集程序从消息总线实时消费数据，存入原始数据层。从消息总线接收的能源设备运行数据可以是物联网采集的标准json格式的数据，主要信息包括设备所属信息、量测属性、量测时间和量测值等信息。
能源系统配置数据102：关于能源系统的配置信息，是能源系统数据模型的关键信息，也是能源数据分析中维度信息的主要来源，如能源系统的结构，园区信息、系统信息、设备属性和关系信息。这部分数据通过从配置库同步，采集到能源数据仓库系统的数据源中。
业务数据103：业务数据是公司业务开展过程中关于人员、组织、流程相关的数据，包括员工信息、部门信息、产品信息、采购信息、销售信息、项目信息等。业务数据通过业务库同步，采集到能源数据仓库系统的数据源中。
互联网数据104：在能源数据分析过程中，需要一些外部数据，例如设备运行环境的天气数据(温度、湿度、风向、风力等)，不同类型能源在不同区域的价格数据等。互联网数据通过互联网数据爬取程序采集到数据仓库系统的数据源中。
第三方数据105：在进行数据分析时，除了通过物联网上传采集的能源设备运行数据外，还需要大量第三方厂商已经采集并存储在第三方系统中的数据。第三方数据主要包括设备信息数据、设备运行数据等，这部分数据通过第三方数据接口服务，采集到能源数据仓库系统的数据源中。
当然，在其他实施例中，能源数据仓库系统的能源数据来源还可以为其他，并不仅限于上述的情形，此处不做限制。
在一个实施例中，操作数据层11还包括原始数据表110。所有采集到的能源数据会进入消息队列，消息程序从消息队列获取能源数据并存储在分布式文件系统中，批处理程序通过外部表的方式定期从分布式文件系统加载能源数据到原始数据表中。可选地，原始数据表结构与分布式文件系统接收到的能源数据格式一致。
由于能源数据在物联网采集阶段可能存在数据中断、数据多次上传、数据异常等情况，在从原始数据表向细节数据表加载时，可以对能源数据进行相应加工和处理。在本实施例中，对数据进行第一数据处理至少包括：
(1)将所述能源数据加载至所述原始数据表。
(2)对所述能源数据进行解析，以确定所述原始数据表中的新加载数据。为了方便后续处理和分析，能源数据在加载到原始数据表中时具有时间戳，从而可以方便后续的数据处理。例如，在对能源数据进行加载时，可以根据能源数据的处理时间来确定哪些能源数据是新加载数据，哪些数据不是新加载数据，只有新加载数据才需要进一步进行处理，而非新加载数据则可以在之前的过程 中已经进行相应的处理后写入细节数据表中。
(3)去除出所述新加载数据中的异常数据。为了提高处理效率，需要对新加载数据中的异常数据进行识别，从而可以从新加载数据中筛选出正常数据，避免对异常数据进行无用处理。而对于正常数据，则对其进行格式转换，使得其转换为与细节数据表相同格式的中间数据。
(4)根据能源设备类型和获取时间对所述新加载数据进行分区。由于能源数据的数据量通常比较多，在细节数据表的设计中，结合能源数据特点，对数据进行获取时间维度和设备类型维度的分区，可以提高能源数据存储和后续处理的高效性。细节数据表的结构以时序为基础，结构中主要包括以下信息：站点信息、设备类型、设备标识(设备ID)、量测属性、量测时间以及量测值等。
进一步地，由于同类能源设备之间的能源数据都是一致的，而不同类型能源设备之间的数据差异性较大，而后续对能源数据的分析主要以同类能源设备内部的分析为主，因此在基础数据层的设计上，主要以能源设备为单位建立相应的基础数据表。在本实施例中，基础数据层12中基础数据表的类型可以根据需要进行设置，例如可以包括燃气蒸汽锅炉基础数据表121、空压机基础数据表122、能量表基础数据表123、业务数据基础数据表124以及环境数据基础数据表125中的一种或多种。当然，在其他实施例中，基础数据层12还可以包括其他类型的基础数据表，并不仅限于上述的情形。
从细节数据表加工到基础数据表的过程中，主要需要做的事情包括：按照能源设备类型将能源数据进行分类、将窄表扁平化为宽表、数据标准化以及加载到相应能源设备的基础数据表中。在本实施例中，对数据进行第二数据处理至少包括：
(1)根据能源设备类型对所述细节数据表中的数据进行分类，以获取每类能源设备对应的数据。根据能源设备的类型不同，后续需要执行的业务分析也不相同，因此需要根据能源设备的类型对能源数据进行分类。
(2)根据最小时间粒度对所述能源设备对应的数据进行时间对齐，以获取第一数据。由于物联上传的能源数据可能会存在时差，同一个能源设备同一个时间的量测值可能在不同的时间点上传，因此为了便于后续分析，需要对这些能源数据进行最小时间粒度(如分钟级)的时间对齐，将时间信息中的秒级差异处理到同一分钟，并放入同一时间维度的同一行中，以获得第一数据。
(3)对所述第一数据进行数据扁平化处理，获得第二数据。通过将能源设备类型对应的数据进行扁平化处理，将同一时间维度的所有量测信息全部放入一行，以便于后续指标分析和数据对比。
(4)将所述第二数据写入各能源设备类型对应的基础数据表中。
在一个实施例中，以变压器为例，表名为FDM-TRAN，其基础数据表结构定义及说明如下表一所示：
Figure PCTCN2020130942-appb-000001
Figure PCTCN2020130942-appb-000002
Figure PCTCN2020130942-appb-000003
表一
通过上述基础数据表结构定义，变压器的描述信息、状态信息和量测信息全部集中在这张基础数据表中，无论是物联采集的能源数据，还是从第三方接口接入的能源数据，都规范化为该定义模式，后续基于变压器的分析可以全部从这里提取数据。
在本实施例中，基础数据层对能源行业超过100多类常用能源设备定义了基础数据表结构，能够支持能源行业的大部分数据存储和分析需要，并且由于各能源设备类型之间的弱关联性，扩展新的能源设备类型非常方便。基础数据层除了大量能源设备信息表外，还有通过数据爬取程序获取的环境数据以及从业务系统同步的业务数据等，这些数据也都是在进入基础数据层之前进行标准化处理，为后续分析提供统一的视图。
进一步地，通用数据层是面向业务的，根据业务分析需要，自顶向下设计数据仓库主题。在本实施例中，第三数据处理包括读取各能源设备类型的基础数据表中的数据、对数据进行聚合等。第三数据处理至少包括：
(1)根据确定的数据仓库主题，读取各能源设备类型对应的基础数据表中的数据；
(2)对各能源设备类型对应的数据进行聚合，以获取聚合数据；
(3)将所述聚合数据写入所述数据仓库主题。
在本实施例中，通用数据层13包括数据质量主题131、设备状态主题132、设备能耗主题133以及企业用能主题134中的一种。然，在其他实施例中，通用数据层13还可以包括其他分析主题，例如能源企业产能分析主题、企业能效分析主题、设备状态运行趋势分析主题、设备预测性维护分析主题等，并不仅限于上述的情形。这里的业务分析会在不同层次进行，以如企业用能分析主题为例，其会在不同的时间维度(小时级、天级、月级、年级)、不同用能单元维度(部门、产线、车间、班组)、不同用能设备维度(制冷、照明、加工设备)进行多维分析，主题设计需要能满足支持多维分析的需要。通用数据层的计算，是由基础数据逐步聚合汇总的过程，首先计算低维度的数据，再由低维度的计算结果聚合为高维度的数据。以计算用电量和用电费用为例，首先计算分钟级别的电量，由分钟级别的电量结合电价策略(尖峰平谷不同时段的电价)计算出小时级的电量和电费，再由小时级电量和电费计算出天级(或班组时段级)电量和电费，继续聚合为月级电量和电费，最终聚合为季度和年度的电量和电费。有了这些不同时间维度的聚合数据，在基于时间维度的分析时，就可以基于已经计算好的结果快速响应分析操作。
进一步地，应用数据层14是将能源数据仓库系统计算的结果对外提供应用访问的数据层，应用包括前端产品、报表系统、算法平台、运营分析等，由于能源数据仓库系统主要计算和存储在大数据平台，这里集成了企业内外的所有能源数据。应用数据层14中可以包括用能数据集市141、供能数据集市142、运营数据集市143以及人工智能数据集市144中的一种或多种。第四数据处理 至少包括：
(1)根据确定的数据集市，对所述数据仓库主题中的数据进行分类。在对数据进行分类前，根据业务单元，确定数据集市，数据集市的类型包括用能数据集市、供能数据集市、运营数据集市、人工智能数据集市等，当然还可以包括其他类型的集市，此处不做限制。
(2)将经过分类的所述数据仓库主题中的数据写入对应的数据集市中，并确定访问权限。这是由于有些能源数据并不是允许所有应用或分析人员访问，所以通过数据集市的方式，将不同业务单元需要的数据分开，将不同业务单元需要的数据放在相应的数据集市中，并在数据集市中对访问权限进行控制，保证数据的安全性。
进一步地，为了对能源数据的加工处理以及多维分析提供标准支撑和统一的视角，本实施例提供的能源数据仓库系统还包括维度数据15，维度数据15可以包括时间维度151、地理维度152、用户维度153、园区维度154、系统维度155、设备维度156中的一种或多种。维度数据可以为通用数据层进行数据加工(包括数据关联、数据聚合以及数据汇总等)提供参照，同时也可以为应用数据层进行数据加工、数据迁移提供维度信息。所述维度数据至少包括时间维度、地理维度、用户维度、园区维度、系统维度、设备维度中的一种，这些维度数据是进行数据清洗、数据处理的基础，也是后续多维建模和数据分析的基础，只有在数据清洗和处理时将数据按照标准进行统一，才能保证后续分析的一致性和准确性。
依据能源数据分析需要，时间维度提供年(包括自然年和企业自定义财年)、季度、月份(包括自然月和自定义计算和结算月)、天(包括自然日和自定义班组时段)、小时、分钟等不同层级的维度定义；地理维度提供包括全国、大区、省(市)、市(区)、区县、园区等不同地理维度的定义；按照用户特性，提供用户所属行业、用户等级、用户类别等维度定义；每个能源系统都属于一个园区，通过系统建模时的园区和系统信息，以及系统所属能源类型，建立相应的维度；能源系统的产能、用能设备，按照设备大类、设备小类、设备厂商等维度提供标准，例如以下常见设备类型：空气压缩机、制冷机、空调、燃气蒸汽锅炉、燃气热水锅炉、变压器、蒸汽表、电表、能量表、温度计、压力计、燃气流量计、液体流量表、压差计等。
进一步地，为了便于构建的能源数据仓库系统中数据和任务相关的配置、运行和监控等，本实施例提供的能源数据仓库系统还包括管理工具16，管理工具16包括元数据管理工具161、工作流管理工具162、数据采集工具163、数据加工工具164以及数据迁移工具165。其中，元数据管理工具161用于提供数据地图；数据采集工具163用于提供数据接入服务，以将所述数据源的能源数据采集至所述操作数据层11；数据加工工具164用于对数据处理提供工具支持；数据迁移工具165用于将数据集市的数据迁移至外部系统提供工具支持；工作流管理工具162用于对数据采集、数据处理以及数据迁移提供工作流支持。
元数据管理工具161：能源数据仓库系统中存在各类错综复杂的能源数据，为了能够让使用者对能源数据有清晰的认识，提供元数据管理工具，主要解决数据的三个问题：各个环节的数据是什么、各个环节的数据从哪里来、各个环节的数据到哪里去。其中，各个环节的数据是什么，可以通过工具搜索和查看各类数据的名称、类型、长度、业务含义等，对于编码数据可以查看编码的含义；各个环节的数据从哪里来，解决数据溯源的问题，对于能源数据仓库系统中的所有数据，都有上游数据，当需要分析一个数据是否有问题、一个任务失败原因、一个计算指标不正确的原因时，都需要对上游数据进行溯源，沿着上游节点逐步上溯，直到找出问题出现的源头；各个环节的数据到哪里去，解决数据影响分析的问题，当需要对能源数据仓库系统中任何一个环节的计算逻辑、处理方式进行改动时，需要评估改动对现有系统的影响，需要找到所有依赖该节点的后续数据处理流程和数据，评估修改是否可行。能源数据的三个问题结合在一起，就形成了一张全局的错综复杂的数据网络，这个数据网络就是数据地图。元数据管理工具可以提供这样一个能够看到数据仓库全局的数据地图，通过这个地图，能够看到能源数据仓库系统全部数据的点，可以查看每个点的说明，可以在任意一个点进行溯源分析和影响分析。
工作流管理工具162：由于能源数据仓库系统中存在大量的数据采集、加工以及迁移任务，并且这些任务之间存在依赖关系，为了支持能源数据仓库系统自动化周期性有序执行，需要工作流支持。在能源数据仓库系统中最大的一个工作流就是从数据增量采集到文件系统开始，依次执行以下任务：
(1)加载能源数据到能源数据仓库系统的原始数据表；
(2)从原始数据表解析获取新加载数据并写入细节数据表；
(3)从细节数据表读取数据进行清洗转换，并按照能源设备类型分别加载到各基础数据表；
(4)并行执行通用数据层各数据仓库主题计算任务，并把计算结果写入各数据仓库主题；
(5)将各数据仓库主题结果按照业务需要加载到数据集市；
(6)将数据集市迁移到各业务数据库。
能源数据仓库系统中的工作流管理工具提供任务调度、工作流或任务依赖关系管理、拓扑关系管理、任务执行策略、执行结果管理、工作流和任务重新执行支持等，工作流系统驱动整个能源数据仓库系统有序正常运转。
上述第(1)至第(3)步构成一个从原始数据到基础数据的工作流，如图2所示。从原始数据到基础数据的工作流每小时运行一次，负责将最近一个小时到达的数据加工到基础数据层中，借助工作流管理工具，将这些任务组织在一起，并设置调度周期为每个小时的第5分钟执行(也可以为其他时间执行，此处不做限制)。工作流的第一步是启动多个并行任务，分别将综合站数据、用能域数据、换气站数据、光伏站数据、供热站数据等分别加载到原始表，当所有这些并行执行的任务全部完成以后，启动第二步任务；将新加载的数据写入细节 数据表，判断新加载数据的依据是，在第一步装载数据时会增加数据的处理时间，依据处理时间判断是否为新装载数据，在将数据写入细节表的过程中，需要进行数据格式的判断，对于异常数据，写入异常日志，只有正确的数据才会写入细节数据表，同时会根据数据所属能源设备类型进行分区，方便后续基于能源设备类型的数据处理；第三步任务基于能源设备类型，将数据处理并加载到基于能源设备类型的基础数据表中(即FDM表)，每个能源设备类型一个单独的任务，这些任务并行执行，每个任务主要处理两方面的逻辑：一是将所有数据的时间进行对齐，由于物联上传的数据可能会存在时差，同一个设备同一个时间的量测值可能在不同的时间点上传，为了便于后续分析，需要对这些数据进行最小时间粒度(如分钟级)的时间对齐，将时间信息中的秒级差异处理到同一分钟，并放入同一时间维度的同一行中；二是将数据扁平化处理，将同一时间维度的所有量测信息全部放入一行，便于后续指标分析和数据对比。所有能源设备类型的基础数据全部处理完成之后，此次工作流结束。
上述第(4)至第(6)步构成多个面向分析主题的工作流，这些工作流都依赖第一个工作流，图3给出了基于基础数据进行企业用能报表的数据加工的工作流。基于基础数据的分析按分析主题会有很多，如变压器分析、空压机设备分析、燃气锅炉设备分析、企业用能结构分析、产能能效分析等。在企业用能报告分析中，会将企业所用到的各类能源结构(包括电能、热能、天然气、蒸汽等)进行综合分析，首先基于各类能源的表底数，即各类能源测量的表计上传的数据，计算出不同时段的各类能源用量，根据计价策略(如电能的按尖峰平谷时段电价或用量阶梯电价、天然气的阶梯价格等)计算出不同时段的费用，再将所有费用按企业(核算单元)进行汇总，生成最低时间粒度(小时级)的聚合数据(包括各能源类型的用量和费用)，接下来汇总高一级时间粒度(天级)的聚合数据，再汇总出更高一级时间粒度(月级和年级)的聚合数据，再给需要报表的企业分别生成数据集市，最终把各企业的数据集市导出到报表系统，用于生成报表。以能源数据导出到报表系统任务为标志，工作流结束。
数据采集工具163：数据采集工具主要为各类数据源接入能源数据仓库系统提供便利，数据采集的主要功能如图4所示，提供各类能源数据的接入服务。物联设备数据直接从消息队列采集，推送到数据仓库的分布式文件系统；业务和配置数据从业务系统进行定期数据同步；第三方数据通过接口调用方式，定期从第三方系统拉取数据，接口调用过程中，需要进行权限认证、数据获取，提取数据之后进行数据映射，之后推送到能源数据仓库系统；互联网数据通过程序爬取方式获取，对爬取的数据进行解析并推送到能源数据仓库系统。数据采集任务需要定期周期性执行，任务的调度由前述工作流管理工具支持。
数据加工工具164：对于能源数据仓库系统中大量的数据加工任务，数据在各层之间的流转、分类、清洗、聚合、计算等都是一个个的数据加工任务，如此大量的加工任务需要提供工具支持，支持界面化配置加工逻辑。在数据加工过程中，需要根据数据的业务时间和处理时间进行区分，业务时间是数据在 业务过程中的实际发生时间，如表计设备的表底数，都是在某个具体时刻对应的数值，与数值对应的时间就是业务时间；而处理时间是指加工程序执行时的系统时间，这个时间一般比业务时间晚，用这两个时间进行不同需求的数据处理，如需要计算某一天的能源量，需要根据业务时间来计算，如果某一天的任务失败或存在问题，需要重新执行，就需要根据处理时间来计算。在数据加工中，可以提供${biztime}、${bizdate}、${systime}和${sysdate}几个变量分别表示业务时间、业务日期、处理时间和处理日期，可以在数据加工时调用，另外还增加了许多能源系统特有的内置变量，如数据所属园区${parkid}、所属系统${systemid}、所属站点${stationid}等，这些也作为数据加工的内置变量供选择。在配置数据加工任务的过程中，对于加工逻辑并不是一步完成的，其中会有多次修改调整的过程，所以数据加工支持调试，每次调试任务会检查语法、生成任务执行计划等，通过调试信息确认任务逻辑是否正确。数据加工任务需要定期周期性执行，任务的调度及依赖关系由前述工作流管理工具支持。
数据迁移工具165：对于能源数据仓库系统与外部系统之间的数据迁移，需要一个工具化支持，可以将原系统的数据定期迁移到能源数据仓库系统，或者将能源数据仓库系统加工的结果数据迁移到数据应用系统，如报表系统等。数据迁移工具支持基于库级别、表级别的数据迁移，设定源表与目的表之间的映射关系，设置增量方式(增量迁移还是全量迁移)，设置对目标的数据覆盖策略等，结合任务调度，让迁移任务周期性执行或者在上游加工任务完成时触发执行。数据迁移任务需要定期周期性执行，任务的调度及依赖关系由前述工作流管理工具支持。
本实施例提供的能源数据仓库系统的有益效果至少在于：
(1)本实施例提供的能源数据仓库系统包括操作数据层、基础数据层、通用数据层以及应用数据层，有效解决了通用数据仓库无法应用于能源领域的问题，有助于形成统一、标准的数据体系，加快能源行业对能源设备数据加工和数据分析的速度，方便数据分析师和数据科学家基于高质量的海量数据进行实时有效分析。
(2)本实施例的能源数据仓库系统在不同层面吸纳了不同构建方法的优势，从而有助于提高能源数据仓库系统的性能，例如在构建基础数据层时采用Bill Inmon的企业数据仓库架构方法，为能源数据仓库系统打造坚实的基础底座；而在通用数据层和应用数据层则采用Ralph Kimball提出的维度数据仓库构建方法，为上层灵活多变的分析提供良好支撑。
(3)本实施例的能源数据仓库系统包括维度数据，从而可以为数据加工和数据多维分析提供标准支撑和统一的视角，确保了数据清洗和处理等数据加工过程可以按照标准进行统一，从而确保后续分析的一致性和准确性。
(4)本实施例的能源数据仓库系统包括管理工具，管理工具包括元数据管理工具、工作流管理工具、数据采集工具、数据加工工具以及数据迁移工具中，可以支持能源数据仓库系统中的数据和任务相关的配置、运行与监控等，有助 于维持能源数据仓库系统的有效运行。
请参阅图5，本实施例提供的能源数据仓库系统可以通过下述的能源数据仓库系统构建方法进行构建：
步骤S21：对数据源的能源数据表进行第一数据处理，获得所述能源数据对应的细节数据表，以构建操作数据层。
步骤S22：根据能源设备类型对所述细节数据表进行第二数据处理，获取基础数据表，以构建基础数据层。
步骤S23：根据业务分析需要对所述基础数据表进行第三数据处理，获取数据仓库主题，以构建通用数据层。
步骤S24：根据业务单元对所述数据仓库主题进行第四数据处理，获取所述业务单元对应的数据集市，以构建应用数据层。
步骤S25：根据能源数据分析维度，构建维度数据。所述维度数据至少包括时间维度、地理维度、用户维度、园区维度、系统维度、设备维度中的一种，这些维度数据是进行数据清洗、数据处理的基础，也是后续多维建模和数据分析的基础。
步骤S26：构建管理工具，所述管理工具至少包括元数据管理工具、工作流管理工具、数据采集工具、数据加工工具以及数据迁移工具中的一种。
请参阅图6，在一个实施例中，步骤S21包括以下步骤：
步骤S211：加载所述数据源中的能源数据至原始数据表。
步骤S212：根据所述能源数据的处理时间解析所述原始数据表，确定所述原始数据表中的新加载数据。
步骤S213：判断所述新加载数据是否为异常数据；
若所述新加载数据不是异常数据，则：
步骤S214：对所述新加载数据进行格式转换，获取中间数据；
若所述新加载数据是异常数据，则：
步骤S215：将所述新加载数据加入数据异常日志；
为了提高处理效率，需要对新加载数据中的异常数据进行识别，从而可以从新加载数据中筛选出正常数据，避免对异常数据进行无用处理。而对于正常数据，则对其进行格式转换，使得其转换为与细节数据表相同格式的中间数据。
步骤S216：根据能源设备类型和获取时间对所述中间数据进行分区，将所述中间数据写入细节数据表中，以构建操作数据层。
请参阅图7，在一个实施例中，步骤S22包括以下步骤：
步骤S2221：根据能源设备类型对所述细节数据表中的数据进行分类，以获取每类能源设备对应的数据。
步骤S222：根据最小时间粒度对所述能源设备对应的数据进行时间对齐，以获取第一数据。
步骤S223：对所述第一数据进行数据扁平化处理，获得第二数据。
步骤S224：将所述第二数据写入各能源设备类型对应的基础数据表中，以 构建基础数据层。
请参阅图8，在一个实施例中，步骤S23包括以下步骤：
步骤S231：根据业务分析需要，确定数据仓库主题。
步骤S232：根据数据仓库主题，读取各能源设备类型对应的基础数据表中的数据。
步骤S233：参照维度数据对各能源设备类型对应的数据进行聚合，以获取聚合数据。
步骤S234：将所述聚合数据写入所述数据仓库主题，以构建通用数据层。
请参阅图9，在一个实施例中，步骤S24包括以下步骤：
步骤S241：根据业务单元，确定数据集市，数据集市的类型包括用能数据集市、供能数据集市、运营数据集市、人工智能数据集市等，当然还可以包括其他类型的集市，此处不做限制。
步骤S242：根据所述数据集市，对所述数据仓库主题中的数据进行分类。
步骤S243：将经过分类的所述数据仓库主题中的数据写入对应的数据集市中，并确定访问权限，以构建应用数据层。
进一步地，为了提升数据访问的效率，步骤S143之后还可以包括：
步骤S244：将所述数据集市迁移至报表系统中，以生成报表。即有些业务或报表系统通过数据迁移的方式将数据集市放入自己的数据库中，提升数据访问效率。
应理解，上述实施例中各步骤的序号的大小并不意味着执行顺序的先后，各过程的执行顺序应以其功能和内在逻辑确定，而不应对本发明实施例的实施过程构成任何限定。
以上所述实施例仅用以说明本发明的技术方案，而非对其限制；尽管参照前述实施例对本发明进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本发明各实施例技术方案的精神和范围，均应包含在本发明的保护范围之内。

Claims (14)

  1. 一种能源数据仓库系统，其特征在于，包括：
    操作数据层，包括细节数据表，数据源的能源数据经第一数据处理后写入所述细节数据表；
    基础数据层，包括各能源设备类型对应的基础数据表，所述细节数据表的数据经第二数据处理后写入所述基础数据表；
    通用数据层，包括与业务分析需要对应的数据仓库主题，所述基础数据表的数据经第三数据处理后写入所述数据仓库主题；
    应用数据层，包括与业务单元对应的数据集市，所述数据仓库主题的数据经第四数据处理后写入所述数据集市。
  2. 如权利要求1所述的能源数据仓库系统，其特征在于，所述数据源的能源数据至少包括能源设备运行数据、能源系统配置数据、业务开展过程中的业务数据、通过网络获取的网络数据以及第三方系统提供的第三方数据中的一种。
  3. 如权利要求1所述的能源数据仓库系统，其特征在于，所述操作数据层还包括原始数据表；
    所述第一数据处理至少包括：
    将所述能源数据加载至所述原始数据表；
    对所述能源数据进行解析，以确定所述原始数据表中的新加载数据；
    去除出所述新加载数据中的异常数据；
    根据能源设备类型和获取时间对所述新加载数据进行分区。
  4. 如权利要求1所述的能源数据仓库系统，其特征在于，所述细节数据表中至少包括站点信息、设备类型、设备标识、量测属性、量测时间以及量测值中的一项。
  5. 如权利要求1所述的能源数据仓库系统，其特征在于，所述基础数据层至少包括燃气蒸汽锅炉基础数据表、空压机基础数据表、能量表基础数据表、业务数据基础数据表以及环境数据基础数据表中的一种。
  6. 如权利要求1所述的能源数据仓库系统，其特征在于，所述第二数据处理至少包括：
    根据能源设备类型对所述细节数据表中的数据进行分类，以获取每类能源设备对应的数据；
    根据最小时间粒度对所述能源设备对应的数据进行时间对齐，以获取第一数据；
    对所述第一数据进行数据扁平化处理，获得第二数据；
    将所述第二数据写入各能源设备类型对应的基础数据表中。
  7. 如权利要求1所述的能源数据仓库系统，其特征在于，所述通用数据层至少包括能源企业产能分析主题、用能企业用能分析主题、企业能效分析主题、设备状态运行趋势分析主题、设备预测性维护分析主题、企业数据接入质量分析主题中的一种。
  8. 如权利要求1所述的能源数据仓库系统，其特征在于，所述第三数据处理至少包括：
    根据确定的数据仓库主题，读取各能源设备类型对应的基础数据表中的数据；
    对各能源设备类型对应的数据进行聚合，以获取聚合数据；
    将所述聚合数据写入所述数据仓库主题。
  9. 如权利要求1所述的能源数据仓库系统，其特征在于，所述应用数据层中至少包括用能数据集市、供能数据集市、运营数据集市以及人工智能数据集市中的一种。
  10. 如权利要求1所述的能源数据仓库系统，其特征在于，所述第四数据处理至少包括：
    根据确定的数据集市，对所述数据仓库主题中的数据进行分类；
    将经过分类的所述数据仓库主题中的数据写入对应的数据集市中，并确定访问权限。
  11. 如权利要求1所述的能源数据仓库系统，其特征在于，所述能源数据仓库系统还包括：
    维度数据，至少包括时间维度、地理维度、用户维度、园区维度、系统维度、设备维度中的一种。
  12. 如权利要求11所述的能源数据仓库系统，其特征在于，所述时间维度提供年、季度、月份、天、小时以及分钟的维度定义；
    所述地理维度提供全国、大区、省、市、区县以及园区的维度定义；
    所述用户维度提供用户所属行业、用户等级以及用户类别的维度定义；
    所述设备维度中能源设备至少包括空气压缩机、制冷机、空调、燃气蒸汽锅炉、燃气热水锅炉、变压器、蒸汽表、电表、能量表、温度计、压力计、燃气流量表、液体流量表、压差计中的一种。
  13. 如权利要求1所述的能源数据仓库系统，其特征在于，所述能源数据仓库系统还包括：
    管理工具，包括元数据管理工具、工作流管理工具、数据采集工具、数据加工工具以及数据迁移工具。
  14. 如权利要求13所述的能源数据仓库系统，其特征在于，所述元数据管理工具用于提供数据地图；
    所述数据采集工具用于提供数据接入服务，以将所述数据源的能源数据采集至所述操作数据层；
    所述数据加工工具用于对数据处理提供工具支持；
    所述数据迁移工具用于将数据集市的数据迁移至外部系统提供工具支持；
    所述工作流管理工具用于对数据采集、数据处理以及数据迁移提供工作流支持。
PCT/CN2020/130942 2019-12-31 2020-11-23 一种能源数据仓库系统 WO2021135727A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201911401106.1A CN111159154A (zh) 2019-12-31 2019-12-31 一种能源数据仓库系统
CN201911401106.1 2019-12-31

Publications (1)

Publication Number Publication Date
WO2021135727A1 true WO2021135727A1 (zh) 2021-07-08

Family

ID=70559377

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/130942 WO2021135727A1 (zh) 2019-12-31 2020-11-23 一种能源数据仓库系统

Country Status (2)

Country Link
CN (1) CN111159154A (zh)
WO (1) WO2021135727A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111159154A (zh) * 2019-12-31 2020-05-15 新奥数能科技有限公司 一种能源数据仓库系统
CN111104394A (zh) * 2019-12-31 2020-05-05 新奥数能科技有限公司 一种能源数据仓库系统构建方法及装置
CN112487106A (zh) * 2020-11-26 2021-03-12 万翼科技有限公司 基于建筑信息模型的数据分层方法及相关装置

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130204836A1 (en) * 2012-02-08 2013-08-08 Electronics And Telecommunications Research Institute Method and apparatus for constructing data warehouse to manage mass building energy information
CN108280084A (zh) * 2017-01-06 2018-07-13 上海前隆信息科技有限公司 一种数据仓库的构建方法、系统及服务器
CN111104394A (zh) * 2019-12-31 2020-05-05 新奥数能科技有限公司 一种能源数据仓库系统构建方法及装置
CN111159154A (zh) * 2019-12-31 2020-05-15 新奥数能科技有限公司 一种能源数据仓库系统

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102117303A (zh) * 2009-12-31 2011-07-06 潘晓梅 一种专利数据分析方法和系统
KR101253335B1 (ko) * 2011-06-07 2013-04-10 백승호 데이터 웨어하우스를 이용한 데이터베이스 구축 방법 및 그 시스템
CN106294521B (zh) * 2015-06-12 2019-09-06 交通银行股份有限公司 数据存储方法及数据仓库系统
CN104915456A (zh) * 2015-07-03 2015-09-16 宁夏隆基宁光仪表有限公司 一种基于数据分析系统下的海量用电数据挖掘方法
CN105843880A (zh) * 2016-03-21 2016-08-10 中国矿业大学 一种基于多数据集市的煤矿多维数据仓库系统
CN106339509A (zh) * 2016-10-26 2017-01-18 国网山东省电力公司临沂供电公司 一种基于大数据技术的电网运营数据共享系统
CN106874353A (zh) * 2016-12-28 2017-06-20 合肥智畅信息科技有限公司 一种操作型商业智能应用系统
CN109033113B (zh) * 2017-06-12 2021-07-30 北京京东尚科信息技术有限公司 数据仓库和数据集市的管理方法及装置
CN110019462B (zh) * 2017-11-14 2021-09-03 南方电网科学研究院有限责任公司 电力科研生产数据分析方法、装置、系统及存储介质
CN109669975B (zh) * 2018-11-09 2020-12-18 成都数之联科技有限公司 一种工业大数据处理系统及方法
CN109597850A (zh) * 2018-11-22 2019-04-09 四川省烟草公司成都市公司 烟草综合信息数据加工储存平台及数据加工方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20130204836A1 (en) * 2012-02-08 2013-08-08 Electronics And Telecommunications Research Institute Method and apparatus for constructing data warehouse to manage mass building energy information
CN108280084A (zh) * 2017-01-06 2018-07-13 上海前隆信息科技有限公司 一种数据仓库的构建方法、系统及服务器
CN111104394A (zh) * 2019-12-31 2020-05-05 新奥数能科技有限公司 一种能源数据仓库系统构建方法及装置
CN111159154A (zh) * 2019-12-31 2020-05-15 新奥数能科技有限公司 一种能源数据仓库系统

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
ANONYMOUS |: "[Talking about data warehouse] How to elegantly design data layering", 20 October 2017 (2017-10-20), XP055825360, Retrieved from the Internet <URL:https://bigdata.51cto.com/art/201710/554810.htm> *
LIU QIFEI: "Research of Oil and Gas Production Report Based on Business Intelligence", CHINESE MASTER'S THESES FULL-TEXT DATABASE, TIANJIN POLYTECHNIC UNIVERSITY, CN, 1 January 2013 (2013-01-01), CN, XP055827905, ISSN: 1674-0246 *
LU SHUAISHUAI: "Study on Distributed Data Warehouse of Oil and Gas Drilling Information Based on Big Data", CHINESE MASTER'S THESES FULL-TEXT DATABASE, TIANJIN POLYTECHNIC UNIVERSITY, CN, 1 January 2018 (2018-01-01), CN, XP055827911, ISSN: 1674-0246 *

Also Published As

Publication number Publication date
CN111159154A (zh) 2020-05-15

Similar Documents

Publication Publication Date Title
WO2021135177A1 (zh) 一种能源数据仓库系统构建方法及装置
WO2021135727A1 (zh) 一种能源数据仓库系统
CN107194621B (zh) 一种供水管网管理系统和方法
CN111291076B (zh) 基于大数据的异常用水监测报警系统及其构建方法
CN102708149A (zh) 数据质量管理方法和系统
CN103646329B (zh) 一种跨区跨省电力交易运营管控方法
CN113822547B (zh) 一种精细化需求侧管理综合信息平台系统及管理方法
CN111160865A (zh) 一种工作流管理方法及装置
CN116681187B (zh) 一种基于企业经营数据的企业碳配额预测方法
CN111552686B (zh) 一种电力数据质量评估方法及其装置
CN105335822A (zh) 一种面向大数据分析的智能电网统一数据模型建模方法
Dong et al. Forecasting smart meter energy usage using distributed systems and machine learning
CN112486959A (zh) 基于图数据库技术的电网动态拓扑结构智能识别方法
CN111127186A (zh) 一种基于大数据技术的客户信用等级评价体系的使用方法
CN115905319B (zh) 一种海量用户电费异常的自动识别方法及系统
Li et al. Prediction of china’s housing price based on a novel grey seasonal model
Xueliang et al. Study of power grid planning integrated information platform based on big-data technology
CN116226293A (zh) 一种电力客户画像生成管理的方法及系统
CN113807573A (zh) 一种基于数据管理和三维搜索的施工预测分析方法及系统
Greenberg A new approach to analyze information contained in a model
CN112001551A (zh) 一种基于大用户电量信息的地市电网售电量预测方法
Li Economic Development Analysis of China’s Coastal Industries Based on Deep Learning and SDA Model Weighting
Lele et al. Research on design and implementation of smart energy system in digital economy era
Li et al. Exploration Practice of Data Mastery Traceability Algorithm Based on Knowledge Graph in Data Governance of Electric Power Industry
CN113935549B (zh) 一种综合智慧能源优化调度系统

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20910159

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20910159

Country of ref document: EP

Kind code of ref document: A1