CN111104394A

CN111104394A - Energy data warehouse system construction method and device

Info

Publication number: CN111104394A
Application number: CN201911401126.9A
Authority: CN
Inventors: 李品新; 徐锡明; 黄博淘; 吴建波
Original assignee: Xinao Shuneng Technology Co Ltd
Current assignee: Xinao Shuneng Technology Co Ltd
Priority date: 2019-12-31
Filing date: 2019-12-31
Publication date: 2020-05-05
Also published as: WO2021135177A1

Abstract

The invention is suitable for the technical field of energy data processing, and provides a method and a device for constructing an energy data warehouse system, wherein the method comprises the following steps: performing first data processing on energy data of a data source to obtain a detail data table corresponding to the energy data so as to construct an operation data layer; performing second data processing on the detail data table according to the type of the energy equipment to obtain a basic data table so as to construct a basic data layer; performing third data processing on the basic data table according to the business analysis requirement to obtain a data warehouse theme so as to construct a general data layer; and performing fourth data processing on the data warehouse theme according to the service unit to obtain a data mart corresponding to the service unit so as to construct an application data layer. According to the energy data warehouse system, the problem that a universal data warehouse cannot be applied to the field of energy is effectively solved by constructing the energy data warehouse system aiming at the energy industry, a unified and standard data system is formed, and the speed of processing and analyzing the energy equipment data in the energy industry is increased.

Description

Energy data warehouse system construction method and device

Technical Field

The invention belongs to the technical field of energy data processing, and particularly relates to a method and a device for constructing an energy data warehouse system.

Background

A data warehouse is a theme-oriented, integrated, relatively stable data collection that reflects historical changes used to support administrative decisions. The enterprise data warehouse architecture proposed by the father Bill Inmon of the data warehouse and the dimensional data warehouse architecture proposed by Ralph Kimball are two mainstream data warehouse construction methods.

Traditional data warehouse vendors all have relatively mature data warehouse products and also have data models for certain industries. For example, in the banking industry, Teradata has its own FS-LDM (Teradata Financial Services logical Data model), while IBM has its own BDWM (banking Data Warehouse model); in the telecommunications industry, Teradata has its own CLDM (Teradata Communications Logical Data model), while IBM has a TDWM (TelecomData Warehouse model). However, these models mainly aim at the traditional industry, do not aim at the data model of the energy industry, can not adapt to the characteristics of the energy industry; and the data models are constructed based on the traditional relational database, cannot adapt to the real-time and quasi-real-time analysis requirements of the current mass data, and cannot be flexibly processed according to the change of an analysis theme.

Disclosure of Invention

In view of this, embodiments of the present invention provide a method and an apparatus for constructing an energy data warehouse system, a terminal device, and a computer-readable storage medium, so as to solve the technical problem that there is no data warehouse system for the energy industry in the prior art.

A first aspect of an embodiment of the present invention provides a method for constructing an energy data warehouse system, including:

performing first data processing on an energy data table of a data source to obtain a detail data table corresponding to the energy data so as to construct an operation data layer;

performing second data processing on the detail data table according to the type of the energy equipment to obtain a basic data table so as to construct a basic data layer;

performing third data processing on the basic data table according to business analysis requirements to obtain a data warehouse theme so as to construct a general data layer;

and performing fourth data processing on the data warehouse theme according to the service unit to obtain a data mart corresponding to the service unit so as to construct an application data layer.

A second aspect of an embodiment of the present invention provides an energy data warehouse system construction apparatus, including:

the operation data layer construction module is used for carrying out first data processing on an energy data table of a data source to obtain a detailed data table corresponding to the energy data so as to construct an operation data layer;

the basic data layer building module is used for carrying out second data processing on the detail data table according to the type of the energy equipment to obtain a basic data table so as to build a basic data layer;

the general data layer construction module is used for performing third data processing on the basic data table according to business analysis requirements to obtain a data warehouse theme so as to construct a general data layer;

and the application data layer construction module is used for performing fourth data processing on the data warehouse theme according to the service units to acquire the data marts corresponding to the service units so as to construct an application data layer.

A third aspect of the embodiments of the present invention provides a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor implements the steps of the energy data warehouse system construction method when executing the computer program.

A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium storing a computer program, which when executed by a processor, implements the steps of the above-described energy data warehouse system construction method.

The energy data warehouse system construction method provided by the embodiment of the invention has the beneficial effects that at least: according to the embodiment of the invention, the operation data layer, the basic data layer, the general data layer and the application data layer are constructed, so that an energy data warehouse system aiming at the energy industry can be constructed, the problem that the general data warehouse cannot be applied to the energy field is effectively solved, a unified and standard data system is formed, the speed of processing and analyzing the energy equipment data in the energy industry is increased, and a data analyst and a data scientist can conveniently perform real-time effective analysis on high-quality mass data.

Drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed for the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.

Fig. 1 is a first schematic flow chart illustrating an implementation of a method for constructing an energy data warehouse system according to an embodiment of the present invention;

fig. 2 is a schematic flow chart illustrating an implementation process of constructing an operation data layer in the energy data warehouse system construction method according to the embodiment of the invention;

fig. 3 is a schematic flow chart illustrating an implementation of building a basic data layer in the method for building an energy data warehouse system according to the embodiment of the present invention;

fig. 4 is a schematic flow chart illustrating an implementation process of building a generic data layer in the energy data warehouse system building method according to the embodiment of the present invention;

fig. 5 is a schematic flow chart illustrating an implementation process of constructing an application data layer in the energy data warehouse system construction method according to the embodiment of the present invention;

fig. 6 is a schematic flow chart of an implementation of the energy data warehouse system construction method according to the embodiment of the present invention;

fig. 7 is a schematic diagram of an energy data warehouse system constructed by the energy data warehouse system construction method provided by the embodiment of the invention;

fig. 8 is a flowchart of implementing a workflow from raw data to basic data in a method for constructing an energy data warehouse system according to an embodiment of the present invention;

fig. 9 is a flowchart illustrating an implementation of a workflow for generating an energy consumption report based on basic data in the energy data warehouse system construction method according to the embodiment of the present invention;

fig. 10 is a schematic diagram of data acquisition in the energy data warehouse system construction method according to the embodiment of the invention;

fig. 11 is a first schematic diagram of an energy data warehouse system construction apparatus provided in an embodiment of the present invention;

fig. 12 is a second schematic diagram of an energy data warehouse system construction apparatus provided in an embodiment of the present invention;

fig. 13 is a schematic diagram of a terminal device according to an embodiment of the present invention.

Detailed Description

In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.

In order to explain the technical means of the present invention, the following description will be given by way of specific examples.

A data warehouse is a theme-oriented, integrated, relatively stable data collection that reflects historical changes used to support administrative decisions. Traditional data warehouse manufacturers all have mature data warehouse products, however, the implementation difficulty of the existing general data warehouse scheme applied to an energy data warehouse system is huge, if the data warehouse is a data warehouse without an industry model, the application to the energy industry needs to design a data model and data processing logic from scratch, and the data warehouse is deployed on a general tool, the workload is huge, and a very professional industry knowledge background is needed, otherwise, the established data model hardly meets the requirement of energy data analysis, and the existing data model facing the traditional industries such as the banking industry, the telecommunication industry and the like cannot be applied to the energy industry due to the difference with the energy industry.

The embodiment provides a method for constructing an energy data warehouse system for an energy industry, which can be combined with business features of the energy industry, wherein the constructed energy data warehouse system is an energy data set which is integrated based on business data of the energy industry, is oriented to an energy analysis theme, is relatively stable, and reflects historical changes, and can be used for data analysis of energy enterprises and support management decisions of the energy enterprises.

Fig. 1 is a method for constructing an energy data warehouse system according to this embodiment, including:

step S11: and carrying out first data processing on an energy data table of a data source to obtain a detail data table corresponding to the energy data so as to construct an operation data layer.

The operational data layer acts as a buffer layer to incrementally store newly generated or updated data between each data acquisition interval. In order to construct the operation data layer, the acquired energy data needs to be subjected to data processing, and the processed energy data is correspondingly written into the detail data table, so that the construction of the operation data layer is realized.

The data source is a data source of the energy data warehouse system, the energy data source of the energy data warehouse system mainly comprises energy equipment operation data, energy system configuration data, business data, internet data, third-party data and the like, and each type of data is collected in different modes, so that the data source is constructed.

Energy equipment operation data: the energy equipment operation data are main data sources of the energy data warehouse system, a large amount of equipment operation data are collected and uploaded to a message bus through the Internet of things, and a data collection program of the energy data warehouse system consumes data from the message bus in real time and stores the data into an original data layer. The energy equipment operation data received from the message bus can be data in a standard json format acquired by the internet of things, and the main information comprises information such as equipment affiliated information, measurement attributes, measurement time and measurement values.

Energy system configuration data: configuration information about the energy system is key information of an energy system data model and is also a main source of dimensional information in energy data analysis, such as the structure of the energy system, park information, system information, equipment attribute and relationship information. The data is collected from the data sources of the energy data warehouse system by synchronizing the data from the configuration library.

Service data: the business data is related to personnel, organizations and processes in the process of company business development, and comprises employee information, department information, product information, purchase information, sales information, project information and the like. And the service data are synchronized through the service library and are collected into a data source of the energy data warehouse system.

Internet data: in the energy data analysis process, some external data are needed, such as weather data (temperature, humidity, wind direction, wind power, etc.) of the equipment operation environment, price data of different types of energy in different areas, and the like. The internet data is collected into a data source of the data warehouse system through an internet data crawling program.

Third party data: when data analysis is performed, besides uploading collected energy equipment operation data through the internet of things, a large amount of data collected and stored in a third-party system by a third-party manufacturer is also needed. The third-party data mainly comprises equipment information data, equipment operation data and the like, and the data are acquired to a data source of the energy data warehouse system through third-party data interface services.

Of course, in other embodiments, the energy data sources of the energy data warehouse system may be other sources, and are not limited to the above-mentioned situations.

In this embodiment, the first data processing at least includes loading the energy data into the original data table, parsing the energy data, screening abnormal data in the energy data, partitioning the energy data according to the type of the energy device and the acquisition time, and the like. Referring to fig. 2, step S11 may include the following steps:

step S111: and loading the energy data in the data source to an original data table.

All the collected energy data can enter a message queue, a message program obtains the energy data from the message queue and stores the energy data in a distributed file system, and a batch processing program loads the energy data from the distributed file system to an original data table regularly in an external table mode. Optionally, the raw data table structure is in accordance with the energy data format received by the distributed file system.

Because the energy data may have the conditions of data interruption, data uploading for multiple times, data abnormity and the like in the acquisition stage of the internet of things, the energy data can be correspondingly processed and processed when the original data table is loaded to the detail data table.

Step S112: and analyzing the original data table according to the processing time of the energy data, and determining new loading data in the original data table.

For convenience of subsequent processing and analysis, the energy data has a time stamp when being loaded into the original data table, so that subsequent data processing can be facilitated. For example, when the energy data is loaded, it may be determined according to the processing time of the energy data which energy data is newly loaded data and which data is not newly loaded data, only the newly loaded data needs to be further processed, and the non-newly loaded data may be written into the detail data table after corresponding processing has been performed in the previous process.

Step S113: judging whether the newly loaded data is abnormal data;

if the new loaded data is not abnormal data, then:

step S114: carrying out format conversion on the newly loaded data to obtain intermediate data;

if the new loaded data is abnormal data, then:

step S115: adding the new loaded data into a data exception log;

in order to improve the processing efficiency, the abnormal data in the newly loaded data needs to be identified, so that the normal data can be screened from the newly loaded data, and useless processing on the abnormal data is avoided. And for normal data, format conversion is performed so that it is converted into intermediate data in the same format as the detailed data table.

Step S116: partitioning the intermediate data according to the type of the energy equipment and the acquisition time, and writing the intermediate data into a detail data table to construct an operation data layer.

Because the data volume of the energy data is usually more, in the design of the detail data table, the characteristics of the energy data are combined, the data are partitioned into the time dimension and the equipment type dimension, and the high efficiency of energy data storage and subsequent processing can be improved. The structure of the detail data table is based on time sequence, and mainly comprises the following information: site information, device type, device identification (device ID), measurement attributes, measurement time, measurement values, and the like.

Step S12: and performing second data processing on the detail data table according to the type of the energy equipment to obtain a basic data table so as to construct a basic data layer.

Because the energy data among the same type of energy equipment are consistent, the data difference among different types of energy equipment is large, and the subsequent analysis of the energy data mainly takes the internal analysis of the same type of energy equipment as a main part, a corresponding basic data table is established mainly by taking the energy equipment as a unit on the design of a basic data layer.

In this embodiment, in the process from the detail data table to the basic data table, the main things that need to be done include: the method comprises the steps of classifying energy data according to the type of the energy equipment, flattening a narrow table into a wide table, standardizing the data and loading the data into a basic data table of the corresponding energy equipment.

In the present embodiment, the second data processing includes at least sorting data of the detail data table, time-aligning the data, and performing data flattening processing and the like. Referring to fig. 3, step S12 may include the following steps:

step S121: and classifying the data in the detail data table according to the type of the energy equipment so as to obtain the data corresponding to each type of energy equipment. According to the different types of the energy equipment, the subsequent service analysis required to be executed is different, so that the energy data needs to be classified according to the types of the energy equipment.

Step S122: and time alignment is carried out on the data corresponding to the energy equipment according to the minimum time granularity so as to obtain first data.

Because the energy data uploaded by the internet of things may have time differences, and the measurement values of the same energy device at the same time may be uploaded at different time points, in order to facilitate subsequent analysis, time alignment of minimum time granularity (such as minute level) needs to be performed on the energy data, the second level difference in the time information is processed to the same minute and is placed in the same row of the same time dimension, so as to obtain the first data.

Step S123: and carrying out data flattening processing on the first data to obtain second data.

The data corresponding to the type of the energy equipment are flattened, and all measurement information of the same time dimension is put into a line, so that subsequent index analysis and data comparison are facilitated.

Step S124: and writing the second data into a basic data table corresponding to each energy equipment type to construct a basic data layer.

In one embodiment, taking the transformer as an example, the table name is FDM-TRAN, and the structure of the basic data table is defined and described as the following table one:

watch 1

Through the definition of the basic data table structure, the description information, the state information and the measurement information of the transformer are all concentrated in the basic data table, no matter the energy data collected by the internet of things or the energy data accessed from a third-party interface are normalized to be in the definition mode, and the data can be extracted from the definition mode through the subsequent analysis based on the transformer.

In this embodiment, the basic data layer defines a basic data table structure for more than 100 types of common energy devices in the energy industry, can support most of data storage and analysis needs of the energy industry, and is very convenient to expand new energy device types due to weak correlation among the energy device types. Besides a large amount of energy equipment information tables, the basic data layer also comprises environmental data acquired through a data crawling program, service data synchronized from a service system and the like, and the data are subjected to standardized processing before entering the basic data layer, so that a unified view is provided for subsequent analysis.

Step S13: and performing third data processing on the basic data table according to the business analysis requirement to obtain a data warehouse theme so as to construct a general data layer.

The universal data layer is service-oriented, and designs a data warehouse theme from top to bottom according to service analysis requirements. In the present embodiment, the third data processing includes reading data in the basic data table of each energy device type, aggregating the data, and the like. Referring to fig. 4, step S13 may include the following steps:

step S131: and determining the data warehouse subject according to the business analysis requirement.

Step S132: and reading data in the basic data table corresponding to each energy equipment type according to the data warehouse theme.

Step S133: and aggregating data corresponding to each energy equipment type by referring to the dimension data to obtain aggregated data.

Step S134: and writing the aggregated data into the data warehouse subject to construct a universal data layer.

Specifically, in the energy field, the data warehouse topics focused on include an energy enterprise capacity analysis topic, an energy consumption enterprise energy consumption analysis topic, an enterprise energy efficiency analysis topic, an equipment state operation trend analysis topic, an equipment predictive maintenance analysis topic, an enterprise data access quality analysis topic, and the like. The business analysis can be performed at different levels, for example, enterprise energy consumption analysis subject matters can be subjected to multi-dimensional analysis at different time dimensions (hour level, day level, month level and year level), different energy consumption unit dimensions (departments, production lines, workshops and teams) and different energy consumption equipment dimensions (refrigeration, lighting and processing equipment), and the subject design needs can meet the needs of supporting the multi-dimensional analysis. The calculation of the general data layer is a process of gradually aggregating and summarizing basic data, and the low-dimensional data is calculated firstly and then aggregated into high-dimensional data by the calculation result of the low dimension. Taking calculation of electricity consumption and electricity consumption as an example, firstly, calculating electricity quantity of a minute level, calculating electricity quantity and electricity consumption of an hour level by combining the electricity quantity of the minute level with an electricity consumption strategy (electricity consumption in different periods of time of peak valley leveling), then calculating electricity quantity and electricity consumption of a day level (or in a group period level) by using the electricity quantity and the electricity consumption of the hour level, continuously aggregating the electricity quantity and the electricity consumption into electricity quantity and electricity consumption of a month level, and finally aggregating the electricity quantity and the electricity consumption of a quarter and a year. With the aggregated data of the different time dimensions, the analysis operation can be responded quickly based on the calculated result when the analysis is based on the time dimension.

Step S14: and performing fourth data processing on the data warehouse theme according to the service unit to obtain a data mart corresponding to the service unit so as to construct an application data layer.

The application data layer is a data layer which provides application access to the external application of the calculated result of the energy data warehouse system, the application comprises a front-end product, a report system, an algorithm platform, operation analysis and the like, and the energy data warehouse system is mainly used for calculating and storing the energy data warehouse system on a big data platform, so that all energy data inside and outside an enterprise are integrated.

In this embodiment, the fourth data processing includes classifying data of the data warehouse topic, determining access rights, and the like. Referring to fig. 5, step S14 may include the following steps:

step S141: determining a data mart according to the service unit, wherein the type of the data mart includes an energy-using data mart, an energy-supplying data mart, an operation data mart, an artificial intelligence data mart and the like, and of course, other types of data marts can be included, which is not limited herein.

Step S142: and classifying the data in the data warehouse topic according to the data mart.

Step S143: and writing the classified data in the data warehouse topics into corresponding data marts, and determining access rights to construct an application data layer. The energy data are not allowed to be accessed by all applications or analysis personnel, so that the data required by different service units are separated in a data mart mode, the data required by different service units are placed in the corresponding data marts, and the access authority is controlled in the data marts, so that the safety of the data is ensured.

Further, in order to improve the efficiency of data access, step S143 may be followed by:

step S144: and migrating the data marts to a reporting system to generate reports. Namely, some business or report systems place data marts into their databases in a data migration manner, thereby improving data access efficiency.

Further, in order to provide standard support and a unified view for processing and multidimensional analysis of the energy data, the method for constructing the energy data warehouse system provided by the embodiment further includes:

step S15: and analyzing the dimension according to the energy data to construct dimension data. The dimension data can provide reference for data processing (including data association, data aggregation, data summarization and the like) of the general data layer, and also can provide dimension information for data processing and data migration of the application data layer. The dimension data at least comprises one of a time dimension, a geographical dimension, a user dimension, a park dimension, a system dimension and an equipment dimension, the dimension data is the basis for data cleaning and data processing and the basis for subsequent multidimensional modeling and data analysis, and the consistency and the accuracy of the subsequent analysis can be ensured only by unifying the data according to standards during the data cleaning and processing.

According to the energy data analysis needs, the time dimension provides dimension definitions of different levels of years (including natural years and enterprise self-defined property years), quarters, months (including natural months and self-defined calculation and settlement months), days (including natural days and self-defined team and group periods), hours, minutes and the like; the geographic dimension provides definitions of different geographic dimensions including nationwide, large district, province (city), city (district), district and county, garden and the like; providing dimension definitions such as industries to which users belong, user grades, user categories and the like according to user characteristics; each energy system belongs to a park, and corresponding dimensionality is established through park and system information during system modeling and the energy type to which the system belongs; the energy production and energy utilization equipment of the energy system provide standards according to the dimensions of equipment major categories, equipment minor categories, equipment manufacturers and the like, such as the following common equipment types: the system comprises an air compressor, a refrigerator, an air conditioner, a gas-fired steam boiler, a gas-fired hot water boiler, a transformer, a steam meter, an electric meter, an energy meter, a thermometer, a pressure meter, a gas flowmeter, a liquid flow meter, a differential pressure meter and the like.

Referring to fig. 6, further, in order to facilitate configuration, operation, monitoring, and the like related to data and tasks in the constructed energy data warehouse system, the method for constructing an energy data warehouse system according to the embodiment further includes:

step S16: and constructing management tools, wherein the management tools at least comprise one of metadata management tools, workflow management tools, data acquisition tools, data processing tools and data migration tools.

Metadata management tool: there are various complicated energy data in the energy data warehouse system, in order to let the user have clear understanding to energy data, provide the metadata management tool, mainly solve three problems of data: what the data of each link is, where the data of each link comes from, and where the data of each link goes. The data of each link is what, names, types, lengths, business meanings and the like of various data can be searched and checked through tools, and the coded meanings can be checked for coded data; the data of each link comes from, the problem of data tracing is solved, all data in the energy data warehouse system have upstream data, when the data need to be analyzed whether have problems, a task failure reason and a reason that a calculation index is incorrect, the upstream data need to be traced, and the data are gradually traced along an upstream node until the source of the problems is found out; and when the calculation logic and the processing mode of any link in the energy data warehouse system need to be changed, the influence of the change on the existing system needs to be evaluated, all subsequent data processing flows and data depending on the node need to be found, and whether the modification is feasible or not is evaluated. The three problems of energy data are combined together to form a global complex data network, and the data network is a data map. The metadata management tool can provide a data map which can see the whole situation of the data warehouse, all data points of the energy data warehouse system can be seen through the map, the description of each point can be seen, and the source tracing analysis and the influence analysis can be carried out at any point.

A workflow management tool: because a large number of data acquisition, processing and migration tasks exist in the energy data warehouse system and dependency relationships exist among the tasks, workflow support is needed to support the automatic periodic and orderly execution of the energy data warehouse system. The largest workflow in the energy data warehouse system is from data increment acquisition to the file system, and the following tasks are sequentially executed:

(1) loading energy data to an original data table of an energy data warehouse system;

(2) analyzing the original data table to obtain new loading data and writing the new loading data into a detail data table;

(3) reading data from the detail data table for cleaning conversion, and loading the data into each basic data table according to the type of the energy equipment;

(4) executing the calculation tasks of the data warehouse topics of the general data layer in parallel, and writing the calculation results into the data warehouse topics;

(5) loading the subject results of each data warehouse into a data mart according to business requirements;

(6) and migrating the data marts to each service database.

The workflow management tool in the energy data warehouse system provides task scheduling, workflow or task dependency relationship management, topological relationship management, task execution strategies, execution result management, workflow and task re-execution support and the like, and the workflow system drives the whole energy data warehouse system to operate normally and orderly.

The above steps (1) to (3) constitute a workflow from the original data to the basic data, as shown in fig. 8. The workflow from the original data to the basic data is run once per hour, and is responsible for processing the data which arrives in the last hour into the basic data layer, organizing the tasks together by means of the workflow management tool, and setting the scheduling period to be executed in the 5 th minute of each hour (or in other times, without limitation). The first step of the workflow is to start a plurality of parallel tasks, respectively load comprehensive station data, energy domain data, ventilation station data, photovoltaic station data, heating station data and the like to an original table, and start the second step of tasks after all the tasks executed in parallel are completed; writing the newly loaded data into a detail data table, judging whether the newly loaded data is newly loaded according to the judgment that the newly loaded data is processed by increasing the processing time of the data when the data is loaded in the first step, judging whether the newly loaded data is the newly loaded data or not according to the processing time, judging the data format in the process of writing the data into the detail table, writing abnormal logs into abnormal data, writing the accurate data into the detail data table, and partitioning according to the type of energy equipment to which the data belongs so as to facilitate the subsequent data processing based on the type of the energy equipment; and a third step of processing and loading data into a basic data table (namely an FDM table) based on the energy equipment types, wherein each energy equipment type has a separate task, the tasks are executed in parallel, and each task mainly processes two logic aspects: firstly, time of all data is aligned, due to the fact that time difference may exist in data uploaded by an internet of things, measurement values of the same equipment at the same time may be uploaded at different time points, for convenience of subsequent analysis, time alignment of minimum time granularity (such as minute level) needs to be carried out on the data, second level difference in time information is processed to the same minute and is placed in the same row of the same time dimension; and secondly, the data is flattened, all the measurement information of the same time dimension is put into one line, and the subsequent index analysis and data comparison are facilitated. After all the basic data of all the energy equipment types are processed, the workflow is ended.

The steps (4) to (6) form a plurality of analysis subject-oriented workflows, the workflows depend on the first workflow, and fig. 9 shows a workflow for data processing of enterprise energy use reports based on basic data. Analysis based on basic data can be many according to analysis subjects, such as transformer analysis, air compressor equipment analysis, gas boiler equipment analysis, enterprise energy utilization structure analysis, energy production and energy efficiency analysis and the like. In the enterprise energy report analysis, various energy structures (including electric energy, heat energy, natural gas, steam and the like) used by an enterprise are comprehensively analyzed, firstly, based on the number of the meter bases of various energy sources, namely data uploaded by a meter measured by various energy sources, various energy source usage in different time periods is calculated, expenses in different time periods are calculated according to pricing strategies (such as electricity price or usage step electricity price according to peak and valley time periods of the electric energy, step price of the natural gas and the like), all the expenses are summarized according to the enterprise (accounting unit), aggregated data (including usage and expenses of various energy types) with the lowest time granularity (hour level) are generated, then aggregated data with higher time granularity (day level) is summarized, aggregated data with higher time granularity (month level and year level) is summarized, and then a data market is respectively generated for the enterprise needing a report, and finally exporting the data marts of the enterprises to a report system for generating reports. And taking the task of exporting the energy data to the report system as a mark, and finishing the workflow.

A data acquisition tool: the data acquisition tool mainly facilitates access of various data sources to the energy data warehouse system, and the main function of data acquisition is as shown in fig. 10, so that access service of various energy data is provided. The data of the internet of things equipment is directly collected from the message queue and pushed to a distributed file system of a data warehouse; the service and the configuration data are subjected to periodic data synchronization from the service system; the third-party data is periodically pulled from the third-party system through an interface calling mode, authority authentication and data acquisition are required in the interface calling process, data mapping is carried out after the data are extracted, and then the data are pushed to the energy data warehouse system; the Internet data is acquired in a program crawling mode, and the crawled data is analyzed and pushed to an energy data warehouse system. The data collection task needs to be periodically executed, and the task scheduling is supported by the workflow management tool.

A data processing tool: for a large number of data processing tasks in the energy data warehouse system, the data flow, classification, cleaning, aggregation, calculation and the like among all layers are data processing tasks one by one, and thus the large number of processing tasks need to provide tool support and support the interface configuration processing logic. In the data processing process, the data needs to be distinguished according to the service time and the processing time of the data, the service time is the actual occurrence time of the data in the service process, such as the number of the table bottom of a meter device, which is a numerical value corresponding to a specific moment, and the time corresponding to the numerical value is the service time; the processing time refers to the system time when the processing program is executed, the time is generally later than the service time, the two times are used for processing data with different requirements, for example, the energy amount of a certain day needs to be calculated, the calculation needs to be carried out according to the service time, and if the task of the certain day fails or has problems and needs to be executed again, the calculation needs to be carried out according to the processing time. In data processing, variables $ { biztime }, $ { biztate }, $ { system } and $ { sysdate } can be provided to respectively represent service time, service date, processing time and processing date, can be called in data processing, and a plurality of energy system-specific built-in variables are added, such as a park $ { parkid } to which data belongs, a system $ { systemid } to which data belongs, a site $ { stable } and the like, and are also used as built-in variables for data processing to select. In the process of configuring the data processing task, the processing logic is not completed in one step, and the processing logic is modified and adjusted for many times, so that the data processing supports debugging, each debugging task checks grammar, generates a task execution plan and the like, and whether the task logic is correct is confirmed through debugging information. The data processing tasks need to be periodically executed, and the scheduling and the dependency relationship of the tasks are supported by the workflow management tool.

A data migration tool: for data migration between the energy data warehouse system and an external system, a tool support is needed, data of an original system can be periodically migrated to the energy data warehouse system, or result data processed by the energy data warehouse system is migrated to a data application system, such as a report system. The data migration tool supports data migration based on a library level and a table level, sets a mapping relation between a source table and a target table, sets an increment mode (increment migration or full migration), sets a data coverage strategy for a target and the like, and combines task scheduling to enable a migration task to be executed periodically or triggered to be executed when an upstream processing task is completed. The data migration task needs to be periodically executed, and the scheduling and the dependency relationship of the task are supported by the workflow management tool.

Referring to fig. 7, the energy data warehouse system constructed by the method for constructing an energy data warehouse system according to the present embodiment includes an operation data layer (ODS), a basic data layer (FDM), a generic data layer (GDM), and an application data layer (ADM), where the operation data layer performs data acquisition from a data source, and data is processed by the operation data layer, the basic data layer, the generic data layer, and the application data layer in sequence, and then can be used by an external system for data application. The energy data warehouse system further includes dimensional Data (DIM) to provide references for data processing by the generic data layer and dimensional information for data processing and migration by the application data layer. The energy data warehouse system also comprises a management tool, and the data and task related configuration, operation, monitoring and the like in the energy data warehouse system are convenient to construct.

The energy data warehouse system construction method provided by the embodiment has the beneficial effects that at least:

(1) the embodiment is through constructing operation data layer, basic data layer, general data layer and application data layer to can construct the energy data warehouse system to the energy industry, effectively solve the problem that general data warehouse can't be applied to the energy field, help forming unified, standard data system, accelerate the energy industry to energy equipment data processing and data analysis's speed, make things convenient for data analyst and data scientist to carry out effective analysis in real time based on high-quality mass data.

(2) The embodiment takes advantages of different construction methods in different layers, thereby being beneficial to improving the performance of the energy data warehouse system, for example, a Bill Inmon enterprise data warehouse architecture method is adopted when constructing a basic data layer, and a solid basic base is created for the energy data warehouse system; and a dimension data warehouse construction method proposed by RalphKimball is adopted in the general data layer and the application data layer, so that good support is provided for upper-layer flexible and variable analysis.

(3) According to the embodiment, the dimension data is constructed, so that standard support and a unified visual angle can be provided for data processing and data multi-dimensional analysis, the data processing processes such as data cleaning and processing can be unified according to the standard, and the consistency and accuracy of subsequent analysis are ensured.

(4) In the embodiment, by constructing a management tool, the management tool includes a metadata management tool, a workflow management tool, a data acquisition tool, a data processing tool, and a data migration tool, and can support configuration, operation, monitoring, and the like related to data and tasks in the energy data warehouse system, which is helpful for maintaining effective operation of the energy data warehouse system.

It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

Referring to fig. 11, the present embodiment is further directed to provide an energy data warehouse system building apparatus, which includes an operation data layer building module 21, a basic data layer building module 22, a general data layer building module 23, and an application data layer building module 24. The operation data layer construction module 21 is configured to perform first data processing on an energy data table of a data source to obtain a detail data table corresponding to the energy data, so as to construct an operation data layer; the basic data layer building module 22 is configured to perform second data processing on the detail data table according to the type of the energy device, obtain a basic data table, and build a basic data layer; the general data layer construction module 23 is configured to perform third data processing on the basic data table according to business analysis needs, and obtain a data warehouse theme to construct a general data layer; the application data layer construction module 24 is configured to perform fourth data processing on the data warehouse theme according to the service unit, and obtain a data mart corresponding to the service unit, so as to construct an application data layer.

Referring to fig. 12, further, the energy data warehouse system building apparatus further includes a dimension data building module 25, where the dimension data building module 25 is configured to analyze dimensions according to the energy data and build dimension data, where the dimension data at least includes one of a time dimension, a geographic dimension, a user dimension, a campus dimension, a system dimension, and a device dimension.

Further, the energy data warehouse system construction device further includes a management tool construction module 26, and the management tool construction module 26 is configured to construct a management tool, where the management tool includes at least one of a metadata management tool, a workflow management tool, a data collection tool, a data processing tool, and a data migration tool.

Fig. 13 is a schematic diagram of a terminal device according to an embodiment of the present invention. As shown in fig. 13, the terminal device 3 of this embodiment includes: a processor 30, a memory 31 and a computer program 32, such as an energy data warehouse system building program, stored in said memory 31 and operable on said processor 30. The processor 30, when executing the computer program 32, implements the steps of the energy data warehouse system construction method embodiments described above, such as the steps S11 to S16 shown in fig. 1 to 6. Alternatively, the processor 30, when executing the computer program 32, implements the functions of each module/unit in each device embodiment described above, for example, the functions of the modules 21 to 26 shown in fig. 11 to 12.

Illustratively, the computer program 32 may be partitioned into one or more modules/units that are stored in the memory 31 and executed by the processor 30 to implement the present invention. The one or more modules/units may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program 32 in the terminal device 3.

The terminal device 3 may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The terminal device may include, but is not limited to, a processor 30, a memory 31. It will be understood by those skilled in the art that fig. 13 is only an example of the terminal device 3, and does not constitute a limitation to the terminal device 3, and may include more or less components than those shown, or combine some components, or different components, for example, the terminal device 3 may further include an input-output device, a network access device, a bus, etc.

The Processor 30 may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components, etc. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.

The memory 31 may be an internal storage unit of the terminal device 3, such as a hard disk or a memory of the terminal device 3. The memory 31 may also be an external storage device of the terminal device 3, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the terminal device 3. Further, the memory 31 may also include both an internal storage unit and an external storage device of the terminal device 3. The memory 31 is used for storing the computer programs and other programs and data required by the terminal device 3. The memory 31 may also be used to temporarily store data that has been output or is to be output.

It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

In the above embodiments, the descriptions of the respective embodiments have respective emphasis, and reference may be made to the related descriptions of other embodiments for parts that are not described or illustrated in a certain embodiment.

Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

In the embodiments provided in the present invention, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described embodiments of the apparatus/terminal device are merely illustrative, and for example, the division of the modules or units is only one logical division, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signals, telecommunications signals, software distribution medium, and the like. It should be noted that the computer readable medium may contain content that is subject to appropriate increase or decrease as required by legislation and patent practice in jurisdictions, for example, in some jurisdictions, computer readable media does not include electrical carrier signals and telecommunications signals as is required by legislation and patent practice.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.

Claims

1. An energy data warehouse system construction method, comprising:

performing first data processing on energy data of a data source to obtain a detail data table corresponding to the energy data so as to construct an operation data layer;

2. The method according to claim 1, wherein the energy data of the data source comprises at least one of energy device operation data, energy system configuration data, business data during business development, internet data acquired through a network, and third party data provided by a third party system.

3. The method for constructing an energy data warehouse system according to claim 1, wherein the performing a first data process on the energy data table of the data source to obtain a detail data table corresponding to the energy data to construct an operation data layer comprises:

loading energy data in the data source to an original data table;

analyzing the original data table according to the processing time of the energy data, and determining new loading data in the original data table;

judging whether the newly loaded data is abnormal data;

if the new loaded data is not abnormal data, carrying out format conversion on the new loaded data to obtain intermediate data;

partitioning the intermediate data according to the type of the energy equipment and the acquisition time, and writing the intermediate data into a detail data table to construct an operation data layer.

4. The method according to claim 3, wherein the detail data table includes at least one of site information, equipment type, equipment identification, measurement attribute, measurement time, and measurement value.

5. The energy data warehouse system construction method according to claim 1, wherein the second data processing of the detail data table according to the energy device type to obtain a basic data table to construct a basic data layer comprises:

classifying the data in the detail data table according to the type of the energy equipment to obtain the data corresponding to each type of energy equipment;

time alignment is carried out on data corresponding to the energy equipment according to the minimum time granularity so as to obtain first data;

performing data flattening processing on the first data to obtain second data;

and writing the second data into a basic data table corresponding to each energy equipment type to construct a basic data layer.

6. The method according to claim 1, wherein the third data processing is performed on the basic data table according to business analysis requirements to obtain a data warehouse topic to construct a general data layer, and the method includes:

determining a data warehouse theme according to business analysis requirements;

reading data in a basic data table corresponding to each energy equipment type according to the data warehouse theme;

aggregating data corresponding to each energy equipment type according to the dimension data to obtain aggregated data;

and writing the aggregated data into the data warehouse subject to construct a universal data layer.

7. The method according to claim 6, wherein the data warehouse topics include at least one of energy enterprise capacity analysis topics, energy consumption enterprise energy consumption analysis topics, enterprise energy efficiency analysis topics, equipment state operation trend analysis topics, equipment predictive maintenance analysis topics, and enterprise data access quality analysis topics.

8. The method according to claim 1, wherein the fourth data processing is performed on the data warehouse topic according to the business unit to obtain the data mart corresponding to the business unit, so as to construct the application data layer, and the method includes:

determining a data mart according to the service unit;

classifying data in the data warehouse topics according to the data marts;

and writing the classified data in the data warehouse topics into corresponding data marts, and determining access rights to construct an application data layer.

9. The energy data warehouse system building method of claim 8, wherein after the writing the classified data in the data warehouse topics into corresponding data marts and determining access rights to build an application data layer, the method further comprises:

and migrating the data marts to a reporting system to generate reports.

10. The energy data warehouse system construction method of claim 1, wherein the energy data warehouse system construction method further comprises:

and analyzing the dimension according to the energy data, and constructing dimension data, wherein the dimension data at least comprises one of a time dimension, a geographical dimension, a user dimension, a garden dimension, a system dimension and a device dimension.

11. The energy data warehouse system construction method according to any one of claims 1 to 10, wherein the energy data warehouse system construction method further comprises:

and constructing management tools, wherein the management tools at least comprise one of metadata management tools, workflow management tools, data acquisition tools, data processing tools and data migration tools.

12. An energy data warehouse system building apparatus, comprising:

13. The energy data warehouse system building apparatus of claim 12, wherein the energy data warehouse system building apparatus further comprises:

the dimension data construction module is used for analyzing dimensions according to the energy data and constructing dimension data, wherein the dimension data at least comprises one of a time dimension, a geographical dimension, a user dimension, a garden dimension, a system dimension and an equipment dimension;

and/or the management tool building module is used for building a management tool, and the management tool at least comprises one of a metadata management tool, a workflow management tool, a data acquisition tool, a data processing tool and a data migration tool.

14. A terminal device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the energy data warehouse system construction method according to any one of claims 1 to 11 when executing the computer program.

15. A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method for constructing an energy data warehouse system according to any one of claims 1 to 11.