CN115348282A - Energy consumption supervision platform adopting distributed data processing - Google Patents
Energy consumption supervision platform adopting distributed data processing Download PDFInfo
- Publication number
- CN115348282A CN115348282A CN202210913583.1A CN202210913583A CN115348282A CN 115348282 A CN115348282 A CN 115348282A CN 202210913583 A CN202210913583 A CN 202210913583A CN 115348282 A CN115348282 A CN 115348282A
- Authority
- CN
- China
- Prior art keywords
- data
- energy consumption
- layer
- data processing
- management
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/451—Execution arrangements for user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q9/00—Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04Q—SELECTING
- H04Q2209/00—Arrangements in telecontrol or telemetry systems
- H04Q2209/60—Arrangements in telecontrol or telemetry systems for transmitting utility meters data, i.e. transmission of data from the reader of the utility meter
Abstract
The invention provides an energy consumption supervision platform adopting distributed data processing, and relates to the field of energy consumption monitoring. The method comprises a software architecture, a hardware architecture, deployment and data security, and is characterized in that: the software architecture comprises a hardware layer, a data transmission layer, a data processing layer, a system application layer and a system management layer, wherein the hardware architecture and the deployment comprise traditional machine room deployment, cloud deployment, interfaces with other platforms and interfaces between terminal devices. The energy consumption monitoring system is not limited to primary functions of data acquisition, summarization, display and the like, technically, on the basis of large amount of data acquisition, prediction analysis and machine learning are executed by means of artificial intelligence, different variable parameters are input by professionals, and an energy consumption diagnosis report is formed, so that the professionals can quickly know energy consumption conditions of all parts, the application range of the energy consumption supervision platform is further enlarged, and the energy consumption supervision platform can supervise various fields.
Description
Technical Field
The invention relates to the field of energy consumption monitoring, in particular to an energy consumption supervision platform adopting distributed data processing.
Background
The energy consumption supervision platform is a large-scale software and hardware system used for energy management and energy-saving decision making of various industries, the system automatically collects real-time data of various energy forms such as electricity, water, steam and the like consumed by energy consumption equipment scattered in various regions by using intelligent instruments such as various electric quantity transmitters, flow sensors, temperature sensors, pressure sensors and the like, and the real-time data is transmitted to a network platform through a wired or wireless mode.
At present, along with the continuous development of each industry, the electric power that each enterprise used is also constantly rising, the extravagant condition of electric energy often can appear in the in-service use in-process, make the holistic energy consumption of enterprise rise, and then the energy consumption cost of enterprise has been increased, the extravagant condition in a large number of electric energy has also been caused, and every enterprise is different at the equipment and the place of in-service use electric energy, traditional fixed energy consumption supervision platform can't carry out quick diagnosis according to the actual electric energy in service behavior of every enterprise, lead to the very big restriction that traditional energy consumption supervision platform's application range received.
Disclosure of Invention
Technical problem to be solved
Aiming at the defects of the prior art, the invention provides an energy consumption supervision platform adopting distributed data processing, which solves the problem that the application range of the traditional energy consumption supervision platform is greatly limited because the traditional fixed energy consumption supervision platform cannot carry out rapid diagnosis according to the actual electric energy use condition of each enterprise because the actual electric energy use equipment and the actual electric energy use place of each enterprise are different.
(II) technical scheme
In order to achieve the purpose, the invention is realized by the following technical scheme: an energy consumption supervision platform adopting distributed data processing comprises a software architecture, a hardware architecture, a deployment layer and a data safety layer, wherein the software architecture consists of a hardware layer, a data transmission layer, a data processing layer, a system application layer and a system management layer, the hardware architecture and the deployment layer consist of traditional machine room deployment, cloud deployment, interfaces between the traditional machine room deployment and other platforms and interfaces between terminal devices, wherein the hardware architecture and the deployed energy consumption monitoring system bottom data come from a large number of electric, steam and water terminal acquisition devices and mainly comprise electric power terminal devices, the software architecture is provided with a reported data stream type processing, a distributed data processing, a data fast searching and counting, a convenient maintenance, a friendly interface and a safety mechanism program through the hardware layer, the data transmission layer, the data processing layer, the system application layer and the system management layer, and the reported data stream type processing consists of real-time, volatility, burstiness, disorder and infinite property.
Preferably, the hardware layer adopts a multifunctional intelligent instrument, dynamically acquires data in real time and uploads the data to the data layer, and the data layer are connected through acquisition software;
the data transmission layer uploads and gathers bottom data to the energy management system through various protocols and protocols, and the data are processed and analyzed by the system;
the data layer comprises a real-time database, a historical database and an energy management database, and is the core foundation of the whole system;
the data processing layer is used for storing and preprocessing mass data and preparing for analysis and decision making;
the system application layer comprises 3D display, real-time monitoring, centralized control, dynamic analysis and the like, and is the core and key of the whole system;
the system management layer comprises configuration and management of basic information and configuration of the whole software.
Further, hardware deployment mainly related to the traditional machine room deployment comprises a panoramic center energy management system, a regional energy management system and local monitoring and data acquisition equipment;
the cloud deployment is that the system deploys a data center on the cloud by relying on technologies such as virtualization and cloud service, and a database server, a bus server, a real-time processing server, a power prediction server, an equipment management server and an analysis processing server are deployed on the cloud in a virtual machine form;
the interface between the energy consumption monitoring system and other platforms is used for establishing communication between the energy consumption monitoring system and other platforms, and supporting the energy consumption monitoring system by using various data to ensure the reliability of data analysis;
the interface composition between the terminal equipment is that the intelligent energy comprehensive management system is communicated with the terminal equipment under the condition of direct mining.
Furthermore, the distributed data processing is a computer system which connects a plurality of computers in different places, or with different functions, or with different data through a communication network and coordinately completes large-scale information processing tasks under the unified management control of a control system;
the data fast searching and counting is that the power station monitoring system supports the inquiry of historical data of the equipment, the tracing of the equipment data is realized, the running condition of each equipment is known, and meanwhile, indexes of different equipment data are analyzed based on the statistics of the historical data.
Furthermore, in the design process of the system convenient for maintenance, the design ideas of self-definition and configuration are adopted to improve the maintainability of the system, which is mainly shown in the following steps:
s1, designing an independent dictionary management function by using platform related option information, and manually maintaining different types of data dictionaries;
s2, managing parameter configuration in the web service in a text form;
s3, carrying out configuration management on parameters in distributed processing by uniformly adopting zookeeper configuration, and having the advantages of convenience in operation, one-site modification and synchronization of multiple sites;
s4, unified management of various statistical tasks of the platform, a complex scheduling function, friendly interface operation and perfect log recording are achieved through an azkaban scheduling framework, complex data processing deployment and operation and maintenance are completed through graphical interface operation, and overall production efficiency is improved;
s5, providing a storm UI interface by the large data stream type storm processing, monitoring the running state and resource consumption of each processing node in real time, and processing tasks;
s6, historical data storage, namely, the processed data can be visually checked on a plug-in interface through the elastic search plug-in, and the quick analysis and timely response to the abnormal problem are realized;
and S7, by installing a mature cluster service management tool (clouderamanager) of a big data manufacturer, the big data tool can be deployed, the complex background terminal installation is realized, the visual interface is used for completing the installation, and meanwhile, the comprehensive monitoring of the operation and maintenance of the big data service is realized, the early warning is responded in time, and the service problem is positioned quickly.
Further, the friendly interface is a friendly interface which directly brings user-friendly experience effect and can enhance the good image of the company in the user mind, which is mainly shown in the following steps:
s1, a simple and practical operation interface is provided;
s2, designing an atmosphere interface with modern aesthetic property;
and S3, enabling the user to get on the hand quickly.
Further, the security mechanism program is a mechanism of network security, and specifically includes:
s1, closing all ports which are open to the outside (not necessarily open);
s2, isolating all application servers in a network gate mode;
s3, changing the access mode of the server into a safe encrypted connection (ssh);
s4, the web service accesses in a certificate mode (tls/ssl);
s5, transmitting parameters through an interface, and transmitting a ciphertext according to the attribute with higher parameter safety;
s6, the http request pair does not need to use a get mode, and a post is uniformly adopted;
s7, transmitting parameters through a restful interface, and performing parameter verification on the parameters with the format of the sql statement to prevent sql injection;
and S8, adding a verification code in system login to prevent the crawler program from being illegally accessed.
Still further, the data security has a security function of:
s1, adopting a multi-backup mechanism for distributed storage data, and setting a default copy of a Hdfs distributed file block as 3 when an elastic search sets a copy number;
s2, closing a terminal connection port of an external network to the data storage server, and closing 9300, 9200 and 3306 for an elastic search when hive closes a default port of the hiveserver 2;
s3, installing network management software to monitor the running condition of the platform server in real time, grasping the consumption condition of hardware resources and network resources in time, and making data migration preparation;
and S4, carrying out data fault tolerance processing, and ensuring that the data is processed by switching nodes in time after a single point of fault through a distributed technology.
(III) advantageous effects
The invention provides an energy consumption supervision platform adopting distributed data processing. The method has the following beneficial effects:
1. the energy consumption monitoring system is not limited to primary functions of data acquisition, summarization, display and the like, technically executes prediction analysis and machine learning by means of artificial intelligence on the basis of large amount of data acquisition, combines different variable parameters input by professionals to form an energy consumption diagnosis report, enables the professionals to quickly know energy consumption conditions of all parts, further increases the application range of the energy consumption supervision platform, and enables the energy consumption supervision platform to supervise various fields.
2. The energy consumption monitoring system can customize an energy consumption index early warning system according to the customer requirements, and remind operation and maintenance personnel to react in advance by executing predictive analysis, so that the high-efficiency utilization of enterprise energy is ensured.
3. The energy consumption monitoring system of the invention not only plays an important role at the energy input end, but also is additionally provided with a monitoring system at the waste discharge end, and reasonable environmental protection investment and process management measures are made according to the environmental protection requirement and the energy utilization condition.
Drawings
FIG. 1 is a schematic diagram of a topology of the system of the present invention;
FIG. 2 is a schematic diagram of a platform architecture diagram of the energy consumption monitoring system according to the present invention;
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The first embodiment is as follows:
as shown in fig. 1, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, including a software architecture, a hardware architecture, a deployment layer, and data security, where the software architecture is composed of a hardware layer, a data transmission layer, a data processing layer, a system application layer, and a system management layer, and the hardware architecture and the deployment layer are composed of a traditional machine room deployment, a cloud deployment, an interface with another platform, and an interface between terminal devices, where the hardware architecture and the deployed energy consumption monitoring system bottom layer data are from a large number of electricity, steam, and water terminal acquisition devices, and mainly include power terminal devices, such as an intelligent electric meter or a meter reader, a distribution feeder terminal unit, and a natural gas terminal device: mainly have the instrument to include actuating mechanism such as flowmeter (electromagnetism, supersound, turbine etc.), pressure transmitter, temperature transmitter, leakage alarm and electric valve, converter, contactor etc. vapour, water terminal equipment: the intelligent flow meter, the pressure meter, the liquid level meter, the electric valve, the process machine pump remote monitoring terminal and other equipment are mainly arranged, the software architecture is provided with a reported data flow type processing, a distributed data processing, a data fast searching and counting, a convenient maintenance, a friendly interface and a safety mechanism program through a hardware layer, a data transmission layer, a data processing layer, a system application layer and a system management layer, and the reported data flow type processing is jointly formed by instantaneity, volatility, burst, disorder and infinite;
the energy consumption monitoring system is not limited to primary functions of data acquisition, summarization, display and the like, technically, on the basis of large data acquisition, the energy consumption monitoring system executes predictive analysis and machine learning by means of artificial intelligence, different variable parameters are input by professionals, an energy consumption diagnosis report is formed, and then the energy consumption monitoring system is increased.
Example two:
as shown in fig. 1, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, which is further extended according to the contents of the first embodiment:
the hardware layer adopts a multifunctional intelligent instrument, dynamically acquires data in real time and uploads the data to the data layer, the data and the data layer are connected through acquisition software, and the data transmission layer uploads and summarizes bottom data to the energy management system through various protocols and sends the data to the system for processing and analysis;
the system application layer comprises 3D display, real-time monitoring, centralized control, dynamic analysis and the like, is the core and the key of the whole system, further enables the energy consumption monitoring system to be a whole set of solution for energy management under the action of a software framework, provides equipment and technical measures from hardware to software, supports most communication collection instruments (supporting protocols such as OPC, modbus and TCP/IP) at home and abroad in the aspect of hardware, and supports a plurality of layer functions such as data collection, real-time data, historical data, energy management analysis data, system management, data display, analysis and control in the aspect of software.
Example three:
as shown in fig. 1 and 2, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, which is further extended according to the contents of the first embodiment:
the system provides a series of safety guarantees on the design of a supporting platform, performs primary guard on important server processes, provides different levels of authority management on the access of a database, a service and a Web system, and ensures the safe, reliable and efficient operation of the system;
the cloud deployment is that the system deploys a data center on the cloud by relying on virtualization, cloud service and other technologies, a database server, a bus server, a real-time processing server, a power prediction server, an equipment management server and an analysis processing server are deployed on the cloud, a virtual machine mode is adopted, the virtual servers are communicated through a virtual network, and the channel bandwidth of the virtual network needs to be determined when the servers are selected;
the intelligent energy comprehensive management system is characterized in that an interface between the intelligent energy comprehensive management system and other platforms is an energy consumption monitoring system which needs to be communicated with the other platforms, and supports various data to ensure the reliability of data analysis, wherein the monitoring platform comprises monitoring platforms of enterprises for producing, transporting and selling various energy sources such as electricity, steam and water, the intelligent energy comprehensive management system is communicated with terminal equipment under the condition of direct acquisition, and the intelligent energy comprehensive management system comprises equipment such as terminal metering instruments of electricity, steam and water, pipe network protection equipment, automatic industrial control equipment and measurement and control units, and supports the forwarding access of other system data.
Example four:
as shown in fig. 1 and 2, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, which is further extended according to the contents of the first embodiment:
the reported data stream processing is directed acyclic graph (DAG for short) describing a calculation process of a large data stream, and is different from large data batch calculation, wherein the real-time property is that stream-type large data is generated in real time and calculated in real time, and the timeliness of result feedback is also required to be ensured;
the data is often disposable and volatile, and even if the data is replayed, the obtained data stream is often different from the previous data stream;
the system has the advantages that the system has good scalability, can dynamically adapt to uncertain incoming data streams, and has strong system computing capability and large data flow dynamic matching capability;
disorder is that in a big data stream type computing environment, data streams are disordered and data elements in the same data stream are disordered: on one hand, because the data sources are independent from each other and the spatio-temporal environments are different, the relative sequence of the data elements in the data streams cannot be ensured, and on the other hand, even if the same data stream is used, the consistency of the sequence of the data elements in the replay data stream and the previous data stream cannot be ensured due to the dynamic change of time and environment;
the data is generated in real time and dynamically increased in a large data flow type calculation, and the data is generated and continuously increased as long as a data source is in an active state. On the one hand, there is not enough space in the hardware to store these infinitely growing data, and on the other hand, there is no suitable software to manage this data effectively, and it is necessary that the system has good stability to ensure long-term and stable operation of the system. In the operation process of photovoltaic power station equipment, a large amount of index data are generated, a communication manager serves as an acquisition module, real-time data of each piece of equipment are collected according to specified frequency (configurable), and are reported to a data processing platform, the continuous reporting data mode forms a batch of data streams, and storm is a typical streaming data processing solving party in the current field of big data real-time processing. The storm can customize different types of data sources and independently process data accessed in different modes. The storm is used as a distributed processing platform and is processed through a data random distribution mechanism, and single-node processing pressure is solved. The storm supports data fault-tolerant processing, and when certain data processing fails, the storm can automatically retransmit the data, so that the data are not lost. By combining the kafka message queue, the processing pressure brought by exponential growth of data is dealt with, and the computing capacity can be increased by transversely expanding nodes when the computing resources reach the bottleneck.
Example five:
as shown in fig. 1 and 2, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, which is further extended according to the contents of the first embodiment:
the distributed data processing is a computer system which connects a plurality of computers in different places, or with different functions, or with different data through a communication network and coordinately completes large-scale information processing tasks under the unified management control of a control system, in brief, the distributed processing is a computer system which respectively bears different parts of the same work task by a plurality of connected computers, simultaneously runs under the control of people to jointly complete the same work task, and in the field of large data processing, the mainstream three-large distributed computing system comprises: hadoop, spark and Storm, because Google has no technology implementation of open source Google distributed computing model, other Internet companies can only build their distributed computing systems according to the relevant principles in three Google technical papers, hadoop adopts MapReduce distributed computing framework, HDFS distributed file system is developed according to GFS, HBase data storage system is developed according to BigTable, hadoop still does not reach the standard in Google paper in terms of operation speed although the principle is the same as that of distributed computing system used inside Google, hadoop has the open source characteristic that Hadoop becomes the de international standard in fact of distributed computing system, and numerous Internet companies such as Yahoo, facebook, amazon and domestic hundredth, and Alaba all build their distributed computing systems on the basis of Hadoop, spark is also an open source project of Apache Foundation, which is developed by laboratories university of Berry of Calif., the distributed computing system is another important distributed computing system, which is improved on the basis of Hadoop in architecture, the greatest difference between Spark and Hadoop is that Hadoop uses a hard disk to store data, spark uses a memory to store data, so Spark can provide an operation speed 100 times faster than Hadoop, but Spark cannot be used for processing data needing long-term storage due to the fact that data is lost after the power of the memory is cut off, storm is a distributed computing system mainly pushed by Twitter, is developed by a BackType team, is an incubation project of an Apache foundation, provides a real-time operation characteristic on the basis of Hadoop, can process a large data stream in real time, is different from Hadoop and Spark, storm does not perform data collection and storage work, receives data directly through a network in real time and processes data in real time, and then returns results directly through the network in real time, spark and Storm are the most important three distributed computing systems at present, hadoop is usually used for offline complex big data processing, spark is usually used for offline rapid big data processing, storm is usually used for online real-time big data processing, spark is designed for real-time big data processing, storm and Spark are both open source frameworks of distributed stream processing which is popular in the big data processing field, photovoltaic power station equipment data processing is a typical stream processing application scene, real-time reporting of large-scale equipment is realized, a traditional single machine mode cannot meet real-time processing of data, so that reference of the distributed stream processing technology is necessary, and due to the consideration of several aspects, storm is selected as the distributed processing technology in the platform, storm is better than Spark in real-time delay degree, the former is pure real-time and the latter is quasi real-time, and the transaction mechanism, robustness, fault tolerance, dynamic adjustment deployment parallelism degree and other characteristics of Storm of the Storm are all better than Spark, if the Storm is relatively low in terms, and the computing of Storm is required by utilizing the Storm resource in a relatively high-time environment, and the computing environment of the Storm is relatively low in the limit of the Storm;
the data fast search and statistics are that a power station monitoring system supports inquiry of historical data of equipment, the equipment data tracing is realized, the operation condition of each equipment is known, indexes of different equipment data are analyzed based on the historical data statistics, such as statistics of power generation increase values of each inverter in each dimension of hour, day, month and year, solar irradiance (environmental monitor data) of a photovoltaic power station in each hour, an Elasticsearch is used as a popular data storage and search engine, complex inquiry in different dimensions and multiple conditions is supported, a storage index is created based on the year and month, a date is used as an index type, time and a service association tag are added to each piece of data, real-time inquiry of different services and different types of equipment data can be responded, a second-level response can be achieved, the Elasticsearch provides comprehensive aggregation functions, such as summation, average value and maximum and minimum value, statistical analysis of various granularities can be realized according to different services, in the aspect of data storage, the Elasticsearch is also an excellent storage tool, the functions of multiple copies, fragmentation and automatic balancing are supported, data safety is ensured, fast search, single node pressure dispersion is achieved, meanwhile, the Elasticsearch search can be expanded through new storage resources are provided, and a new storage node is added;
in the design process of the convenient maintenance system, the self-defined and configurable design concept is adopted to improve the maintainability of the system, which is mainly characterized in that the platform relates to the information of options, an independent dictionary management function is designed, different types of data dictionaries can be manually maintained, the parameter configuration in the web service is managed in a text form, the parameter configuration in distributed processing is uniformly managed by adopting zookeeper configuration, the operation is convenient, one part is modified, the unified management of various statistical tasks of the platform, the complex scheduling function, the friendly interface operation and the perfect log record are realized by synchronously adopting an azkaban scheduling framework at a plurality of positions, the complex data processing deployment and the operation and maintenance are completed by the graphical interface operation, the method has the advantages that the overall production efficiency is improved, the large data stream type processing storm provides a storm UI interface, each processing node is monitored in real time, the running state and resource consumption of processing tasks are processed, then historical data are stored, the processed data can be visually checked on the plug-in interface through an elastic search plug-in, the abnormal problems are quickly analyzed and responded in time, finally, the large data tool can be deployed through installing a mature cluster service management tool (clouderamaranager) of a large data manufacturer, the complex background terminal installation is realized, the visual interface is used for completing, the comprehensive monitoring on the running and maintenance of the large data service, the early warning is timely responded, and the service problems are quickly positioned;
the friendly interface is a friendly interface, which directly brings a user-friendly experience effect, can enhance the good image of the company in the user mind, and is mainly reflected in a simple and practical operation interface and atmosphere, modern aesthetic interface design and user's convenience;
the security mechanism program is a network security mechanism, and specifically comprises the steps of closing all externally opened ports (unnecessary opening), isolating all application servers in a gateway mode, changing a server access mode into a secure encrypted connection (ssh), enabling web services to access by adopting a certificate mode (tls/ssl) and interface parameter transmission, aiming at the attribute with higher parameter security, adopting ciphertext transmission and http request to transmit parameters without using a get mode, uniformly adopting post and restful interface parameter transmission, and carrying out parameter verification on parameters with an sql statement format, so that sql is prevented from being injected, verification codes are increased during system login, and a crawler program is prevented from being illegally accessed.
Example six:
as shown in fig. 1 and 2, an embodiment of the present invention provides an energy consumption monitoring platform using distributed data processing, which is further extended according to the contents of the first embodiment:
the data security system is characterized in that a multi-backup mechanism (at least two copies) is adopted for distributed storage data, if an elastic search is set to have a copy number Hdfs distributed file block default copy of 3, then a terminal connection port of an external network to a data storage server is closed, if a hive is closed to a hive server2 default port, the elastic search is closed to 9300, 9200 and mysql is closed to 3306, then network management software is installed to monitor the running condition of a platform server in real time, consumption conditions of hardware resources and network resources are mastered in time, data migration preparation is made, finally data fault-tolerant processing is carried out, and the data are guaranteed to be switched to node processing in time after single-point fault through a distributed technology.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (8)
1. An energy consumption supervision platform adopting distributed data processing comprises a software architecture, a hardware architecture, deployment and data security, and is characterized in that: the software architecture comprises a hardware layer, a data transmission layer, a data processing layer, a system application layer and a system management layer, wherein the hardware architecture and the deployment comprise traditional machine room deployment, cloud deployment, interfaces with other platforms and interfaces between terminal devices, the hardware architecture and the deployed energy consumption monitoring system bottom data are from a large number of electric, steam and water terminal acquisition devices and mainly comprise electric power terminal devices, the software architecture is provided with a reported data stream type processing, a distributed data processing, a data processing layer, a system application layer and a system management layer through the hardware layer, the data transmission layer, the data processing layer, the system application layer and the system management layer, the reported data stream type processing, the distributed data processing, the data fast searching and counting, the maintenance is convenient, the interface is friendly, and the safety mechanism program is arranged, and the reported data stream type processing is jointly formed by real-time, volatility, burstiness, disorder and infinite property.
2. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: the hardware layer adopts a multifunctional intelligent instrument, dynamically acquires data in real time and uploads the data to the data layer, and the data layer are connected through acquisition software;
the data transmission layer uploads and gathers bottom data to the energy management system through various protocols and protocols, and the data are processed and analyzed by the system;
the data layer comprises a real-time database, a historical database and an energy management database;
the data processing layer is used for storing and preprocessing mass data;
the system application layer comprises 3D display, real-time monitoring, centralized control and dynamic analysis;
the system management layer comprises configuration and management of basic information and configuration of the whole software.
3. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: hardware deployment mainly related to the traditional machine room deployment comprises a panoramic center energy management system, a regional energy management system and local monitoring and data acquisition equipment;
the cloud deployment is that the system deploys a data center on the cloud by relying on virtualization and cloud service technologies, and a database server, a bus server, a real-time processing server, a power prediction server, an equipment management server and an analysis processing server are deployed on the cloud in a virtual machine form;
the interface between the energy consumption monitoring system and other platforms is that the energy consumption monitoring system needs to establish communication with other platforms;
the interface composition between the terminal equipment is that the intelligent energy comprehensive management system is communicated with the terminal equipment under the condition of direct mining.
4. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: the distributed data processing is realized by connecting a plurality of computers with different functions and different data at different places through a communication network and controlling the computers by the unified management of a control system;
the data fast search and statistics are that the power station monitoring system supports the inquiry of historical data of the equipment.
5. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: in the design process of the system convenient to maintain, the design ideas of self-definition and configuration are adopted, and the main expression is as follows:
s1, designing an independent dictionary management function according to information of options related to a platform;
s2, managing parameter configuration in the web service in a text form;
s3, configuring parameters in distributed processing, and uniformly managing zookeeper configuration;
s4, unified management of various statistical tasks of the platform, a complex scheduling function, friendly interface operation and perfect log recording are realized by an azkaban scheduling framework, and complicated data processing deployment and operation and maintenance are completed by graphical interface operation;
s5, providing a storm UI interface by the large data stream type storm processing, monitoring the running state and resource consumption of each processing node in real time, and processing tasks;
s6, storing historical data through an elastic search plug-in;
and S7, installing a mature cluster service management tool of a big data manufacturer.
6. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: the friendly interface is a friendly interface and directly brings user-friendly experience effects, and is mainly represented by:
s1, a simple and practical operation interface is provided;
s2, designing an atmosphere interface with modern aesthetic property;
and S3, enabling the user to get on the hand quickly.
7. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: the security mechanism program is a network security mechanism, and specifically includes:
s1, closing all ports which are open to the outside;
s2, isolating all application servers in a network gate mode;
s3, changing the access mode of the server into safe encrypted connection;
s4, the web service is accessed in a certificate mode;
s5, transmitting parameters through an interface, and transmitting a ciphertext according to the attribute with higher parameter safety;
s6, the http request pair does not need to use a get mode, and a post is uniformly adopted;
s7, transmitting parameters through a restful interface, and performing parameter verification on the parameters with the format of the sql statement to prevent sql injection;
and S8, adding a verification code in system login to prevent the crawler program from being illegally accessed.
8. The energy consumption supervision platform adopting distributed data processing according to claim 1, characterized in that: the data security has a security value of:
s1, adopting a multi-backup mechanism for distributed storage data, and setting a default copy of a Hdfs distributed file block as 3 when an elastic search sets a copy number;
s2, closing a terminal connection port of the external network to the data storage server, and when the hive closes the default port of the hiveserver2, closing the elastic search 9300 and 9200, and closing the mysql 3306;
s3, installing network management software to monitor the running condition of the platform server in real time, grasping the consumption condition of hardware resources and network resources in time, and making data migration preparation;
and S4, carrying out data fault tolerance processing, and ensuring that the data is processed by switching nodes in time after a single point of fault through a distributed technology.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210913583.1A CN115348282A (en) | 2022-07-30 | 2022-07-30 | Energy consumption supervision platform adopting distributed data processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210913583.1A CN115348282A (en) | 2022-07-30 | 2022-07-30 | Energy consumption supervision platform adopting distributed data processing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115348282A true CN115348282A (en) | 2022-11-15 |
Family
ID=83949205
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210913583.1A Pending CN115348282A (en) | 2022-07-30 | 2022-07-30 | Energy consumption supervision platform adopting distributed data processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115348282A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116498908A (en) * | 2023-06-26 | 2023-07-28 | 成都秦川物联网科技股份有限公司 | Intelligent gas pipe network monitoring method based on ultrasonic flowmeter and Internet of things system |
-
2022
- 2022-07-30 CN CN202210913583.1A patent/CN115348282A/en active Pending
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116498908A (en) * | 2023-06-26 | 2023-07-28 | 成都秦川物联网科技股份有限公司 | Intelligent gas pipe network monitoring method based on ultrasonic flowmeter and Internet of things system |
| CN116498908B (en) * | 2023-06-26 | 2023-08-25 | 成都秦川物联网科技股份有限公司 | Intelligent gas pipe network monitoring method based on ultrasonic flowmeter and Internet of things system |
| US11953356B2 (en) | 2023-06-26 | 2024-04-09 | Chengdu Qinchuan Iot Technology Co., Ltd. | Methods and internet of things (IoT) systems for monitoring smart gas pipeline networks based on ultrasonic flowmeters |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Daki et al. | Big Data management in smart grid: concepts, requirements and implementation | |
| CN109613329B (en) | Fine line loss analysis system | |
| WO2017036244A1 (en) | Digital simulation system of power distribution network | |
| CN109151072A (en) | A kind of edge calculations system based on mist node | |
| RU2583703C2 (en) | Malicious attack detection and analysis | |
| CN104751280A (en) | Petrochemical enterprise electric energy consumption monitoring system and data management method thereof | |
| CN112383141A (en) | Intelligent fusion terminal based on block chain | |
| CN108268569A (en) | The acquisition of water resource monitoring data and analysis system and method based on big data technology | |
| CN110838936A (en) | Power distribution communication network management device and method | |
| CN115348282A (en) | Energy consumption supervision platform adopting distributed data processing | |
| CN114281790B (en) | Multi-type load resource aggregator access system and method | |
| CN208890843U (en) | A kind of edge calculations system based on mist node | |
| CN116485136A (en) | Cloud edge cooperation-based lithium battery production line data platform construction method and system | |
| CN116366692A (en) | High-performance intelligent edge terminal system | |
| CN210670150U (en) | Energy data management system | |
| Yang et al. | Technology research on panoramic situation awareness of operation state of smart distribution network | |
| CN114567568A (en) | Electric power Internet of things data processing method and device based on edge calculation | |
| Guo et al. | Key Technologies and Applications for Intelligent Control of Power 5G Virtual Private Network | |
| Zhang et al. | Open architecture design of universal terminal for multiple measurement and control scenarios of distribution network | |
| CN205038504U (en) | Wind stores up monitored control system | |
| Wang et al. | Research on Distributed PV Monitoring System Based on Ubiquitous Power IOT Architecture | |
| Sahoo et al. | Artificial intelligence based electric grid operation enabled with data encryption | |
| CN218512824U (en) | Intelligent heating management and control system based on BIM and edge computing unit | |
| KR20180137622A (en) | Apparatus for remote managing new renewable energy plant | |
| Wang et al. | Research on the SCADA system constructing methodology based on SOA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |