CN112214541A - Deep decoupling and data cooperation method for power monitoring data - Google Patents
Deep decoupling and data cooperation method for power monitoring data Download PDFInfo
- Publication number
- CN112214541A CN112214541A CN202010922700.1A CN202010922700A CN112214541A CN 112214541 A CN112214541 A CN 112214541A CN 202010922700 A CN202010922700 A CN 202010922700A CN 112214541 A CN112214541 A CN 112214541A
- Authority
- CN
- China
- Prior art keywords
- data
- module
- processing
- processing module
- power monitoring
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/214—Database migration support
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/26—Visual data mining; Browsing structured data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9537—Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2216/00—Indexing scheme relating to additional aspects of information retrieval not explicitly covered by G06F16/00 and subgroups
- G06F2216/03—Data mining
Abstract
The invention discloses a deep decoupling and data cooperation method for power monitoring data, and belongs to the technical field of power monitoring. According to the method, a storm architecture and a Python language are used as an electric power monitoring data mining tool, various technical modules with different functions are designed, all module systems are integrated, a set of detailed data monitoring flow system is constructed, so that the data are deeply mined through a series of modules, and finally, deep decoupling and data cooperation of the electric power monitoring data are realized. The invention improves the data quality and ensures the reliability, accuracy, timeliness and effectiveness of the data, thereby realizing the omnibearing quality management of the data of the power supply enterprise and ensuring the integration and mining application of the data.
Description
Technical Field
The invention relates to a deep decoupling and data cooperation method for power monitoring data, and belongs to the technical field of power monitoring.
Background
With the deep development of the smart grid technology and the continuous progress of the sensor technology, the internet of things technology and the communication technology, the grid data grows in a geometric exponential manner, and meanwhile, the data is diverse, real-time, volatile and infinite, and many data are streaming data which need to be monitored continuously. The big data of the power system implies huge commercial value and social value, and the mining of the massive data can obtain larger value.
The electric power big data comprises the following characteristics:
(1) the quantity is large, in the construction of a smart power grid, various communication and sensing devices generate massive data, and the data increase speed is faster and faster.
(2) The data types are multiple, the power grid data are in a diversified development trend, and unstructured data such as texts, videos and audios are rapidly increased.
(3) The response speed is high, and the data generated in the process from monitoring, diagnosis to overhaul of the power equipment needs to be processed in real time.
(4) The data accuracy is high, and the accuracy of the power data relates to all aspects of the power system, so the accuracy is high.
The key of big data processing is solving the data quality problem, avoiding data errors and guaranteeing data quality to really enable enterprises to obtain benefits from big data application, and guaranteeing data quality is a prerequisite condition for bringing value to the enterprises by big data. Big data analysis can not be separated from data quality and data management, and high-quality data and effective data management can guarantee the reality and the value of an analysis result in both academic research and commercial application fields. In recent years, informatization of the power industry has been greatly developed, from initial power production automation to management informatization construction represented by financial computing in the 80 s, to large-scale enterprise informatization construction in recent years, and particularly along with comprehensive construction of a next-generation intelligent power grid, wide application of a new-generation IT technology represented by internet of things and cloud computing in the power industry is started, and power data resources are rapidly increased and form a certain scale.
With the continuous construction and the deepened application of the informatization of the power supply enterprises, various services of the power supply enterprises are preliminarily fused with the informatization, the quantity and the types of service data in an information system are gradually increased, and the data sharing requirement is urgent. The data quality and the data sharing utilization level are not high, firstly, the data pair analysis decision support degree is low, multiple sources and inconsistent statistical calibers exist in the same data; secondly, the support degree of the data on operation management needs to be improved, the data quality is uneven, part of the data has no service system support, and unified specification, standard and definite data accountability are lacked; and thirdly, the data quality control is lagged, the control work is one-sidedness, an integral data quality control system and a comprehensive and effective data quality guarantee mechanism are not formed, and the deep mining of the data value is restricted.
At present, when a conventional infrastructure and a conventional technical scheme are used for processing massive and heterogeneous power grid data, the limitation is large, and the reliability and the real-time performance of processing the multivariate data in the smart power grid environment are poor. When the power equipment is in an extremely severe environment, such as fog, ice rain, storms, thunderstorms and the like, the power equipment frequently sends alarm data to a monitoring center due to the fact that the monitoring value is out of limit, so that the well blowout phenomenon of the monitoring data occurs in the monitoring center, the receiving and processing of the data by the conventional platform cannot meet the actual requirements, the real-time performance cannot be met, and the data is lost and covered.
In summary, the existing power data processing generally analyzes and processes data in a remote central control management platform, and has low processing efficiency and poor real-time performance, and the processing requirement of power data intellectualization is far from being met.
Disclosure of Invention
In view of the defects in the prior art, the invention provides a deep decoupling and data cooperation method for power monitoring data, so as to improve data quality, ensure data reliability, accuracy, timeliness and effectiveness, realize comprehensive quality management on power supply enterprise data, and guarantee data integration and mining application.
The invention adopts the following technical scheme for solving the technical problems:
a depth decoupling and data cooperation method for power monitoring data comprises the following steps:
(1) after the data is accessed to the platform or the edge device, the data is transmitted and stored to a database;
(2) adopting a Storm architecture to transmit data in real time and process a normalized panoramic data set, and then switching to a data processing model;
(3) data are sorted according to a unified data model to obtain ordered data;
(4) migrating a data source;
(5) carrying out format classification on the data packet to generate a CSV format file;
(6) processing a tool set by a Pandas structured data set and creating a merged data table;
(7) performing intersection processing on the merged data table by using the Merge function to realize matching and merging;
(8) remodeling the matched merged data set by utilizing an appendix function, and establishing a physical mapping between a time sequence and a monitored object;
(9) establishing a useful information deduction key mechanism through a Levels process to realize grouping marking based on data characteristics;
(10) performing correlation combination and redundancy check on the grouped marked data set by utilizing a Group _ by function and a Pivote _ table function;
(11) and giving a panoramic data reproduction set fused with the time sequence and the spatial sequence, and realizing data visualization under an edge calculation model through a Pyecharts class library.
And (3) combing the data comprises performing real-time decoding work on the data packets of each region according to a multidimensional state data region, a check data region, a fixed ending byte region data region and a time sequence of the power equipment materials.
The specific process of the step (4) is as follows: after the data are combed, the data are sent to different areas for processing according to the distribution of the data processing modules: if the data is data processed by local edge calculation, an edge calculation model is merged; if the platform is processing, the data is transmitted to the platform.
A deep decoupling and data cooperation system for power monitoring data comprises an integrated data transmission and storage module, a Storm framework processing module, a data carding module, a data source migration module, an edge processing module, a platform processing module, a structured data processing module, an intersection processing and matching merging module, a data fusion module, a feature fusion module and a fused data visualization module, wherein the integrated data transmission and storage module, the Storm framework processing module, the data carding module and the data source migration module are sequentially connected, the data source migration module is respectively connected with the edge processing module and the platform processing module, the edge processing module and the platform processing module are respectively connected with the structured data processing module, the structured data processing module, the intersection processing and matching combination module, the data fusion module, the feature fusion module and the fusion data visualization module are sequentially connected.
The invention has the following beneficial effects:
1. the software and hardware deep decoupling and data cooperation sub-framework based on the edge calculation has better data fusion and edge data migration efficiency, realizes the functions of data positioning based on functional area division, data merging based on redundancy analysis, data set matching merging, remodeling processing and the like, and improves the data processing efficiency and the real-time property.
2. According to the invention, by constructing the edge-calculation-based software and hardware deep decoupling and data cooperation sub-architecture such as the panoramic data migration set fusing time and space sequences under the grouping mark based on the data characteristics, the complete life cycle realization efficiency is improved, and powerful support is provided for realizing the software and hardware deep decoupling and data cooperation under the power distribution Internet of things architecture.
Drawings
FIG. 1 is a functional block diagram of deep decoupling and data synergy.
FIG. 2 is a flow chart of a deep decoupling and data collaboration method.
Detailed Description
In order to make the technical problems, technical solutions and advantageous effects to be solved by the present invention more clearly apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular internal procedures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present invention with unnecessary detail.
The core technology of the invention is that a Storm architecture and Python language are used as an electric power monitoring data mining tool, and the deep decoupling data cooperation of the electric power monitoring data is finally realized by integrating modules such as a data transmission and storage module, a Storm architecture processing module, a data carding module, a data source migration module, an edge/platform processing module, structured data processing, intersection processing and matching combination, data fusion, feature fusion, fusion data visualization and the like. The specific embodiment is as a functional block diagram of deep decoupling and data cooperation, see fig. 1.
And (3) architecture: the most basic data processing application unit is a packed topological operation, and data processed by Storm has real-time and continuous characteristics. The biggest advantage of Storm framework over other real-time computing frameworks is the reliable processing of data, which can be tracked. The method is very suitable for the field that the data needs to be accurately processed. The Storm framework is strong in the transverse expansion capability of cluster computing, and the overall computing is divided into a plurality of subtasks to realize parallel computing in a cluster.
The Storm architecture has low requirement on the accuracy of data processing, the Storm architecture focuses on carrying out real-time settlement on data of a data source and feeding back a result in time, the Storm adopts a distributed architecture and has scalability, and the processing capacity of a platform can be changed by the change of the number of cluster nodes. The platforms are responsible for data transmission and task allocation, and users only need to design data processing algorithms according to requirements without considering interaction between nodes.
Language: a cross-platform computer programming language. Is a high-level scripting language that combines interpretive, compiled, interactive, and object-oriented capabilities. Originally designed for writing automated scripts (shells), the more they are used for the development of independent, large projects with the continual updating of versions and the addition of new functionality in language.
JavaScript is a lightweight, interpreted or just-in-time high-level programming language with function precedence.
NumPy is the abbreviation of Numerical Python, and is a basic package for Python scientific computation. The method not only can complete scientific calculation tasks, but also can be used as an efficient multidimensional data container and can be used for storing and processing large matrixes.
CSV is a versatile, relatively simple file format that is widely used by users, businesses, and science. The most widespread application is the transfer of tabular data between programs that themselves operate on incompatible formats (often proprietary and/or non-canonical formats).
The Python plug-in Pandas is very functional. The Pandas has a very convenient and fast data processing function, and mainly comprises 3 aspects of data input and output, data cleaning processing and data mining. The Pandas has a powerful data cleaning function, and can efficiently fill and replace imported missing data, inquire and delete abnormal data and the like.
The Merge function realizes the compatibility and combination of data according to the data processing rule;
group _ by and Pivot _ table implement data grouping and marking lists;
the apend () function enables the addition of a new object at the end of the data list, and the isin () function leaves only the data containing the particular object in the original data.
PyEcharts is a class library used to generate Echarts charts. Is an interface further developed using Python language, which calls must be in Python environment. Echarts is an open-source business-level data graph technology, is a graph library written by pure JavaScript, and supports numerous mainstream browsers. Is widely applied to application systems in various fields.
If the deep decoupling and data cooperation method is adopted, the data correspondingly changes along with different processing modes of the module, the processing flow is shown in fig. 2, and the implementation effect of the module on the data is specifically described in combination with the flow chart:
1. after the data is accessed to the platform or the edge device, the data is transmitted and stored to a database;
2. adopting a Storm architecture to transmit data in real time and process a normalized panoramic data set, and then switching to a data processing model;
3. data carding is carried out on the data according to a unified data model, including data partitioning such as a power equipment material multidimensional state data area, a check data area and a fixed ending byte area, and data packet real-time decoding is carried out on each area according to a time sequence, so that ordered data are obtained;
4. data source migration: after the data are combed, the data are sent to different areas for processing according to the distribution of the data processing modules: if the data is data processed by local edge calculation, an edge calculation model is merged; if the platform processing is carried out, the data are transmitted to the platform;
5. carrying out format classification on the data packet to generate a CSV format file;
6. creating a merged data table by the Pandas structured data set processing tool set;
7. performing intersection processing on the merged data table by using the Merge function to realize matching and merging;
8. remodeling the matched merged data set by utilizing an appendix function, and establishing a physical mapping between a time sequence and a monitored object;
9. establishing a useful information deduction key mechanism through a Levels process to realize grouping marking based on data characteristics;
10. performing correlation combination and redundancy check on the grouped marked data set by utilizing a Group _ by function and a Pivote _ table function;
11. and giving a panoramic data reproduction set fused with the time sequence and the spatial sequence, and realizing data visualization under an edge calculation model through a Pyecharts class library.
Specific embodiments of the present invention have been described above in detail. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.
Claims (4)
1. A depth decoupling and data cooperation method for power monitoring data is characterized by comprising the following steps:
(1) after the data is accessed to the platform or the edge device, the data is transmitted and stored to a database;
(2) adopting a Storm architecture to transmit data in real time and process a normalized panoramic data set, and then switching to a data processing model;
(3) data are sorted according to a unified data model to obtain ordered data;
(4) migrating a data source;
(5) carrying out format classification on the data packet to generate a CSV format file;
(6) processing a tool set by a Pandas structured data set and creating a merged data table;
(7) performing intersection processing on the merged data table by using the Merge function to realize matching and merging;
(8) remodeling the matched merged data set by utilizing an appendix function, and establishing a physical mapping between a time sequence and a monitored object;
(9) establishing a useful information deduction key mechanism through a Levels process to realize grouping marking based on data characteristics;
(10) performing correlation combination and redundancy check on the grouped marked data set by utilizing a Group _ by function and a Pivote _ table function;
(11) and giving a panoramic data reproduction set fused with the time sequence and the spatial sequence, and realizing data visualization under an edge calculation model through a Pyecharts class library.
2. The method for deep decoupling and data cooperation of power monitoring data according to claim 1, wherein the data combing in the step (3) comprises performing real-time data packet decoding work on each zone according to a multidimensional state data zone, a check data zone, a fixed ending byte zone data zone and a time sequence of power equipment materials.
3. The method for deep decoupling and data collaboration of power monitoring data as claimed in claim 2, wherein the specific process of the step (4) is as follows: after the data are combed, the data are sent to different areas for processing according to the distribution of the data processing modules: if the data is data processed by local edge calculation, an edge calculation model is merged; if the platform is processing, the data is transmitted to the platform.
4. The method for deep decoupling and data cooperation of power monitoring data as claimed in claim 1, wherein the system adopted by the method comprises an integrated data transmission and storage module, a Storm architecture processing module, a data combing module, a data source migration module, an edge processing module, a platform processing module, a structured data processing module, an intersection processing and matching merging module, a data fusion module, a feature fusion module and a fused data visualization module, wherein the integrated data transmission and storage module, the Storm architecture processing module, the data combing module and the data source migration module are sequentially connected, the data source migration module is respectively connected with the edge processing module and the platform processing module, the edge processing module and the platform processing module are respectively connected with the structured data processing module, the intersection processing and matching merging module, the intersection processing module, the matching merging module, the data processing module and the data, The data fusion module, the characteristic fusion module and the fusion data visualization module are sequentially connected.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010922700.1A CN112214541A (en) | 2020-09-04 | 2020-09-04 | Deep decoupling and data cooperation method for power monitoring data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010922700.1A CN112214541A (en) | 2020-09-04 | 2020-09-04 | Deep decoupling and data cooperation method for power monitoring data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112214541A true CN112214541A (en) | 2021-01-12 |
Family
ID=74049384
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010922700.1A Withdrawn CN112214541A (en) | 2020-09-04 | 2020-09-04 | Deep decoupling and data cooperation method for power monitoring data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112214541A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114356502A (en) * | 2021-12-31 | 2022-04-15 | 国家电网有限公司 | Unstructured data marking, training and publishing system and method based on edge computing technology |
| CN114490618A (en) * | 2022-02-15 | 2022-05-13 | 北京大数据先进技术研究院 | Ant-lion algorithm-based data filling method, device, equipment and storage medium |
| CN115599367A (en) * | 2022-10-16 | 2023-01-13 | 国网吉林省电力有限公司经济技术研究院（Cn） | Method for collecting and sorting energy big data and establishing visual platform |
| CN116415206A (en) * | 2023-06-06 | 2023-07-11 | 中国移动紫金(江苏)创新研究院有限公司 | Operator multiple data fusion method, system, electronic equipment and computer storage medium |
-
2020
- 2020-09-04 CN CN202010922700.1A patent/CN112214541A/en not_active Withdrawn
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114356502A (en) * | 2021-12-31 | 2022-04-15 | 国家电网有限公司 | Unstructured data marking, training and publishing system and method based on edge computing technology |
| CN114356502B (en) * | 2021-12-31 | 2024-02-13 | 国家电网有限公司 | Unstructured data marking, training and publishing system and method based on edge computing technology |
| CN114490618A (en) * | 2022-02-15 | 2022-05-13 | 北京大数据先进技术研究院 | Ant-lion algorithm-based data filling method, device, equipment and storage medium |
| CN114490618B (en) * | 2022-02-15 | 2022-11-11 | 北京大数据先进技术研究院 | Ant-lion algorithm-based data filling method, device, equipment and storage medium |
| CN115599367A (en) * | 2022-10-16 | 2023-01-13 | 国网吉林省电力有限公司经济技术研究院（Cn） | Method for collecting and sorting energy big data and establishing visual platform |
| CN116415206A (en) * | 2023-06-06 | 2023-07-11 | 中国移动紫金(江苏)创新研究院有限公司 | Operator multiple data fusion method, system, electronic equipment and computer storage medium |
| CN116415206B (en) * | 2023-06-06 | 2023-08-22 | 中国移动紫金(江苏)创新研究院有限公司 | Operator multiple data fusion method, system, electronic equipment and computer storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112214541A (en) | Deep decoupling and data cooperation method for power monitoring data | |
| Bansal et al. | A survey on iot big data: current status, 13 v’s challenges, and future directions | |
| US11836533B2 (en) | Automated reconfiguration of real time data stream processing | |
| US10447772B2 (en) | Managed function execution for processing data streams in real time | |
| US10685283B2 (en) | Demand classification based pipeline system for time-series data forecasting | |
| CN110430260B (en) | Robot cloud platform based on big data cloud computing support and working method | |
| CN109995546B (en) | Intelligent factory automation system with edge computing and cloud computing cooperating | |
| CN105224445A (en) | Distributed tracking system | |
| Obitko et al. | Big data challenges in industrial automation | |
| Bajic et al. | EDGE COMPUTING VS. CLOUD COMPUTING: CHALLENGES AND OPPORTUNITIES IN INDUSTRY 4.0. | |
| El‐Hasnony et al. | Leveraging mist and fog for big data analytics in IoT environment | |
| Arantes et al. | General architecture for data analysis in industry 4.0 using SysML and model based system engineering | |
| CN109639791A (en) | Cloud workflow schedule method and system under a kind of container environment | |
| CN113568938B (en) | Data stream processing method and device, electronic equipment and storage medium | |
| CN114372084A (en) | Real-time processing system for sensing stream data | |
| Zeydan et al. | Recent advances in data engineering for networking | |
| CN105550238A (en) | Architecture system of database appliance | |
| Cao et al. | Analytics everywhere for streaming iot data | |
| CN105320711A (en) | Mass data access method and system using the same | |
| CN112506887B (en) | Vehicle terminal CAN bus data processing method and device | |
| CN113676525A (en) | Network collaborative manufacturing-oriented industrial internet public service platform | |
| Borelli et al. | Architectural software patterns for the development of iot smart applications | |
| Halenár et al. | Design of a communication scheme in a modern factory in accordance with the standard of industry 4.0 | |
| Tianshu et al. | Intelligent prognostic and health management based on IOT cloud platform | |
| CN113836235A (en) | Data processing method based on data center and related equipment thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20210112 |
|
| WW01 | Invention patent application withdrawn after publication |