CN115982232A - Hadoop-based power grid data processing method and system - Google Patents

Abstract

The present invention provides a Hadoop-based power grid data processing method and system, the method comprising: step 1: collecting power grid big data based on Hadoop-based power grid data mining and analysis technology, the power grid big data including real-time data information and equipment parameters of the power grid Data, power generation and load data; Step 2: Based on MapReduce technology, store and manage the collected grid big data on the grid big data storage platform for data security situation storage; Step 3: Establish a zero-trust framework to realize power terminal security protection. The present invention can improve data quality, improve data storage security, improve terminal security protection, reduce the probability of data being tampered with, destroyed, and leaked, promote data circulation, give full play to the value of power grid data, and meet the national requirements for data sharing and exchange. Use data innovation, mine data dividends, and promote data economy to provide support.

Description

A Hadoop-based power grid data processing method and system

technical field

The invention relates to the technical field of smart grids, in particular to a Hadoop-based grid data processing method and system.

Background technique

With the increasing demand for social power energy, the intelligence of the power grid is also developing in depth; the customer-side energy consumption control system in the smart grid, as the link connecting customers and the smart energy service platform, is an important support for the ubiquitous power Internet of Things on the customer side. It is also the execution unit for implementing various comprehensive energy businesses such as demand response and energy efficiency improvement.

With the continuous expansion of the scale of the power grid, the complexity of the power grid operation continues to increase, and the problem of data security risks is also becoming increasingly prominent. In view of the lack of understanding of the common needs of the power grid for data collection and monitoring, demand response, etc., the development cost is high, and the portability and replication However, due to defects such as poor usability, it is necessary to use related technologies based on big data analysis to realize power grid data security situation storage, power grid data mining and analysis, and power terminal security protection.

Contents of the invention

The purpose of the present invention is to provide a Hadoop-based power grid data processing method and system to realize power grid data security situation storage, power grid data mining and analysis, and power terminal security protection, while solving the problems of high development costs, poor portability, and poor reusability of existing technologies And other issues.

A Hadoop-based power grid data processing method, comprising the following steps:

Step 1: Collect grid big data based on Hadoop-based power grid data mining and analysis technology. The grid big data includes real-time data information, equipment parameter data, power generation and load data of the grid;

Step 2: based on MapReduce technology, store and manage the collected grid big data on the grid big data storage platform for data security situation storage;

Step 3: Establish a zero trust framework to realize the security protection of power terminals.

Further, the Hadoop-based power grid data mining and analysis technology is implemented using a data acquisition layer, a data storage layer, a business application layer and a user layer;

The data collection layer adopts a distributed directional collection architecture and uses terminal sites in different networks as a basic task unit of network data collection to collect original network data and aggregate and transmit them to the data storage layer. The basic task unit adopts independent collection rules and strategies;

The data storage layer is used to complete the aggregation, storage and original processing of the original data of the data, and provide different types of function call services, and the data storage layer is realized by the Hadoop framework;

The business application layer is used to retrieve and analyze the network data processed by the data storage layer to realize the separation of public components and individual business application components, and transmit the analyzed results of the network data to the user layer for real-time display;

The user layer is used to transmit and display data information of the business application layer.

Further, the basic task unit includes a data collection unit, which is used to collect data through a dynamic webpage collection method and a webpage information extraction method, and extract information based on a row block distribution function, and then obtain data.

Further, the data collection unit acquires feed addresses by traversing the sites in breadth, collects information corresponding to each feed address in real time, tracks and updates information, and collects information in an incremental update manner.

Further, the collection rules and strategies include vertical search template semi-automatic generation technology, dynamic page optimization access technology and intelligent crawling process scheduling strategy.

Further, the processing of the original data in the data storage layer includes using the window technology to block the data to be processed, using the sliding window model to describe the change of the stream data, and using the sliding window model to save the pattern in the original data.

Further, use the sliding window model to save the pattern in the original data, specifically:

Block the data according to the change of the data, store the mode of the unchanged part of the data in the sliding window; calculate the mode of adding and deleting part of the data respectively; update the mode saved in the sliding window according to the mode of the changed part of the data;

Use the multi-window method to support users' online mining requests; the multi-window method divides the data stream into multiple fixed-length segments, and each segment forms a window. When the number of windows in the memory reaches a certain number, the multiple Merge windows to form a window with a higher summary level. With the inflow of data streams, multiple windows with different summary levels form a hierarchical structure. At this time, each window is equivalent to the data between two predefined time stamps on the data stream. A snapshot of .

Further, the storage of data security situation based on MapReduce technology includes the following steps:

Step 2.1: Call the data information of the user layer and input it to the user program;

Step 2.2: The MapReduce library divides the input file of the user program into M parts, and M is defined by the user;

Step 2.3: The worker assigned to the Map job reads the input data of the corresponding shard, and the Map job extracts key-value pairs from the input data. Each key-value pair is passed as a parameter to the map function, and the intermediate key value generated by the map function pairs are cached in memory;

Step 2.3: The cached intermediate key values are regularly written to the local disk and divided into R areas. The size of R is defined by the user. In the future, each area will correspond to a Reduce job; the location of the intermediate key-value pairs will be notified to master, the master is responsible for forwarding the information to the Reduce worker;

Step 2.5: The master notifies the worker assigned to the Reduce job to be responsible for the specific location of the partition. After the Reduceworker reads all the responsible intermediate key values, it sorts the intermediate key values so that the key-value pairs of the same key are gathered together;

Step 2.5: The reduce worker traverses the sorted intermediate key-value pairs, and for each unique key, passes the key and associated value to the reduce function, and the output generated by the reduce function will be added to the output file of this partition;

Step 2.7: When all Map and Reduce jobs are completed, the master wakes up the user program, and the MapReduce function call returns the code of the user program.

Further, the establishment of a zero-trust framework to realize the security protection of electric power terminals includes the following steps:

Step 3.1, build a zero trust module, collect the equipment information of the power terminal equipment, and perform a trust score based on the collected equipment information, give a trust value, evaluate the power terminal equipment according to the trust value, and classify the power terminal equipment into Trusted equipment, abnormal equipment;

Step 3.2, collect data from the trusted device in step 3.1, and obtain collected data;

Step 3.3, building a security situation awareness module, performing situation awareness on the collected data in step 3.2, and converting the collected data into perception data when the perception is qualified;

Step 3.4, build a real-time control module, control the sensing data in step 3.3, and generate security instructions;

Step 3.5, sending the safety instruction in step 3.4 to the power terminal equipment, and performing security protection and security reinforcement on the power terminal equipment.

A Hadoop-based grid data processing system, comprising: a computer-readable storage medium and a processor;

The computer-readable storage medium is used to store executable instructions;

The processor is configured to read executable instructions stored in the computer-readable storage medium, and execute the Hadoop-based grid data processing method.

The present invention improves the security protection ability for grid data security business, improves the identification accuracy of data security events, and the timeliness of traceability, reduces the probability of data being tampered with, destroyed, and leaked, promotes data circulation, and fully utilizes the value of grid data. The requirements of sharing and exchange provide support for using data innovation, mining data dividends, and promoting the data economy.

Description of drawings

Fig. 1 is a flow chart of a Hadoop-based grid data processing method according to an embodiment of the present invention;

FIG. 2 is a flowchart of data security situation storage based on MapReduce technology in an embodiment of the present invention.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments It is a part of embodiments of the present invention, but not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Referring to Fig. 1, the first aspect of the present invention provides a Hadoop-based grid data processing method, comprising the following steps:

Step 1: Collect grid big data based on Hadoop-based power grid data mining and analysis technology. The grid big data includes real-time data information, equipment parameter data, power generation and load data of the grid;

Step 2: based on MapReduce technology, store and manage the collected grid big data on the grid big data storage platform for data security situation storage;

Step 3: Establish a zero trust framework to realize the security protection of power terminals.

The Hadoop-based power grid data mining and analysis technology described in the present invention is realized by using a data acquisition layer, a data storage layer, a business application layer and a user layer.

The data acquisition layer adopts a distributed directional acquisition system architecture and uses terminal sites in different networks as a basic task unit for grid data acquisition to collect real-time data information, equipment parameter data, power generation and load data, and send data to the data collection layer. Convergence and transmission at the storage layer; the basic task unit includes a data collection unit, which is used to collect data through a dynamic webpage collection method and a webpage information extraction method, and extract information by using a method based on a row block distribution function, and then obtain data. Specifically, The data collection unit acquires feed addresses by traversing the sites in breadth, collects information corresponding to each feed address in real time, tracks and updates information, and collects information in an incremental update manner. Wherein, each basic task unit adopts independent acquisition rules and strategies; said acquisition rules and strategies include semi-automatic generation technology of vertical search template, dynamic page optimization access technology and intelligent crawling process scheduling strategy.

The data storage layer is used to complete the aggregation, storage and original processing of the original data of the data, and provide different types of function call services; the data storage layer is implemented using the Hadoop framework;

The business application layer is used to retrieve and analyze the data processed by the data storage layer to realize the separation of public components and individual business application components, and transmit the analyzed results of network data to the user layer for real-time display.

The user layer is used to transmit and display data information of the business application layer.

The storage of data security situation based on MapReduce technology adopts distributed file system HDFS and MapReduce to realize, and described distributed file system HDFS is the file system of Hadoop, is used for storing ultra-large files; MapReduce is the parallel programming model of Hadoop, is used for In-depth analysis of data stored on the distributed file system HDFS.

As shown in Figure 2, the described storage of data security situation based on MapReduce technology includes the following steps:

Step 2.1: Call the data information of the user layer and input it to the user program;

Step 2.2: The MapReduce library divides the input file of the user program into M parts, and M is defined by the user;

Step 2.3: The worker assigned the Map job starts to read the input data corresponding to the shard. The number of Map jobs is determined by M, which corresponds to the split one by one; the Map job extracts key-value pairs from the input data, and each key Value pairs are passed to the map function as parameters, and the intermediate key-value pairs generated by the map function are cached in memory;

Step 2.3: The cached intermediate key values will be regularly written to the local disk and divided into R areas. The size of R is defined by the user. In the future, each area will correspond to a Reduce job; the location of these intermediate key-value pairs will be notified to the master, and the master is responsible for forwarding the information to the Reduce worker;

Step 2.5: The master notifies the worker assigned the Reduce job to be responsible for the specific location of the partition. When the Reduce worker reads all the intermediate key-value pairs it is responsible for, it first sorts them so that the key-value pairs with the same key are gathered in Together;

Step 2.5: The reduce worker traverses the sorted intermediate key-value pairs, and for each unique key, passes the key and associated value to the reduce function, and the output generated by the reduce function will be added to the output file of this partition;

Step 2.7: When all Map and Reduce jobs are completed, the master wakes up the user program, and the MapReduce function call returns the code of the user program.

The establishment of a zero-trust framework to realize the security protection of electric power terminals includes the following steps:

Step 3.1: Construct a zero-trust module to collect equipment information of power terminal equipment; perform a trust score based on the collected equipment information, and give a trust value; evaluate power terminal equipment according to the trust value, and classify power terminal equipment into Trusted equipment, abnormal equipment;

The process of collecting device information by the zero trust module is: reading device data, reading rule files, parsing the rule base, and collecting device information; at the same time, the zero trust module performs continuous dynamic device identity verification on power terminal devices to block false Device information; the trust value is an indicator of identity verification, which is obtained by comprehensive scoring based on the basic attributes of the device and access delay; the maintenance of the trust value includes the following:

(1) The maximum trust value is M, and the minimum is N; M>N

(2) The trust value threshold is H, which is higher than or equal to H as a legitimate user, and lower than H is an illegal user;

(3) Add T to the trust value for each successful verification;

(4) The trust value minus T for each verification failure;

The trust value includes a direct trust value, a delay evaluation trust value, and an abnormal behavior evaluation trust value, and its calculation formula is as follows:

T=T _d +T _t +T _a

T is the trust value, T _d is the direct trust value, T _t is the delay evaluation trust value, T _a is the abnormal behavior evaluation trust value;

The direct trust value is an S-type function, and its calculation formula is:

Where T _d is the direct trust value, f is the direct trust value constraint coefficient of different devices; the time delay evaluation trust value and the abnormal behavior evaluation trust value form the indirect trust value;

The delay evaluation trust value is evaluated based on the device response time, and its calculation formula is:

Among them, T _t is the trust value of delay evaluation, τ is the maximum allowable delay of equipment response, and D is the amount of information transmission delay;

The abnormal behavior evaluation trust value is evaluated according to the ratio of abnormal behavior to normal behavior of the device. The calculation formula is:

where T _a is the trust value of abnormal behavior evaluation,

A _u is the amount of abnormal behavior, A _n is the amount of normal behavior;

Step 3.2: Collect data from the trusted device in step 3.1 to obtain collected data;

Step 3.3: Construct a security situation awareness module to perform situation awareness on the collected data in step 3.2; when the perception is qualified, convert the collected data into perception data;

The situational awareness includes intrusion detection, vulnerability awareness, file integrity detection, and log monitoring operations;

Step 3.4: Build a real-time management and control module to control the sensing data in step 3.3 and generate safety instructions;

Step 3.5: Issue the security instruction in step 3.4 to the power terminal equipment, and perform security protection and security reinforcement for the power terminal equipment;

The present invention can improve data quality, improve data storage security, improve terminal security protection, reduce the probability of data being tampered with, destroyed, and leaked, promote data circulation, and give full play to the value of power grid data, which meets the national requirements for data sharing and exchange. Use data innovation, mine data dividends, and promote data economy to provide support.

Another aspect of the present invention provides a Hadoop-based grid data processing system, including: a computer-readable storage medium and a processor;

The computer-readable storage medium is used to store executable instructions;

The processor is configured to read executable instructions stored in the computer-readable storage medium, and execute the Hadoop-based power grid data processing method described in the first aspect.

Another aspect of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the Hadoop-based grid data processing method described in the first aspect is implemented.

Those skilled in the art should understand that the embodiments of the present application may be provided as methods, systems, or computer program products. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.

The present application is described with reference to flowcharts and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each procedure and/or block in the flowchart and/or block diagram, and a combination of procedures and/or blocks in the flowchart and/or block diagram can be realized by computer program instructions. These computer program instructions may be provided to a general purpose computer, special purpose computer, embedded processor, or processor of other programmable data processing equipment to produce a machine such that the instructions executed by the processor of the computer or other programmable data processing equipment produce a An apparatus for realizing the functions specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing apparatus to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising instruction means, the instructions The device realizes the function specified in one or more procedures of the flowchart and/or one or more blocks of the block diagram.

These computer program instructions can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby The instructions provide steps for implementing the functions specified in the flow chart or blocks of the flowchart and/or the block or blocks of the block diagrams.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to the above embodiments, those of ordinary skill in the art should understand that: the present invention can still be Any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention shall fall within the protection scope of the claims of the present invention.

Claims (10)

1. A power grid data processing method based on Hadoop is characterized by comprising the following steps:

the method comprises the following steps: the method comprises the steps that power grid big data are collected through a Hadoop-based power grid data mining and analyzing technology, wherein the power grid big data comprise real-time data information, equipment parameter data and power generation and load data of a power grid;

step two: storing and managing the collected power grid big data on a power grid big data storage platform based on a MapReduce technology to store data security situations;

step three: and establishing a zero trust framework to realize the safety protection of the power terminal.

2. The Hadoop-based power grid data processing method according to claim 1, wherein the Hadoop-based power grid data mining and analyzing technology is implemented by a data acquisition layer, a data storage layer, a service application layer and a user layer;

the data acquisition layer adopts a distributed directional acquisition system architecture, and terminal stations in different networks are used as a basic task unit for network data acquisition to acquire original network data and gather and transmit the original network data to the data storage layer, wherein each basic task unit adopts an independent acquisition rule and strategy;

the data storage layer is used for finishing the aggregation, storage and original processing of original data of the data and providing different types of function calling services, and the data storage layer is realized by adopting a Hadoop framework;

the service application layer is used for calling and analyzing the network data processed by the data storage layer to realize the stripping of the public component and the individual service application component and transmitting the result of the network data analysis to the user layer for real-time display;

and the user layer is used for transmitting and displaying the data information of the service application layer.

3. The Hadoop-based power grid data processing method as claimed in claim 2, wherein the basic task unit comprises a data acquisition unit for acquiring data by a dynamic web page acquisition method and a web page information extraction method, and extracting information by a method based on a row-block distribution function to further acquire data.

4. The Hadoop-based power grid data processing method as claimed in claim 3, wherein the data acquisition unit acquires Feed addresses through a breadth traversal site, acquires information corresponding to each Feed address in real time, tracks updated information, and acquires information in an incremental updating manner.

5. The Hadoop-based power grid data processing method as claimed in claim 2, wherein the collection rules and policies include vertical search template semi-automatic generation technology, dynamic page optimization access technology, and intelligent capture process scheduling policy.

6. The Hadoop-based power grid data processing method according to claim 2, wherein the processing of the raw data in the data storage layer comprises blocking the data to be processed by using a window technique, describing changes in the stream data by using a sliding window model, and saving a pattern in the original data by using the sliding window model.

7. The Hadoop-based power grid data processing method as claimed in claim 6, wherein a sliding window model is used to store patterns in the original data, specifically:

storing the mode of the unchanged partial data into a sliding window according to the changed block data of the data; respectively calculating modes of adding and deleting partial data; updating the mode stored in the sliding window according to the mode of the changed part of data;

using a multi-window method to support the online mining request of a user; the multi-window method divides the data stream into a plurality of segments with fixed length, each segment forms a window, when the number of windows in the memory reaches a certain number, the windows are combined to form a window with higher summary level along with the inflow of the data stream, the windows with different summary levels form a hierarchical structure, and at the moment, each window is equivalent to a snapshot of data between two predefined time stamps on the data stream.

8. The Hadoop-based power grid data processing method according to claim 1, wherein the data security situation storage based on the MapReduce technology comprises the following steps:

step 2.1: calling data information of a user layer and inputting the data information into a user program;

step 2.2: dividing an input file of a user program into M parts by a MapReduce library, wherein M is defined by a user;

step 2.3: reading input data of the corresponding fragments by a worker to which Map operation is allocated, extracting key value pairs from the input data by the Map operation, transmitting each key value pair to a Map function as a parameter, and caching middle key value pairs generated by the Map function in a memory;

step 2.3: the cached intermediate key value is periodically written into a local disk and is divided into R areas, the size of R is defined by a user, and each area corresponds to a Reduce operation in the future; the position of the middle key-value pair is notified to a master, and the master is responsible for forwarding the information to a Reduce worker;

step 2.5: the master informs the specific position of a partition responsible for a worker distributing Reduce operation, and after the Reduce worker reads all responsible intermediate key values, the intermediate key values are sorted so that the key value pairs of the same key are gathered together;

step 2.5: traversing the sorted intermediate key value pairs by the reduce worker, transmitting the key and the associated value to a reduce function for each unique key, and adding the output generated by the reduce function into the output file of the partition;

step 2.7: when all Map and Reduce jobs are completed, the master wakes up the user program, and the MapReduce function call returns the code of the user program.

9. The Hadoop-based power grid data processing method as claimed in claim 1, wherein the establishing of the zero trust framework to realize the power terminal security protection comprises the following steps:

step 3.1, a zero trust module is constructed, equipment information of the electric power terminal equipment is collected, trust scoring is carried out according to the collected equipment information, a trust value is given, the electric power terminal equipment is evaluated according to the trust value, and the electric power terminal equipment is divided into trusted equipment and abnormal equipment;

step 3.2, data acquisition is carried out on the trusted equipment in the step 3.1, and acquired data are obtained;

3.3, constructing a security situation awareness module, carrying out situation awareness on the acquired data in the step 3.2, and converting the acquired data into awareness data when the awareness is qualified;

step 3.4, a real-time management and control module is constructed, the perception data in the step 3.3 are managed and controlled, and a safety instruction is generated;

and 3.5, issuing the safety command in the step 3.4 to the electric power terminal equipment, and carrying out safety protection and safety reinforcement on the electric power terminal equipment.

10. A Hadoop-based power grid data processing system comprises: a computer-readable storage medium and a processor;

the computer-readable storage medium is used for storing executable instructions;

the processor is used for reading executable instructions stored in the computer-readable storage medium and executing the Hadoop-based power grid data processing method of any one of claims 1 to 9.

Images