CN116109058A - Substation inspection management method and device based on deep reinforcement learning - Google Patents
Substation inspection management method and device based on deep reinforcement learning Download PDFInfo
- Publication number
- CN116109058A CN116109058A CN202211502140.XA CN202211502140A CN116109058A CN 116109058 A CN116109058 A CN 116109058A CN 202211502140 A CN202211502140 A CN 202211502140A CN 116109058 A CN116109058 A CN 116109058A
- Authority
- CN
- China
- Prior art keywords
- cloud
- task
- edge
- substation
- scheduling model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000007726 management method Methods 0.000 title claims abstract description 43
- 238000007689 inspection Methods 0.000 title claims abstract description 41
- 230000002787 reinforcement Effects 0.000 title claims abstract description 27
- 238000012549 training Methods 0.000 claims abstract description 51
- 238000004364 calculation method Methods 0.000 claims abstract description 12
- 238000012545 processing Methods 0.000 claims description 14
- 230000035945 sensitivity Effects 0.000 claims description 6
- 238000013468 resource allocation Methods 0.000 claims description 5
- 238000010276 construction Methods 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 7
- 230000009471 action Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 8
- 238000004519 manufacturing process Methods 0.000 description 6
- 238000012423 maintenance Methods 0.000 description 4
- 238000013439 planning Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 238000007405 data analysis Methods 0.000 description 2
- 238000013480 data collection Methods 0.000 description 2
- 238000013523 data management Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000011161 development Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000012806 monitoring device Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06311—Scheduling, planning or task assignment for a person or group
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06312—Adjustment or analysis of established resource schedule, e.g. resource or task levelling, or dynamic rescheduling
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0631—Resource planning, allocation, distributing or scheduling for enterprises or organisations
- G06Q10/06313—Resource planning in a project environment
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
- G06Q10/063—Operations research, analysis or management
- G06Q10/0633—Workflow analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q10/00—Administration; Management
- G06Q10/10—Office automation; Time management
- G06Q10/103—Workflow collaboration or project management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
- G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
- G06Q50/06—Energy or water supply
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Business, Economics & Management (AREA)
- Human Resources & Organizations (AREA)
- Engineering & Computer Science (AREA)
- Strategic Management (AREA)
- Economics (AREA)
- Entrepreneurship & Innovation (AREA)
- Tourism & Hospitality (AREA)
- Marketing (AREA)
- Physics & Mathematics (AREA)
- General Business, Economics & Management (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Operations Research (AREA)
- Quality & Reliability (AREA)
- Development Economics (AREA)
- Game Theory and Decision Science (AREA)
- Educational Administration (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Data Mining & Analysis (AREA)
- Biodiversity & Conservation Biology (AREA)
- Public Health (AREA)
- Water Supply & Treatment (AREA)
- General Health & Medical Sciences (AREA)
- Primary Health Care (AREA)
- Supply And Distribution Of Alternating Current (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
The invention provides a substation inspection management method and device based on deep reinforcement learning, comprising the steps of constructing an inference task scheduling model which takes real-time calculation forces of edge nodes and cloud as input and takes specific edge node equipment numbers or cloud as prediction results for output by utilizing a Muzero algorithm; the method comprises the steps of training an inference task scheduling model by using data information generated by various patrol tasks in a transformer substation with load balancing as a target; based on the current state of the transformer substation, performing node prediction by using a trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation. According to the invention, through a deep reinforcement learning model based on a Muzero algorithm, the edge nodes and the cloud are used as characteristic input networks together, and the distributed edge nodes are selected with load balance as a target, so that edge cloud coordination is realized, and the reliability and the operation efficiency of a substation inspection system are improved.
Description
Technical Field
The invention belongs to the technical field of substation management, and particularly relates to a substation inspection management method and device based on deep reinforcement learning.
Background
With the rapid development of intelligent digital power grids, the number of substations is continuously increased, the operation and maintenance workload is continuously increased, part of the substations are remote and distributed and scattered, and when special weather is met, manual inspection is blocked, so that the states of equipment in the substations and the field environment cannot be mastered in time. Therefore, the unmanned inspection system of the transformer substation, which can replace manual inspection of the transformer substation, reduces the workload of operators on duty and improves inspection efficiency and quality, is necessary.
The power transformation field has the problems of large manual management workload, multiple types of algorithm platforms, low automation degree and the like; the patrol equipment terminal is provided with a large number of image and video monitoring devices, and the problems of mass data collection, management, analysis and identification and the like are difficult to effectively cope with.
With the deep advancement of intelligent technology in the technical field of production, the application level of various production video (image) data of a substation inspection system is faced with challenges of intelligent operation and detection systems such as power grid equipment, channel environment, operation and maintenance, overhaul and production management: the 'cloud and edge' have insufficient cooperative capability. The pressure of the main station platform is overlarge, including network bandwidth pressure, centralized calculation pressure and centralized storage pressure, and the sub-stations need to exert cloud edge cooperative advantages through edge calculation.
Disclosure of Invention
In view of the above, the invention aims to solve the problem that the existing substation inspection system cannot realize intelligent coordination and unified dispatching of end-to-side clouds, so that the pressure of a main station platform is overlarge.
In order to solve the technical problems, the invention provides the following technical scheme:
in a first aspect, the present invention provides a substation inspection management method based on deep reinforcement learning, including the following steps:
constructing an inference task scheduling model which takes real-time computing power of the edge node and the cloud as input and takes a specific edge node equipment number or the cloud as a prediction result for output by using a Muzero algorithm;
training an inference task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, wherein the load balancing is used for distributing resources based on real-time computing power of edge nodes and cloud, so that average service time of the patrol tasks is minimized;
based on the current state of the transformer substation, performing node prediction by using a trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation.
Further, the reasoning task scheduling model is deployed at the cloud end, the cloud end performs centralized training on the reasoning task scheduling model, and the training is distributed to each edge node after the training is completed.
Further, with load balancing as a target, training an inference task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation, specifically including:
dividing a substation inspection task into a computation-intensive task and a delay-sensitive task;
for delay sensitive tasks, edge nodes are distributed for processing, and for computationally intensive tasks, the tasks are sent to a cloud for processing;
training an inference task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation;
and calculating task failure rates in all the inspection tasks, and optimizing the reasoning task scheduling model by taking the lowest task failure rate as a target.
Further, the data information includes real-time computing power of the edge node and the cloud, network bandwidth and delay, task size and delay sensitivity.
Further, the reasoning task scheduling model specifically includes: a terminal layer, an edge calculation layer and a cloud layer;
the terminal layer comprises a plurality of types of equipment, the equipment periodically collects environment data and generates a computation intensive task and a delay sensitive task, and the terminal equipment is accessed to the edge computing layer through TCP;
the edge computing layer is used for providing edge computing resources for the terminal equipment and is connected with the cloud layer through a 5G network;
the cloud layer is used for providing cloud computing capability for the terminal equipment.
In a second aspect, the present invention provides a substation patrol management device based on deep reinforcement learning, including:
the model construction module is used for constructing an inference task scheduling model which takes real-time computing power of the edge node and the cloud as input and takes a specific edge node equipment number or the cloud as a prediction result for output by using a Muzero algorithm;
the model training module is used for training the reasoning task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, and the load balancing is used for distributing resources based on real-time computing power of edge nodes and cloud so as to minimize the average service time of the patrol tasks;
the resource allocation module is used for carrying out node prediction by utilizing the trained reasoning task scheduling model based on the current state of the transformer substation, and allocating edge computing resources for the patrol task according to the prediction result so as to realize load balance of patrol management of the transformer substation.
Furthermore, in the model training module, the reasoning task scheduling model is deployed at the cloud end, the cloud end performs centralized training on the reasoning task scheduling model, and the training is distributed to each edge node.
Further, in the model training module, with load balancing as a target, training the reasoning task scheduling model by using data information generated by various patrol tasks in the transformer substation, specifically including:
dividing a substation inspection task into a computation-intensive task and a delay-sensitive task;
for delay sensitive tasks, edge nodes are distributed for processing, and for computationally intensive tasks, the tasks are sent to a cloud for processing;
training an inference task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation;
and calculating task failure rates in all the inspection tasks, and optimizing the reasoning task scheduling model by taking the lowest task failure rate as a target.
Further, in the model training module, the data information includes real-time computing power of the edge node and the cloud, network bandwidth and delay, task size and delay sensitivity.
Further, in the model building module, the reasoning task scheduling model specifically includes: a terminal layer, an edge calculation layer and a cloud layer;
the terminal layer comprises a plurality of types of equipment, the equipment periodically collects environment data and generates a computation intensive task and a delay sensitive task, and the terminal equipment is accessed to the edge computing layer through TCP;
the edge computing layer is used for providing edge computing resources for the terminal equipment and is connected with the cloud layer through a 5G network;
the cloud layer is used for providing cloud computing capability for the terminal equipment.
In summary, the invention provides a substation inspection management method and device based on deep reinforcement learning, which comprises the steps of constructing an inference task scheduling model which takes real-time computing power of edge nodes and cloud as input and takes specific edge node equipment numbers or cloud as a prediction result for output by utilizing a Muzero algorithm; training an inference task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, wherein the load balancing is used for distributing resources based on real-time computing power of edge nodes and cloud, so that average service time of the patrol tasks is minimized; based on the current state of the transformer substation, performing node prediction by using a trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation. According to the invention, through a deep reinforcement learning model based on a Muzero algorithm, the edge nodes and the cloud are used as characteristic input networks together, and the distributed edge nodes are selected with load balance as a target, so that edge cloud coordination is realized, and the reliability and the operation efficiency of a substation inspection system are improved.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained from these drawings without inventive faculty for a person skilled in the art.
Fig. 1 is a schematic flow chart of a substation inspection management method based on deep reinforcement learning according to an embodiment of the present invention;
fig. 2 is a relationship diagram of edge nodes and cloud end provided in an embodiment of the present invention.
Detailed Description
In order to make the objects, features and advantages of the present invention more obvious and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is apparent that the embodiments described below are only some embodiments of the present invention, not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
With the rapid development of intelligent digital power grids, the number of substations is continuously increased, the operation and maintenance workload is continuously increased, part of the substations are remote and distributed and scattered, and when special weather is met, manual inspection is blocked, so that the states of equipment in the substations and the field environment cannot be mastered in time. Therefore, the unmanned inspection system of the transformer substation, which can replace manual inspection of the transformer substation, reduces the workload of operators on duty and improves inspection efficiency and quality, is necessary.
The power transformation field has the problems of large manual management workload, multiple types of algorithm platforms, low automation degree and the like; the patrol equipment terminal is provided with a large number of image and video monitoring devices, and the problems of mass data collection, management, analysis and identification and the like are difficult to effectively cope with.
With the deep advancement of intelligent technology in the technical field of production, the application level of various production video (image) data of a substation inspection system is faced with challenges of intelligent operation and detection systems such as power grid equipment, channel environment, operation and maintenance, overhaul and production management: the 'cloud and edge' have insufficient cooperative capability. The pressure of the main station platform is overlarge, including network bandwidth pressure, centralized calculation pressure and centralized storage pressure, and the sub-stations need to exert cloud edge cooperative advantages through edge calculation.
Based on the method, the invention provides a substation inspection management method and device based on deep reinforcement learning.
Referring to fig. 1, the present embodiment provides a substation inspection management method based on deep reinforcement learning, which includes the following steps:
s100: and constructing an inference task scheduling model which takes real-time computing power of the edge node and the cloud as input and takes a specific edge node equipment number or the cloud as a prediction result for output by using a Muzero algorithm.
It should be noted that the inference task scheduling model constructed by the MuZero algorithm is an abstract Markov decision process (Markov DecisionProcess, MDP) model, on which future data (policies, cost functions and rewards) directly related to the planning are predicted, and on the basis of the future data, the planning is performed by predicting the data, so as to select the operation nodes of the inference task.
The MuZero algorithm combines tree-based searches with learned models, first converting states acquired in the real environment to a hidden state in an abstract state space without direct constraints by an encoder. The hidden state is a dynamic model trained in an abstract state space and a value prediction network, and the value and the return of the future k steps can be predicted after the initial hidden state and the future k steps are executed, so that the value and the return of the future k steps are as close as possible to the value and the observed return of the real environment through searching.
The planning in the abstract MDP remains equivalent to the planning in the real environment by ensuring that the value equivalents are maintained. I.e. starting from the same real state, the accumulated reward of the trace through the abstract MDP matches the accumulated reward of the trace in the real environment. Given a hidden state and a candidate action, the dynamic model needs to generate an instant prize and a new hidden state. The strategy and value functions are calculated from the prediction functions by input. The actions are sampled from the search strategy. The environment receives an action to generate a new observation and rewards.
The reasoning task scheduling model is set to be three layers, namely a terminal layer, an edge computing layer and a cloud layer from bottom to top, wherein the terminal layer comprises a plurality of types of equipment, each equipment is represented by a symbol u, the equipment periodically collects environment data and generates computationally intensive and delay sensitive tasks; the terminal equipment is accessed to an edge computing layer through TCP, and the edge end is responsible for providing edge computing resources for the terminal equipment; the edge layer comprises edge infrastructure capability EC-IaaS, which is composed of a wireless base station, a small data center (edge server), edge nodes and interfaces, and mainly provides computing, storage, network and virtualized resources; the edge platform capability EC-PaaS provides functions for the edge cloud environment, and mainly performs preprocessing and analysis of data and deployment and arrangement of applications; the edge application capability EC-SaaS expands the functions to the edge, and provides services for the application program to the greatest extent. The edge computing layer is connected with the cloud layer through 5G, and the cloud comprises infrastructure capability IaaS for providing infrastructure such as computing, storage, network and virtual machines; platform capability PaaS provides functions of device management, resource management, data processing and analysis, data modeling and analysis, service components, edge management, and business orchestration. The cloud computing power is denoted by the symbol fc. The relationship of the edge computation layer and the cloud layer is shown in fig. 2.
S200: and training an inference task scheduling model by using data information generated by various patrol tasks in the transformer substation by taking load balancing as a target, wherein the load balancing is used for distributing resources based on real-time computing power of edge nodes and cloud, so that the average service time of the patrol tasks is minimized.
The training of the reasoning task scheduling model is performed based on data information generated by various patrol tasks in the transformer substation, the reasoning task scheduling model utilizes the data information to conduct cloud-edge node selection prediction, and the transformer substation patrol management system conducts resource allocation according to a prediction result, so that average service time (sum of task processing time and communication delay) is reduced, and load balance is achieved.
The reasoning task scheduling model is deployed at the cloud for centralized training, and is distributed to each edge node after training is completed. And at each edge node, based on the environmental information collected by each terminal device, carrying out node reasoning to obtain the probability of the device receiving the task, and for the computationally intensive task, obtaining the probability of unloading the task from the edge server to the cloud server.
For training of the inference task scheduling model constructed based on the Muzero algorithm, the state S, action A, reward R and value v (S) are defined as follows:
the state S is a CPU usage rate of each edge server and cloud server, a network usage state such as LAN and WAN between the device and each server, a task size, and real-time attributes. The cloud server parameters comprise the number of servers, the number of virtual machines, the number of cores of the virtual machines, the average execution speed of single word length fixed-point instructions, memory and storage capacity.
The state S definition contains the following: (1) CPU usage (edge/cloud), network (local area network bandwidth, WAN bandwidth, LAN latency, WAN latency), task size, and latency sensitivity (load check interval, location check interval, device counter size).
In the proposed method, the real-time properties of the tasks are regarded as one of the factors that determine the actions, and whether each task requires real-time properties is set.
The action space a contains the following: (1) edge processing; (2) offloading to the cloud. Action a either processes the task at the edge server or offloads the task from the edge server to the cloud server. The reward R is: the service is time consuming. The reward R represents the service time of the task. The smaller the service time, the larger the prize, and the better the value v(s) can be obtained. Based on the definition and the data information generated by the substation patrol task, training of an inference task scheduling model is achieved.
For a general model constructed based on the Muzero algorithm, the training process is as follows:
s1: obtaining a strategy p by using an MCTS algorithm k And estimated value v k 。
S2: selecting action a from a policy t+1 This action is performed and an intermediate prize and state space is generated.
S3: in each step, policy, value, reward is matched to the actual observed target image by co-training all parameters of the model.
S4: all parameters of the model are jointly trained to match the strategy, value and rewards of each hypothesized step k exactly to the corresponding target values observed after the k actual time steps.
S5: adding an L2 regularization term to obtain a final loss function;
s6: and repeating the steps S3-S4, and finishing training when the loss function converges.
S300: based on the current state of the transformer substation, performing node prediction by using a trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation.
It should be noted that, for the substation patrol task, it is divided into a computationally intensive and a delay sensitive task, and the delay sensitive task is further divided into two levels of level 0 and level 1. The task set to level 1 is sensitive to communication latency and should therefore be handled on the edge server to reduce communication latency. As the number of devices increases, the number of tasks to be processed increases, and the substation inspection management system discards tasks that have not been saved waiting for processing or cannot be transmitted due to network congestion, and these tasks are regarded as failed tasks. The smaller the task failure rate, the higher the efficiency of the load balancing method. Therefore, the training goal of the reasoning task scheduling model is to minimize the task failure rate of the whole system as much as possible, thereby realizing load balancing.
The embodiment provides a substation inspection management method based on deep reinforcement learning, which comprises the steps of constructing an inference task scheduling model which takes real-time calculation forces of edge nodes and cloud as input and takes specific edge node equipment numbers or cloud as prediction results to output by utilizing a Muzero algorithm; training an inference task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, wherein the load balancing is used for distributing resources based on real-time computing power of edge nodes and cloud, so that average service time of the patrol tasks is minimized; based on the current state of the transformer substation, performing node prediction by using a trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation. According to the invention, through a deep reinforcement learning model based on a Muzero algorithm, the edge nodes and the cloud are used as characteristic input networks together, and the distributed edge nodes are selected with load balance as a target, so that edge cloud coordination is realized, and the reliability and the operation efficiency of a substation inspection system are improved.
The foregoing is a detailed description of an embodiment of a substation inspection management method based on deep reinforcement learning according to the present invention, and the following is a detailed description of an embodiment of a substation inspection management device based on deep reinforcement learning according to the present invention.
The invention provides a substation inspection management device based on deep reinforcement learning, which comprises: the system comprises a model construction module, a model training module and a resource allocation module.
In this embodiment, the model building module is configured to build an inference task scheduling model using real-time computing power of the edge node and the cloud as input and using a specific edge node device number or the cloud as a prediction result to output by using a MuZero algorithm.
It should be noted that, in the model building module, the reasoning task scheduling model specifically includes: a terminal layer, an edge calculation layer and a cloud layer;
the terminal layer comprises a plurality of types of equipment, the equipment periodically collects environment data and generates a computation intensive task and a delay sensitive task, and the terminal equipment is accessed to the edge computing layer through TCP;
the edge computing layer is used for providing edge computing resources for the terminal equipment and is connected with the cloud layer through a 5G network;
the cloud layer is used for providing cloud computing capability for the terminal equipment.
In this embodiment, the model training module is configured to train the inference task scheduling model by using data information generated by various patrol tasks in the substation with load balancing as a target, where load balancing is to allocate resources based on real-time computing forces of the edge node and the cloud, so that average service time of the patrol tasks is minimized.
In the model training module, the inference task scheduling model is deployed at the cloud end, the cloud end performs centralized training on the inference task scheduling model, and the inference task scheduling model is distributed to each edge node after the training is completed.
Further, in the model training module, with load balancing as a target, training the reasoning task scheduling model by using data information generated by various patrol tasks in the transformer substation, specifically including:
dividing a substation inspection task into a computation-intensive task and a delay-sensitive task;
for delay sensitive tasks, edge nodes are distributed for processing, and for computationally intensive tasks, the tasks are sent to a cloud for processing;
training an inference task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation;
and calculating task failure rates in all the inspection tasks, and optimizing the reasoning task scheduling model by taking the lowest task failure rate as a target.
Further, in the model training module, the data information includes real-time computing power of the edge node and the cloud, network bandwidth and delay, task size and delay sensitivity.
In this embodiment, the resource allocation module is configured to perform node prediction by using a trained inference task scheduling model based on a current state of the substation, and allocate edge computing resources for the patrol task according to a prediction result, so as to implement load balance of patrol management of the substation.
The above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. The substation inspection management method based on deep reinforcement learning is characterized by comprising the following steps of:
constructing an inference task scheduling model which takes real-time computing power of the edge node and the cloud as input and takes a specific edge node equipment number or the cloud as a prediction result for output by using a Muzero algorithm;
training the reasoning task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, wherein the load balancing is to allocate resources based on real-time computing power of the edge nodes and the cloud so as to minimize average service time of the patrol tasks;
and based on the current state of the transformer substation, performing node prediction by using the trained reasoning task scheduling model, and distributing edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation.
2. The substation inspection management method based on deep reinforcement learning according to claim 1, wherein the inference task scheduling model is deployed at a cloud end, the cloud end performs centralized training on the inference task scheduling model, and the inference task scheduling model is distributed to each edge node after training is completed.
3. The substation patrol management method based on deep reinforcement learning according to claim 2, wherein the training of the inference task scheduling model by using data information generated by various patrol tasks in the substation is performed with load balancing as a target, specifically comprises:
dividing a substation inspection task into a computation-intensive task and a delay-sensitive task;
for the delay sensitive task, the edge node is distributed for processing, and for the computation intensive task, the computation intensive task is sent to the cloud for processing;
training the reasoning task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation;
and calculating task failure rates in all the inspection tasks, and optimizing the reasoning task scheduling model by taking the lowest task failure rate as a target.
4. The deep reinforcement learning-based substation patrol management method according to claim 3, wherein the data information includes real-time computing power of the edge node and the cloud, network bandwidth and delay, task size and delay sensitivity.
5. The substation patrol management method based on deep reinforcement learning according to claim 2, wherein the inference task scheduling model specifically comprises: a terminal layer, an edge calculation layer and a cloud layer;
the terminal layer comprises a plurality of types of equipment, the equipment periodically collects environment data and generates a computation intensive task and a delay sensitive task, and the terminal equipment is accessed to the edge computing layer through TCP;
the edge computing layer is used for providing edge computing resources for the terminal equipment and is connected with the cloud layer through a 5G network;
the cloud layer is used for providing cloud computing capability for the terminal equipment.
6. Substation inspection management device based on deep reinforcement study, characterized by comprising:
the model construction module is used for constructing an inference task scheduling model which takes real-time computing power of the edge node and the cloud as input and takes a specific edge node equipment number or the cloud as a prediction result for output by using a Muzero algorithm;
the model training module is used for training the reasoning task scheduling model by using data information generated by various patrol tasks in the transformer substation with load balancing as a target, wherein the load balancing is used for distributing resources based on real-time calculation forces of the edge nodes and the cloud so as to minimize the average service time of the patrol tasks;
the resource allocation module is used for carrying out node prediction by utilizing the trained reasoning task scheduling model based on the current state of the transformer substation, and allocating edge computing resources for the patrol task according to a prediction result so as to realize load balance of patrol management of the transformer substation.
7. The substation patrol management device based on deep reinforcement learning according to claim 6, wherein in the model training module, the inference task scheduling model is deployed at a cloud end, the cloud end performs centralized training on the inference task scheduling model, and the training is distributed to each edge node after the training is completed.
8. The substation patrol management device based on deep reinforcement learning according to claim 7, wherein in the model training module, with load balancing as a target, the inference task scheduling model is trained by using data information generated by various patrol tasks in the substation, and specifically includes:
dividing a substation inspection task into a computation-intensive task and a delay-sensitive task;
for the delay sensitive task, the edge node is distributed for processing, and for the computation intensive task, the computation intensive task is sent to the cloud for processing;
training the reasoning task scheduling model by utilizing data information generated by various patrol tasks in the transformer substation;
and calculating task failure rates in all the inspection tasks, and optimizing the reasoning task scheduling model by taking the lowest task failure rate as a target.
9. The deep reinforcement learning based substation patrol management device of claim 8, wherein in the model training module, the data information comprises real-time computing power, network bandwidth and delay, task size, and delay sensitivity of the edge nodes and the cloud.
10. The substation patrol management device based on deep reinforcement learning according to claim 7, wherein in the model building module, the inference task scheduling model specifically includes: a terminal layer, an edge calculation layer and a cloud layer;
the terminal layer comprises a plurality of types of equipment, the equipment periodically collects environment data and generates a computation intensive task and a delay sensitive task, and the terminal equipment is accessed to the edge computing layer through TCP;
the edge computing layer is used for providing edge computing resources for the terminal equipment and is connected with the cloud layer through a 5G network;
the cloud layer is used for providing cloud computing capability for the terminal equipment.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211502140.XA CN116109058A (en) | 2022-11-28 | 2022-11-28 | Substation inspection management method and device based on deep reinforcement learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211502140.XA CN116109058A (en) | 2022-11-28 | 2022-11-28 | Substation inspection management method and device based on deep reinforcement learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116109058A true CN116109058A (en) | 2023-05-12 |
Family
ID=86262843
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211502140.XA Pending CN116109058A (en) | 2022-11-28 | 2022-11-28 | Substation inspection management method and device based on deep reinforcement learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116109058A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117579625A (en) * | 2024-01-17 | 2024-02-20 | 中国矿业大学 | Inspection task pre-distribution method for double prevention mechanism |
-
2022
- 2022-11-28 CN CN202211502140.XA patent/CN116109058A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117579625A (en) * | 2024-01-17 | 2024-02-20 | 中国矿业大学 | Inspection task pre-distribution method for double prevention mechanism |
| CN117579625B (en) * | 2024-01-17 | 2024-04-09 | 中国矿业大学 | Inspection task pre-distribution method for double prevention mechanism |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111629380B (en) | Dynamic resource allocation method for high concurrency multi-service industrial 5G network | |
| CN111459628A (en) | Spark platform task scheduling method based on improved quantum ant colony algorithm | |
| CN113282368A (en) | Edge computing resource scheduling method for substation inspection | |
| CN115175217A (en) | Resource allocation and task unloading optimization method based on multiple intelligent agents | |
| EP4024212B1 (en) | Method for scheduling inference workloads on edge network resources | |
| CN114546608A (en) | Task scheduling method based on edge calculation | |
| CN116109058A (en) | Substation inspection management method and device based on deep reinforcement learning | |
| Tekiyehband et al. | An efficient dynamic service provisioning mechanism in fog computing environment: A learning automata approach | |
| Shimonishi et al. | Energy optimization of distributed video processing system using genetic algorithm with bayesian attractor model | |
| CN113676357A (en) | Decision method for edge data processing in power internet of things and application thereof | |
| Aiswarya et al. | Latency Reduction in Medical IoT Using Fuzzy Systems by Enabling Optimized Fog Computing | |
| CN116760722A (en) | Storage auxiliary MEC task unloading system and resource scheduling method | |
| CN116684472A (en) | Service deployment system and service deployment method for terminal-side computing network | |
| CN116436980A (en) | Real-time video task end network edge cooperative scheduling method and device | |
| CN113157344B (en) | DRL-based energy consumption perception task unloading method in mobile edge computing environment | |
| Li et al. | H-BILSTM: a novel bidirectional long short term memory network based intelligent early warning scheme in mobile edge computing (MEC) | |
| Du et al. | OctopusKing: A TCT-aware task scheduling on spark platform | |
| CN111784029A (en) | Fog node resource allocation method | |
| Sengupta et al. | Collaborative learning-based schema for predicting resource usage and performance in F2C paradigm | |
| CN115866678B (en) | Mobile edge cache resource optimization method based on network energy consumption hot spot detection | |
| Damaševičius et al. | Short time prediction of cloud server round-trip time using a hybrid neuro-fuzzy network | |
| Karimi et al. | Intelligent and Decentralized Resource Allocation in Vehicular Edge Computing Networks | |
| CN113485718B (en) | Context-aware AIoT application program deployment method in edge cloud cooperative system | |
| CN117997906B (en) | Node computing resource allocation method, network switching subsystem and intelligent computing platform | |
| Wang et al. | Efficient resources allocation and computation offloading model for ap-based edge cloud |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |