CN118113459A

CN118113459A - Multi-party cooperative control meteorological big data processing efficiency method and system

Info

Publication number: CN118113459A
Application number: CN202410077151.0A
Authority: CN
Inventors: 孙收余; 田钺; 王益彰; 李文科; 姚舜; 汪浩; 张克贤; 朱昌会; 缪新萍; 董若烟
Original assignee: Guizhou Power Grid Co Ltd
Current assignee: Guizhou Power Grid Co Ltd
Priority date: 2024-01-18
Filing date: 2024-01-18
Publication date: 2024-05-31

Abstract

The invention discloses a method and a system for processing meteorological big data by multiparty cooperative control, wherein the method comprises the following steps: acquiring data of complex nodes of a weather system in a current area through weather network business bottom data; processing and analyzing by a mobile edge calculation force deconstructing method to obtain an intelligent architecture with a deconstructable service model; the method comprises the steps of designing multiple factors of a weather network service, carrying out multi-factor optimal planning in the weather network service through an anti-resistance agent optimization strategy of reinforcement learning, and carrying out multi-factor joint calculation to adaptively and dynamically generate scheduling of the weather service to obtain optimal path scheduling to a computing node. The invention can furthest reduce the risks of data sharing and privacy disclosure, improve the performance and generalization capability of the model, improve the business resource efficiency of the weather network, and is beneficial to improving the overall performance of the weather network business and accelerating the business processing speed, thereby optimizing the resource utilization rate.

Description

Multi-party cooperative control meteorological big data processing efficiency method and system

Technical Field

The invention relates to the technical field of intelligent weather, in particular to a weather big data processing efficiency method and system for multiparty cooperative control.

Background

The rapid development of digital technology has brought many innovations to the meteorological field, such as the application of technologies such as the internet of things, big data analysis, cloud computing and artificial intelligence, so that the meteorological system can better collect, process and analyze meteorological data, thereby providing more intelligent and efficient meteorological management and transportation services. The development and popularization of the intelligent weather system provide a foundation for weather network business, and the ITS integrates weather management, transportation and user requirements by using modern communication, information and control technologies so as to realize a more efficient, safe and sustainable weather system.

However, for more complex meteorological scenarios, some existing intelligent meteorological systems can only perform simple resource allocation and single-mode data monitoring. On the one hand, weather network business involves a large amount of data exchange and information sharing, and ensuring the security and privacy of these data is an important challenge. If the security measures of the weather network service are not in place, the problems of personal privacy disclosure, network attack or data tampering and the like can be caused. Furthermore, the weather network business needs to interoperate with a variety of different weather systems and devices. However, the technical standards and interfaces between the different systems may not be compatible, which may lead to difficulties in data integration and communication. Technical interoperability is a key issue for achieving seamless connectivity and interoperability of weather network services. Common systems of intelligent weather at present, such as weather flow detection, weather anomaly detection, extreme weather forecast, long-term weather forecast business and the like, utilize part of characteristics of current weather nodes to operate a single target task, upload weather node data to a cloud for large-scale data mining and intelligent identification on the premise of wasting a large number of effective characteristics, and upload cloud remote servers consume a large number of resources and simultaneously bring instant time delay problems, and a central server is provided with a large-scale data analysis center, but needs to transmit rapid network supporting facilities and physical conditions depending on the data nodes.

Disclosure of Invention

This section is intended to outline some aspects of embodiments of the application and to briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section as well as in the description of the application and in the title of the application, which may not be used to limit the scope of the application.

The present invention has been made in view of the above-described problems occurring in the prior art. Therefore, the invention provides a multi-party cooperative control meteorological big data processing efficiency method, which solves the problems of how to improve the overall performance of meteorological network business, accelerate business processing speed and optimize resource utilization.

In order to solve the technical problems, the invention provides the following technical scheme:

in a first aspect, the present invention provides a method for processing efficiency of weather big data under cooperative control of multiple parties, including:

Acquiring data of complex nodes of a weather system in a current area through weather network business bottom data;

Based on the node data, processing and analyzing by a mobile edge calculation power deconstructing method to obtain an intelligent architecture with a deconstructable service model;

Based on the business model, designing multiple factors of the weather network business, performing multi-factor optimal planning in the weather network business through an opposite intelligent agent optimization strategy of reinforcement learning, and performing multi-factor joint calculation to adaptively and dynamically generate scheduling of the weather business to obtain optimal path scheduling to an algorithm node.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: the processing analysis of the node data comprises the following steps:

Analyzing real-time meteorological node data acquired in a meteorological network service into an intelligent architecture which can be deconstructed by a current service model by a mobile edge computing force deconstructing method, and adding structure federal learning and intention driving;

the federal learning calculates a new round of weight update by multiplying a loss function trained by a local client model by a fixed learning rate eta, and the model weight update of the local client is expressed as:

w_t,k＝w_t-1,k-ηΔF_k(w)

Where F _k (w) represents the loss function of the model on the kth participant in federal learning, w _t,k represents the local model weight of the kth participant at iteration t round, w _t-1,k represents the local model weight of the kth participant at the last round of iteration t-1, ΔF _k (w) represents the gradient of the loss function of the local model of the kth participant with respect to model weight w;

Based on vertical federal learning, a Cloud-VFL model is designed for moving edge computing power deconstructing, the ML model in the Cloud-VFL model being divided into three parts, comprising: local model of multiple participants Service terminal aggregation model M ^w and specific meteorological task model M ^t, whereFor the platformAnd (5) maintaining an ith local model.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: the moving edge computing force deconstructing by the Cloud-VFL model includes:

Grouping different feature subsets of the same specific meteorological task dispersed across a plurality of data source nodes into two groups, comprising: the characteristics insensitive to fairness and the characteristics sensitive to fairness are input as a Cloud-VFL model;

there are m fairness sensitive features related to weather types, the ith has type The platform of (2) is expressed asAll fairness and insensitivity features are concentrated on n platforms, denoted i-th platform asWherein i represents the ith participant in federal learning, and is provided with a label y for reserving a sample on a target task by a specific task platform P ^t, and a server is used for information aggregation;

Uniformly encoding the fairness sensitive features and the fairness insensitive features, removing the bias of certain feature sets to specific meteorological tasks, and encoding the local fairness insensitive features into local representations by a local model in a Cloud-VFL;

The specific meteorological task model M ^t is maintained by the specific platform P ^t, and uses the information in the unified representation S to predict, detect, classify, and identify anomalies for the target task.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: the encoding includes:

When the task platform P ^t needs to train a particular weather task, the task platform P ^t first distributes the ID of the particular weather task to nodes that are insensitive to fairness At the ith participant platformIn, local coding modelFor storage inThe specific meteorological task fairness insensitive characteristics in the model (1) are encoded, and a local representation/> isconstructed

Obtaining multiple local representationsDifferent fairness insensitive characteristics of the target task are encoded, the local representations are uploaded to a server P ^w, and information in all the local representations is aggregated into a unified representation through an aggregation model M ^w;

for each fairness sensitivity characteristic Mapping the unified representation S into a fairness-sensitive feature encoder a _i;

Given coded representations in the same training batch According to the coding representationAnd a _i, expressed as:

wherein, S ^j represents unified representation of fair sensitive feature codes and fair insensitive feature codes in a j-th specific meteorological task respectively, r _j is a correlation score, and E is a batch size;

Based on the top E _i tasks of the relevance rank, a unified coding representation is randomly selected Wherein E _i is a super-parameter,Training contrast discriminator/>, as a negative sample in contrast learningThe input to the map encoder a _i is classified, expressed as:

wherein, Representing a negative log likelihood loss function,Representing an MLP feedforward network, wherein omega represents a training set, and x represents data samples in the training set omega;

minimizing by iteration Training contrast discriminatorObtain optimal contrast discriminator

Using an optimal contrast discriminatorSTraining contrast against learning lossGradient of decline of contrast challenge learningExpressed as:

Contrast learning gradient of feature code a _i Further back to code mapper a _i, denoted:

Wherein a _i (·) represents a mapping encoder that removes the bias of the ith participant's data characteristics;

Uploading the optimized feature code a _i to a fairness sensitive feature platform OnUsing error discriminatorPredicting fairness sensitivity feature/> from a _i TagAnd calculates error discrimination lossExpressed as:

by iterating out an optimum error discriminator Fairness sensitivity feature generated for the ith participant platformCalculating code mapper lossExpressed as:

Obtaining a loss function of the contrast model Expressed as:

wherein, Respectively represent performance, fairness and privacy training targets of the contrast model,Indicating the task objective loss, lambda _i and gamma _i indicate the set hyper-parameters.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: the obtaining the optimal path schedule includes:

Designing multiple factors of meteorological network service, including data flow factor, time efficiency factor, resource utilization factor, reliability factor and energy efficiency factor;

The obtaining of the optimal path scheduling comprises the following steps: and acquiring meteorological and early warning events identified by the business model, constructing different command processing flows for various converged meteorological type events through multi-factor joint calculation, and adaptively butting all subsystems of an upper layer according to a treatment instruction issued by command processing scheduling and combining optimal path scheduling.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: reinforcement learning resistant agent optimization strategies include: modeling other intelligent agents through a trusted opponent model, learning the intelligent agents through opponent model opposition with the other intelligent agents, and performing multi-factor optimal planning in meteorological network service, wherein the method comprises the following steps:

Initializing a meteorological environment, modeling multi-factor joint calculation as a multi-agent reinforcement learning environment, and determining a state space S ^E, an action space U and a reward function R of an agent;

The state space S ^E of the intelligent agent comprises service model node deployment positions, real-time states, load information, service demands and data flow conditions, and the action space U of the intelligent agent represents a scheduling strategy of meteorological service or selection of an optimal service model path;

constructing meteorological agents, designing a plurality of agents to simulate and optimize the operation of a meteorological system, wherein the agents comprise pollutant agents, wind direction agents, precipitation agents and scheduling agents, each agent adopts an independent RNN network, each agent has an action observation network to record historical actions τ ^α, and the local action network is a decision pi ^α(u^α|τ^α made based on the historical actions;

And modeling by the trusted opponent, observing the behaviors of other agents through the trusted opponent model to estimate, providing an opponent model for each meteorological agent, and alternately updating the current strategy of decision-making agent and opponent agent in the training process.

As a preferable scheme of the multi-party cooperative control meteorological big data processing efficiency method, the invention comprises the following steps: mapping the current service model node state and the current selected path optimization strategy to the expected strategy of each agent through a value decomposition network of each agent, wherein the input of the value decomposition network is the current service model node state and the real-time strategy of all agents, the output is the expected strategy of each agent, and the D ^a is used for representing independent return after a single agent takes action and is expressed as follows:

D^a＝R(s,u)-R(s,(u^-a,c_a))

Wherein c _a represents a default behavior action, R (s, u) represents a return function, s represents a return obtained after taking the action u in the state s, u represents a current state, u represents a behavior selected by the agent, u ^-a represents a complementary behavior of the behavior u, and another behavior different from the behavior a;

adding an average effect value Q as a default behavior, and confirming the optimal default behavior, wherein the default behavior is expressed as follows:

Wherein u' _a represents a supplemental behavior, representing an alternative behavior selectable by the agent under a particular historical action τ ^a;

the independent rewards D ^a are equivalently approximated as scheduling policy scores R (s, u), expressed as:

Inputting the current global state, the observation behaviors of the current intelligent agent, the joint action space of other intelligent agents except the self intelligent agent, the one-hot codes of the self current intelligent agent and the behaviors of all intelligent agents at the same time into a central evaluation network Critic, outputting all executable Q values of the current intelligent agent by the central evaluation network Critic, and carrying out SoftMax normalization operation on the Q values to obtain a required average effect value Q (s, c _a);

The meteorological agents make decisions according to respective strategies and value functions, and simultaneously fight against trusted opponent models of other agents, and each agent optimizes respective strategy networks and value function networks by maximizing scheduling strategy evaluation scores.

In a second aspect, the present invention provides a weather big data processing system controlled by multiple parties, comprising:

the data acquisition module is used for acquiring data of complex nodes of the weather system in the current area through weather network business bottom data;

The processing module is used for processing and analyzing by a mobile edge calculation power deconstructing method based on the node data to obtain an intelligent architecture with a deconstructable service model;

The dispatching output module is used for designing multiple factors of the weather network business based on the business model, carrying out multi-factor optimal planning in the weather network business through the reinforcement learning countermeasure agent optimization strategy, and carrying out multi-factor joint calculation to self-adaptively and dynamically generate dispatching of the weather business so as to obtain optimal path dispatching to the computing nodes.

In a third aspect, the present invention provides an electronic device, comprising:

a memory and a processor;

The memory is configured to store computer-executable instructions that, when executed by the processor, implement the steps of the multi-party cooperatively controlled weather big data processing efficiency method.

In a fourth aspect, the present invention provides a computer readable storage medium storing computer executable instructions that when executed by a processor perform the steps of the multi-party cooperatively controlled weather big data processing efficiency method.

Compared with the prior art, the invention has the beneficial effects that: the invention designs the Cloud-VFL (Variational FEDERATED LEARNING) model to perform moving edge calculation power deconstruction, adopts a contrast and countermeasure learning method, is favorable for carrying out safety protection on different meteorological data and inhibiting the situation that the model has prejudice on a part of characteristic data platform, and can furthest reduce the risks of data sharing and privacy leakage; because each data source only needs to share characteristic data, but not all data, the sensitive information can be better protected, the characteristic complementarity of the meteorological data sources is utilized to the greatest extent, the performance and generalization capability of the model are improved, the meteorological network service can be better adapted to the dynamic change of a meteorological system, and real-time meteorological requirements and optimization strategies can be responded quickly; the multi-factor joint calculation method based on reinforcement learning is used for adaptively and dynamically generating the weather service dispatch, so that optimal path dispatch to an algorithm node is realized, the resource efficiency of a weather network service is improved, meanwhile, through intelligent sensing and adaptive dispatch, the weather network service can better adapt to continuously changing weather environment and service requirements, and more reliable and efficient weather service is provided; the system can calculate by utilizing the position information of the computing power node, the network topological structure, the processing capacity of the node and other factors, and simultaneously, takes the real-time load condition of the node and the demand characteristics of the current meteorological service, such as processing time delay, bandwidth requirement and the like, into consideration, the system performs optimal strategy planning by integrating the factors to determine an optimal path scheduling strategy, and the meteorological network service can more efficiently utilize resources by realizing optimal path scheduling to the computing power node, thereby being beneficial to improving the overall performance of the meteorological network service and accelerating the service processing speed, so that the resource utilization rate is optimized.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art. Wherein:

FIG. 1 is a schematic overall flow chart of a method for controlling the processing efficiency of meteorological big data according to an embodiment of the invention;

FIG. 2 is a schematic diagram of an overall architecture of a multi-party cooperative control weather big data processing efficiency method according to an embodiment of the invention;

FIG. 3 is a schematic diagram of a weather federal learning Cloud-VFL framework for a multi-party co-controlled weather big data processing efficiency method in accordance with one embodiment of the present invention;

FIG. 4 is a schematic diagram of an antagonistic strategy optimization planning method according to an embodiment of the invention;

FIG. 5 is a schematic diagram of a business model intelligent commanding and dispatching platform logic architecture of a multi-party cooperative control meteorological big data processing efficiency method according to an embodiment of the present invention.

Detailed Description

So that the manner in which the above recited objects, features and advantages of the present invention can be understood in detail, a more particular description of the invention, briefly summarized above, may be had by reference to the embodiments, some of which are illustrated in the appended drawings. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments of the present invention without making any inventive effort, shall fall within the scope of the present invention.

In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present invention is not limited to the specific embodiments disclosed below.

Further, reference herein to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic can be included in at least one implementation of the invention. The appearances of the phrase "in one embodiment" in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments.

While the embodiments of the present invention have been illustrated and described in detail in the drawings, the cross-sectional view of the device structure is not to scale in the general sense for ease of illustration, and the drawings are merely exemplary and should not be construed as limiting the scope of the invention. In addition, the three-dimensional dimensions of length, width and depth should be included in actual fabrication.

Also in the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "upper, lower, inner and outer", etc. are based on the orientation or positional relationship shown in the drawings, are merely for convenience of describing the present invention and simplifying the description, and do not indicate or imply that the apparatus or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as limiting the present invention. Furthermore, the terms "first, second, or third" are used for descriptive purposes only and are not to be construed as indicating or implying relative importance.

The terms "mounted, connected, and coupled" should be construed broadly in this disclosure unless otherwise specifically indicated and defined, such as: can be fixed connection, detachable connection or integral connection; it may also be a mechanical connection, an electrical connection, or a direct connection, or may be indirectly connected through an intermediate medium, or may be a communication between two elements. The specific meaning of the above terms in the present invention will be understood in specific cases by those of ordinary skill in the art.

Example 1

Referring to fig. 1 to 5, for one embodiment of the present invention, a method for processing meteorological big data with coordinated control of multiple parties is provided, including:

S1, acquiring data of complex nodes of a weather system in a current area through weather network business bottom data.

Fig. 1 shows a general frame diagram of the present invention, preferably, through the sensing of data at the bottom layer of the weather network business, the relevant information of the complex node of the weather system in the current area, such as the fusion data of the temperature, humidity, wind pressure, wind speed, area flow, section flow, classified flow, etc. of the node can be obtained.

S2, processing and analyzing by a mobile edge calculation power deconstructing method based on the node data to obtain an intelligent architecture with a deconstructable service model.

Specifically, the processing analysis of the node data includes:

the federal learning enables the edge device to cooperatively train out an optimal global machine learning model under the condition that source data is not transmitted to the cloud device through cooperation and shared protocol specifications of all parties, and a federal learning algorithm is expressed as:

Where F _k (w) represents the loss function of the model on the kth participant in federal learning, n _k represents the number of samples of participant k, dataset d _k represents the local dataset of the kth participant, parameter w represents the local model weight for the particular meteorological task that includes the kth participant, and F _i (w) represents the loss function generated by the model with parameter w for the instance in dataset d _k;

w_t,k＝w_t-1,k-ηΔF_k(w)

Wherein w _t,k represents the local model weight of the kth participant at iteration t round, w _t-1,k represents the local model weight of the kth participant at the last iteration t-1, Δf _k (w) represents the gradient of the loss function of the local model of the kth participant with respect to the model weight w;

Based on vertical federal learning, a Cloud-VFL model is designed for moving edge computing power deconstructing, and an ML model in the Cloud-VFL model is divided into three parts, including: local model of multiple participants Service terminal aggregation model M ^w and specific meteorological task model M ^t, whereFor the platformAnd maintaining an ith local model, namely, a local model of each platform, wherein the ith participant corresponds to the ith platform, the ith fairness sensitive feature, the ith local model and the ith local feature representation.

It should be noted that, federal learning is used as an operating system of edge computing, and a protocol specification for cooperation and sharing of all parties is provided, which enables edge devices to cooperatively train an optimal global machine learning model under the condition that source data is not sent to cloud devices, and in different information departments of an area, such as emergency weather, intelligent command and the like, a large amount of heterogeneous data can be generated to form a plurality of data islands, so that the heterogeneous data processing capability of federal learning can help a decision maker to create an intelligent weather platform which rapidly responds to the requirements of citizens, and the problem of the data islands is solved.

Further, the moving edge computing force deconstructing by the Cloud-VFL model includes:

As shown in FIG. 3, in a weather network business architecture, different feature subsets of the same specific weather task are distributed across multiple data source nodes, for a specific weather-aware task, the corresponding weather task training model would be more prone to employ its preferred feature set and thus would lose some performance and generalization capability, from a fairness perspective, separating the different feature subsets of the same specific weather task distributed across multiple data source nodes into two groups, including: the characteristics insensitive to fairness and the characteristics sensitive to fairness are input as a Cloud-VFL model, and the characteristics sensitive to fairness are predicted to be irrelevant to the prediction of a specific meteorological task training model;

Without loss of generality, there are m fairness sensitive features related to weather types, the ith is of type The platform of (2) is expressed asAll fairness and insensitivity features are concentrated on n platforms, denoted the ith platform asWherein i represents the ith participant in federal learning, preferably, the participant is a device, a client or a node, a specific task platform P ^t is provided with a label y for reserving a sample on a target task, and a trustworthy server is provided for information aggregation;

Uniformly encoding the fairness sensitive features and the fairness insensitive features, removing bias (bias) of certain feature sets to specific meteorological tasks, and encoding the local fairness insensitive features into local representations by a local model in a Cloud-VFL;

The specific meteorological task model M ^t is maintained by the specific platform P ^t, and uses the information in the unified representation S to perform tasks such as prediction, detection, classification, and anomaly identification on the target task, and is expressed as:

wherein, Is the output of the particular meteorological task model M ^t.

It should be noted that, in the Cloud-VFL, the local model is used to encode the local fairness insensitive feature into the local representation, and for removing the bias of some feature sets to specific meteorological tasks, it is preferable to implement the local encoding by transformers model for the traffic statistics of the encoded meteorological node area, and implement the local representation by PLMs model for the real-time position of the vehicle and the traffic number of the meteorological node.

Still further, the encoding includes:

When the task platform P ^t needs to train a particular weather task, the task platform P ^t first distributes the ID of the particular weather task to nodes that are insensitive to fairness At the ith participant platformIn, local coding model(Local model of each participant) pair is stored atThe specific meteorological task fairness insensitive characteristics in the model (1) are encoded, and a local representation/> isconstructed

Obtaining multiple local representationsDifferent fairness insensitive characteristics of the target task are encoded, the local representation is uploaded to a server P ^w for task processing, an aggregation model M ^w is maintained by the server P ^w, and information in all the local representations is aggregated into a unified representation through an aggregation model M ^w;

It should be noted that, in particular, different feature data of the same weather-aware task usually have inherent relevance, mining can enhance the performance and generalization of the target task model, so in this embodiment, a multi-headed self-care network is first applied to capture the relevance between local representations, where The context of (1) is expressed asThen, applying the attention network to the context representation to model the relative importance of the representation raw data and build a unified representation S; because the unified representation S can encode various information in the scattered fair insensitive feature domain, the unified representation S is further uploaded to the task platform P ^t instead of the original data to provide information for the target task; the data from different meteorological systems are subjected to fusion analysis through the aggregation model, and meanwhile, the independence and privacy problems of related meteorological systems are guaranteed.

Although the fairness sensitivity feature is not an input to the specific meteorological task model M ^t, M ^t may still protect user privacy by mining the deviation of the local feature unified representation S from the data plane, applying the fairness learning to suppress the unified representation S from encoding the fairness sensitivity feature, and modifying the unified representation S.

For each fairness sensitivity characteristicThe unified representation S is mapped into the fairness-sensitive feature encoder a _i, denoted as:

a_i＝a_i(S)

wherein a _i represents an MLP-based mapping encoder with the ith participant fairness sensitivity characteristic removed, and a _i only retains the unified representation S at the fairness sensitivity characteristic, wherein the unified representation S is the input of the mapping encoder a _i Information on the user and eliminating privacy of other users;

Given coded representations in the same training batch First of all expressed according to these codesAnd a _i, expressed as:

Based on the top E _i tasks of the relevance rank, a unified coding representation is randomly selected The representationVery likely to be equal to fairness sensitivity featureS after encoding is the same, wherein E _i is a superparameter,Training contrast discriminator/>, as a negative sample in contrast learningThe input to the map encoder a _i is classified, expressed as:

wherein, Representing a negative log-likelihood loss function for training a contrast discriminatorTo classify the input a _i, its goal is to compare the discriminators/>, by trainingTo make the score of the unified representation S output by the aggregation model M ^w higher than the negative sampleTo classify the input a _i, which aids in learning the feature representation of task a _i,Representing an MLP feed-forward network, Ω representing a training set, x representing data samples in the training set Ω, exp () function representing an exponential function based on a natural constant e;

minimizing by iteration Training contrast discriminatorObtain optimal contrast discriminatorExpressed as:

Using an optimal contrast discriminator STraining contrast against learning lossGradient of decline of contrast challenge learningExpressed as:

Wherein A _i (·) represents a mapping encoder that removes feature bias of the ith participant data, A _i (S) aims to learn how to convert input data S into a unified representation that removes fairness sensitive feature bias for classification or other tasks in contrast learning, the function is designed and trained to protect privacy information of different aerial platforms (by contrast gradient descent) and reduce fairness sensitive feature information in the representation, thereby improving privacy and fairness of the model, the unified representation S and fairness sensitive feature encoding negative samples generated by the aggregation model Is not adjusted with the decline of contrast gradient, in this way, contrast gradient forced code mapper a _i protects privacy information of different meteorological platforms in a _i and reduces fair sensitive characteristic information in coding;

Obtaining a loss function of the contrast model Expressed as:

It should be noted that, the Cloud-VFL (Variational FEDERATED LEARNING) model is designed to perform the moving edge computing power deconstructing, and the frame diagram of the Cloud-VFL model is shown in fig. 2, the whole model is divided into two large blocks of local participants and service terminals to perform learning training, and local primary feature extraction models are trained by local different participants using local data sets, where the network used by the local platform is insensitive to feature fairness. All the participants generate local characteristic representations from the original data and send the local characteristic representations to the service terminal for performing global weather task iterative learning. The service terminal module mainly comprises an aggregation model, a fairness sensitive characteristic identification platform and a specific meteorological task model M ^t, wherein local characteristics obtained by local model training are used for different participants, firstly, unified characteristic representation is generated by adopting the aggregation model, and secondly, prejudice information in the unified representation is restrained by the fairness sensitive characteristic identification platform. The specific meteorological task model M ^t is trained by using the unified characteristic representation after the prejudice is removed, and the framework adopts a contrast countermeasure learning method, so that the method is favorable for carrying out safety protection on different meteorological data and inhibiting the condition that the model has prejudice on a part of characteristic data platform. The framework of the design of the invention is based on vertical federal learning, which is a variant of federal learning, and aims to solve the problem that different characteristics exist between data owners. In traditional federal learning, participants typically have the same feature space between them, but in vertical federal learning, the participants possess different but complementary features. Vertical federal learning is typically used to handle scenarios involving multiple data sources, each with different characteristics. The mobile edge computing power deployment has a plurality of edge nodes, the nodes locally process data perception tasks of the meteorological network service, and the data sources have different characteristics, but can cooperate through vertical federal learning to jointly train a meteorological task model, so that a more comprehensive and accurate analysis result is obtained; the main advantage of vertical federal learning is that the risk of data sharing and privacy disclosure can be reduced to the maximum extent; because each data source only needs to share the characteristic data, but not all the data, the sensitive information can be better protected, the characteristic complementarity of the meteorological data source is utilized to the greatest extent, and the performance and generalization capability of the model are improved.

In the bottom layer sensing of the meteorological network business, the fusion data of the temperature, humidity, wind pressure, wind speed, regional flow, section flow, classified flow and the like of the meteorological nodes are analyzed and predicted, and the processing analysis is carried out by the mobile edge calculation power deconstructing method provided by the invention, so that on one hand, diversified, large-granularity and complex calculation power tasks are decomposed into small-granularity and simplified calculation power tasks aiming at different meteorological network business tasks and requirements; on the other hand, the fusion supply and the optimal matching of multi-element capability are realized according to the actual service requirement, and more abundant service model services are provided while resources are efficiently utilized; aiming at a meteorological network service platform, the mobile edge computing power deconstructing method analyzes real-time meteorological node data acquired in a service model into an intelligent architecture which can be deconstructed by a current service model, introduces structure federal learning, gradually builds fusion deconstructing capability of other elements, capabilities and applications except computing power and a network, realizes fusion arrangement of combined services, and introduces leading edge technologies such as intention driving and the like to improve the intelligent capability.

By combining the Cloud-VFL model with the meteorological heterogeneous data, local fairness insensitive characteristics related to the current task in all edge nodes of the meteorological network service can be obtained in real time, complex meteorological service processing analysis is performed, and the fusion deconstructing capacity and the fusion arrangement capacity of the meteorological network service are improved.

S3, designing multiple factors of the weather network business based on the business model, performing multi-factor optimal planning in the weather network business through an opposite intelligent agent optimization strategy of reinforcement learning, and performing self-adaptive dynamic generation of scheduling of the weather business through multi-factor joint calculation to obtain optimal path scheduling to an algorithm node.

Based on the mobile edge calculation power deconstructing method in the step S2, intelligent perception of deployment position, real-time state, load information and service demand of the weather network service is realized, furthermore, a weather service scheduling strategy is adaptively and dynamically generated through multi-factor joint calculation of a service model, namely optimal path scheduling to calculation power nodes is realized, and the weather network service resource efficiency is improved.

Further, designing multiple factors including a data flow factor, a time efficiency factor, a resource utilization factor, a reliability factor and an energy efficiency factor;

Preferably, the data flow factor considers the flow condition of real-time meteorological data, including the channel spectrum degree and the port multiplexing density; the time efficiency factor considers the time sensitivity of the weather service, and different weather services have different requirements on time; the resource utilization factor considers the utilization rate and availability of meteorological network service resources; reliability factors, which consider the reliability requirements of meteorological service; the energy efficiency factor considers the energy consumption of the weather network business.

Reinforcement learning resistant agent optimization strategies include: modeling other intelligent agents through the trusted opponent model, learning the intelligent agents through opponent model opposition with the other intelligent agents, and performing multi-factor optimal planning in meteorological network business;

specifically, as shown in FIG. 3, in the local behavior network Actor section, where Representing the local information or state observed by agent a at time step t, determined by the local observation function of agent a,Representing the action taken by agent a at the last moment,AndRepresenting observed information and states of the single agent at historical time instants; atInIs the policy function value that agent a takes at time step t, pi (x) represents the policy evaluation function that agent takes, epsilon is typically the value sampled from a random distribution, which introduces randomness such that the policy function value is not in the same stateThe exact same action is chosen, i.e. it makes the policy somewhat exploratory.

The Critic part of the network is evaluated at the center,Representing the complementary actions of the behaviour u before the time step t' (generally before the current time step t), modified on the basis of the original behaviour u, S _t representing the environmental conditions observed by the agent at the current time step t, at Wherein Q represents an action value function that provides an expected return value for taking a particular action in a particular state,/>, for a particular actionRepresenting the expected return that agent a takes action "1" given the environmental conditions and other agents 'actions, the central evaluation network outputs the Q value for each of agent a's actions, using a SoftMax normalization function to reduce the order of action |u| ⁿ to |u|.

In the overall framework, S ^E refers to a set of meteorological environmental conditions, S _t represents the environmental conditions observed by the agent at the current time step t, R _t represents the return value obtained by the agent after taking action u at state S at the current time step t,AndRepresenting the state value observed locally by each agent at the current environmental state S _t, the state information observed by different agents is different, and U represents all the motion sample spaces: at the same time, each agent takes an actionAnd forms a joint action space U epsilon U, and a P finger state transfer function: calculating the current state P (s' |s, u) at a certain moment according to the current state s and the joint action space u; s' represents the state at the next moment, typically used in a Markov decision process (Markov Decision Process, MDP) or state transfer function, representing the possible states at the next moment given the current state s and a joint action space u, s represents the state at the current moment, and in the state space of the system or environment, representing the state of the environment observed by the agent or decision agent at a certain moment, an action/>, can be selected by the agent based on the current state sTo influence the state s' at the next moment, R represents the global scheduling policy score: r (s, u), Z represents the local observation set of the single agent at each moment, O is the local observation function, and is expressed as Z=O (s, a),AndThen the local scheduling policy score for each agent is represented at the current time step t.

It should be noted that, by modeling the behaviors of other agents through the trusted opponent model, each agent can evaluate the influence of the policies of other agents on itself more accurately, and by adopting the idea of antagonism optimization, the agents learn by antagonizing the opponent models of other agents, and at the same time, introduce the concept of collaborative learning, so that each agent can learn from the experiences of other agents and understand the global situation better.

The obtaining of the optimal path scheduling comprises the following steps: acquiring meteorological and early warning events identified by a business model, constructing different command processing flows for various converged meteorological type events through multi-factor joint calculation, and adaptively butting all subsystems of an upper layer according to a treatment instruction issued by command processing scheduling and combining optimal path scheduling;

Specifically, fig. 4 shows a logic architecture diagram of an intelligent commanding and dispatching platform of a business model, firstly, the meteorological data and early warning events identified by all subsystems of the meteorological data in a converged area, such as monitoring, hidden danger risk early warning and identification, event rule mining and the like; then, introducing multi-factor combined calculation, and constructing different command processing flows for various converged meteorological type events; meanwhile, according to the treatment instruction issued by the command processing schedule, the upper subsystem is adaptively docked by combining with an optimal path scheduling algorithm, such as fusion monitoring, situation awareness prediction, statistical report forms, service management, emergency linkage and the like. Finally, based on the service model digital twin technology, intelligent command and dispatch of meteorological network service is realized. Through the platform, road weather can fully utilize the existing hardware equipment resources and software data to form high-efficiency and available structured and intelligent multi-source data. Through simulation prediction of meteorological data, weather congestion is prejudged in advance, and road service level and traffic capacity are effectively improved. Meanwhile, the prediction data of future traffic flow, road conditions and the like generated by the simulation subsystem are utilized to truly predict the future, so that the situation is prevented. The platform is divided into a perception layer, a data source layer, a supporting layer, an engine layer and an application layer. Each level is further divided into different application modules according to different functions, as shown in fig. 4. The platform is based on J2EE and WPF technical frames, adopts an SOA service-oriented software architecture, and guarantees scientificity and advancement of application software from the aspect of bottom architecture design. The platform has good expansibility, wide adaptability, strong compatibility and reliable stability, the system is guaranteed to have the capacity of mass access and large concurrency access, and the platform can meet the technical requirements of building integral informatization intelligent large integration.

It should be noted that, the business model intelligently directs and dispatches the meteorological events and the early warning event identified by each subsystem of the regional meteorological data of platform convergence, then, through multi-factor joint calculation, constructs different directing and processing flows for each type of converged meteorological event, meanwhile, according to the processing instruction issued by the directing and processing dispatch, combines the optimal path dispatching algorithm, adaptively interfaces each subsystem of the upper layer, finally, based on the business model digital twin technology, achieves the intelligent directing and dispatching of the meteorological network business.

Further, the reinforcement learning-based antagonistic agent optimization strategy solves the multi-factor optimal planning problem in the meteorological network business, comprising:

The state space S ^E of the intelligent agent comprises service model node deployment positions, real-time states, load information, service demands and data flow conditions, the action space U of the intelligent agent represents a scheduling strategy of meteorological service or selection of an optimal service model path, and the reward function R is used for evaluating the quality of the strategy calculated after the intelligent agent performs optimal path planning each time;

The method comprises the steps of constructing meteorological agents, designing a plurality of agents to simulate and optimize operation of a meteorological system, wherein the agents comprise pollutant agents, wind direction agents, precipitation agents and scheduling agents, and an independent RNN network is adopted for each agent, namely each agent uses a strategy network and a value function network. Each agent makes decisions according to its own policy and value function, each agent has an action observation network to record historical actions τ ^α, the local behavior network (actor) is a decision pi ^α(u^α|τ^α made based on historical actions, and because the decision network parameters are updated before including historical information, making decisions using the updated actor network can be regarded as a result of recording historical experience;

Preferably, the pollutant intelligent agent represents pollutant content at different nodes in the atmosphere, real-time meteorological information is collected through a sensor, and a decision is made according to the current meteorological conditions; the wind direction intelligent body represents the wind direction condition of the terminal position and monitors the condition of the atmospheric flow; the precipitation intelligent agent represents regional precipitation distribution conditions, and a travelling decision is made according to terminal node information, meteorological signals and surrounding environments; the scheduling agent is responsible for scheduling and coordination of the whole meteorological system, collects information from wind directions, precipitation and pollutant agents, comprehensively considers meteorological flow, channel capacity and business requirements, and formulates the optimal meteorological planning and scheduling strategy.

Trusted adversary modeling is used to model the behavior and strategies of other meteorological agents. In order to learn a good system scheduling strategy, each intelligent agent needs to consider the influence of the behaviors of other intelligent agents on the self scheduling strategy score, the behaviors of other intelligent agents are observed through a trusted adversary model to estimate, an adversary model is provided for each meteorological intelligent agent, and the current decision intelligent agent and the strategies of the adversary intelligent agent are updated alternately in the training process, specifically, when the adversary intelligent agent fixes the strategies, the current intelligent agent strategy is updated so as to maximize the rewards, and otherwise, when the decision intelligent agent fixes the strategies, the adversary intelligent agent strategy is updated so as to minimize the rewards of the current intelligent agent;

Mapping the current service model node state and the current selected path optimization strategy to the expected strategy of each agent through a value decomposition network of each agent, wherein the input of the value decomposition network is the current service model node state and the real-time strategy of all agents, the output is the expected strategy of each agent, and the D ^a is used for representing independent return after a single agent takes action and is expressed as follows:

D^a＝R(s,u)-R(s,(u^-a,c_a))

Wherein c _a represents a default behavior action, R (s, u) represents a return function, s represents a return obtained after taking a behavior u in a state s, s represents a current state, u represents a behavior selected by an agent, u ^-a represents a complementary behavior of the behavior u, and represents another behavior different from the behavior a, which is modified based on the original behavior u;

Where u' _a represents a supplemental behavior (ALTERNATIVE ACTION) representing alternative behavior of the agent selectable at a particular historical action τ ^a, typically for comparison with the primary behavior a;

It should be noted that the invention can make decisions and optimizations of multi-factor joint calculation in a multi-agent environment, and allows participants to make model training and updating locally by utilizing the advantages of distributed learning, and the participants can make autonomous decisions according to local requirements and specific conditions, and simultaneously realize overall optimization of the model by sharing model parameters, and the model cooperation can effectively utilize distributed calculation resources, and improve the accuracy and performance of the model; by the method for comprehensively considering the factors, a weather service scheduling strategy can be adaptively and dynamically generated, and weather tasks are scheduled to the optimal computing nodes, so that the efficiency of weather network service resources is improved.

The edge computing architecture and the optimal strategy planning method designed by the invention can fully sense the modal characteristics of the meteorological scene, realize faster real-time interaction between modules, adaptively and dynamically generate the meteorological service scheduling strategy, and improve the utilization rate of meteorological network service resources.

According to the vertical federal learning framework designed by the invention, for specific meteorological tasks, unified training is carried out by combining a plurality of edge node calculation force platforms, heterogeneous data of each platform is fully fused on the basis of ensuring data safety, the generation of data islands is reduced, intelligent perception of deployment positions, real-time states, load information and business requirements of meteorological network business is realized by a mobile edge calculation force deconstructing method, and after that, the invention designs influence factors such as data flow factors, time efficiency factors, reliability factors, energy dependency factors and the like suitable for meteorological scenes by combining the characteristics, and designs an optimal strategy planning model based on reinforcement learning according to the meteorological influence factors. By comparing different expected planning strategies, the self-adaptive dynamic generation of the weather service scheduling strategy can be realized, and weather tasks are scheduled to the optimal calculation nodes, so that the efficiency of weather network service resources is improved.

The foregoing is a schematic scheme of a method for processing efficiency of weather big data under coordinated control of multiple parties in this embodiment. It should be noted that, the technical solution of the weather big data processing system with multi-party cooperative control and the technical solution of the weather big data processing efficiency method with multi-party cooperative control belong to the same concept, and the details of the technical solution of the weather big data processing system with multi-party cooperative control in this embodiment are not described in detail, and all reference may be made to the description of the technical solution of the weather big data processing efficiency method with multi-party cooperative control.

In this embodiment, the weather big data processing system with cooperative control of multiple parties includes:

The dispatching output module is used for designing multiple factors of the weather network business based on the business model, carrying out multi-factor optimal planning in the weather network business through the reinforcement learning antagonism agent optimization strategy, and carrying out multi-factor joint calculation to self-adaptively and dynamically generate dispatching of the weather business so as to obtain optimal path dispatching to the calculation nodes.

The embodiment also provides an electronic device, which is suitable for the situation of weather big data processing under multi-party cooperative control, and comprises:

A memory and a processor; the memory is used for storing computer executable instructions, and the processor is used for executing the computer executable instructions to realize the weather big data processing efficiency method for realizing multiparty cooperative control as proposed by the embodiment.

The present embodiment also provides a storage medium having stored thereon a computer program which, when executed by a processor, implements a weather big data processing efficiency method for implementing multi-party cooperative control as proposed in the above embodiment.

The storage medium proposed in this embodiment belongs to the same inventive concept as the method for implementing the multiparty cooperative control of weather big data processing efficiency proposed in the above embodiment, and technical details not described in detail in this embodiment can be seen in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

From the above description of embodiments, it will be clear to a person skilled in the art that the present invention may be implemented by means of software and necessary general purpose hardware, but of course also by means of hardware, although in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art in the form of a software product, which may be stored in a computer readable storage medium, such as a floppy disk, a read only Memory (ReadOnly, memory, ROM), a random access Memory (Random Access Memory, RAM), a FLASH Memory (FLASH), a hard disk or an optical disk of a computer, etc., including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the method of the embodiments of the present invention.

Example 2

Referring to table 1, for one embodiment of the present invention, a method for processing meteorological big data with coordinated control of multiple parties is provided, and in order to verify the beneficial effects, a comparison result of two schemes is provided.

According to the application scene of the meteorological network, the simulation experiment is carried out by using the meteorological big data processing efficiency method of multiparty cooperative control, and compared with the traditional dividing method, the comparison verification is carried out on aspects of service processing performance and speed improvement, model performance and generalization capability, system overall performance and the like.

Table 1 comparative data for the verification of the process of the invention and the conventional process

	Conventional method	The method of the invention
			Processing speed(s)	0.17	0.036
Resource utilization (%)	70	92
			Business model accuracy (%)	83.1	97.3
System response time(s)	1.74	0.3
			Path result matching degree (%)	80	95

As can be seen from Table 1, the method of the invention has the advantages of obviously improved service processing performance and speed, improved processing speed by comparing the processing speed of meteorological service before and after the method is used, and successfully improved processing speed by analyzing the execution time of service processing tasks. The resource utilization rate is optimized, and the effective utilization degree of the system to the resources is improved; the performance and generalization capability of the model are improved, and the performance and generalization capability of the model are improved by comparing the performance indexes of the service model before and after use.

Meanwhile, the overall performance of the system is improved, the overall performance of the system is comprehensively evaluated, and the overall performance of the weather network service is improved through analysis of the overall performance of the system; and comparing the optimal planning effect, and obtaining optimal path scheduling through analysis of the planning process and the result.

It should be noted that the above embodiments are only for illustrating the technical solution of the present invention and not for limiting the same, and although the present invention has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that modifications and equivalents may be made thereto without departing from the spirit and scope of the technical solution of the present invention, which is intended to be covered by the scope of the claims of the present invention.

Claims

1. A weather big data processing efficiency method controlled by multiple parties in a cooperative way is characterized by comprising the following steps:

2. The multi-party cooperatively controlled weather big data processing efficiency method of claim 1, wherein the processing analysis of the node data comprises:

w_t,k＝w_t-1,k-ηΔF_k(w)

3. The multi-party cooperatively controlled weather big data processing efficiency method of claim 2, wherein the moving edge computing force deconstructing by the Cloud-VFL model comprises:

4. The multi-party cooperatively controlled weather big data processing efficiency method as set forth in claim 3, wherein said encoding includes:

Based on the top E _i tasks of the relevance rank, a unified coding representation is randomly selected Wherein E _i is a superparameter,Training contrast discriminator/>, as a negative sample in contrast learningThe input to the map encoder a _i is classified, expressed as:

a_i＝A_i(S)

Obtaining a loss function of the contrast model Expressed as:

5. The multi-party cooperative control weather big data processing efficiency method according to claim 1 or 2, wherein,

6. The multi-party cooperatively controlled weather big data processing efficiency method of claim 1 or 5, wherein the reinforcement learning antagonistic agent optimization strategy comprises: modeling other intelligent agents through a trusted opponent model, learning the intelligent agents through opponent model opposition with the other intelligent agents, and performing multi-factor optimal planning in meteorological network service, wherein the method comprises the following steps:

7. The multi-party cooperative control weather big data processing efficiency method of claim 6,

D^a＝R(s,u)-R(s,(u^-a,c_a))

the independent rewards D ^α are equivalently approximated as scheduling policy scores R (s, u), expressed as:

8. A multi-party cooperatively controlled weather big data processing system, comprising:

9. An electronic device, comprising:

a memory and a processor;

The memory is configured to store computer-executable instructions that, when executed by the processor, perform the steps of the multi-party cooperatively controlled weather big data processing efficiency method of any of claims 1 to 7.

10. A computer readable storage medium storing computer executable instructions which when executed by a processor perform the steps of the multi-party co-controlled weather big data processing efficiency method of any of claims 1 to 7.