CN117811907A - Satellite network micro-service deployment method and device based on multi-agent reinforcement learning - Google Patents

Abstract

The embodiment of the application provides a satellite network micro-service deployment method and device based on multi-agent reinforcement learning, wherein the method comprises the following steps: acquiring resource demand information of the micro service; determining resource utilization rate information and time delay information of the satellite node according to a pre-established resource utilization rate model and time delay model and configuration information of the satellite node; under the condition that the resource utilization rate information is smaller than a first preset value or the time delay information is smaller than a second preset value, a pre-trained multi-agent strategy deployment model is adopted to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the micro-service, the server terminal is configured according to the deployment strategy of the satellite nodes, the resource demand of the micro-service on the server and the resource surplus of each satellite node can be utilized, the resource configuration of each satellite node is realized, the resource utilization balance of the satellite nodes is improved, the calling time delay is reduced, and the configuration efficiency is improved.

Description

Satellite network microservice deployment method and device based on multi-agent reinforcement learning

Technical Field

The present application relates to the field of artificial intelligence technology, and more specifically, to a satellite network microservice deployment method and device based on multi-agent reinforcement learning.

Background Art

With the continuous evolution of software construction and operation mode, the traditional centralized architecture cannot meet various application requirements due to lack of flexibility, difficulty in expansion and migration, and has gradually evolved into a distributed microservice architecture. Microservice architecture is to split complex applications into several relatively independent small applications according to logical relationships. These microservices can be independently developed, updated, expanded and deployed without affecting each other. They use lightweight protocols for communication and can be deployed to different satellite edge nodes. Based on the microservice architecture, the engineering project has higher scalability, reliability and flexible distributed deployment capabilities.

At present, with the growing demand for communication services, the ground communication network has shown its shortcomings such as insufficient spectrum resources and small coverage area. Compared with ground communication, satellite communication has unique advantages, such as wide coverage, high system reliability, large communication capacity, and no impact from natural disasters such as earthquakes. However, each satellite has different resource amounts, different microservers have different requests for satellite resources, and the remaining resources of each satellite are also different. How to deploy different microservers to each satellite so that satellite resources can be reasonably allocated is an urgent problem to be solved.

Summary of the invention

The purpose of some embodiments of the present application is to provide a satellite network microservice deployment method and device based on multi-agent reinforcement learning. Through the technical solution of the embodiments of the present application, by obtaining the resource demand information of the microservice; determining the resource utilization information and delay information of the satellite node according to the pre-established resource utilization model and delay model, and the configuration information of the satellite node; when the resource utilization information is less than a first preset value, or the delay information is less than a second preset value, using a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice, wherein the pre-trained multi-agent strategy deployment model is a multi-agent deep determination policy gradient algorithm for the intelligent network The parameters of the network model are trained; the server terminal is configured according to the deployment strategy of the satellite node. The embodiment of the present application establishes a resource utilization model and a delay model based on the microservice architecture, determines the resource utilization information and delay information of the satellite node according to the configuration information of the satellite node, and then adopts the pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice, and configures the server terminal according to the deployment strategy of the satellite node. In this way, the resources of each satellite node can be configured according to the microservice demand for server resources and the remaining resources of each satellite node, thereby improving the resource utilization balance of the satellite node, reducing the call delay, and improving the configuration efficiency.

In a first aspect, some embodiments of the present application provide a satellite network microservice deployment method based on multi-agent reinforcement learning, including:

Get resource demand information of microservices;

Determine resource utilization information and delay information of the satellite node according to a pre-established resource utilization model and delay model and configuration information of the satellite node;

When the resource utilization information is less than a first preset value, or the delay information is less than a second preset value, a pre-trained multi-agent strategy deployment model is used to determine a deployment strategy of a satellite node corresponding to the resource demand information of the microservice, wherein the pre-trained multi-agent strategy deployment model is obtained by training various parameters of an agent network model using a multi-agent deep determination policy gradient algorithm;

The server terminal is configured according to the deployment strategy of the satellite node.

Some embodiments of the present application establish a resource utilization model and a delay model, determine the resource utilization information and delay information of the satellite nodes according to the configuration information of the satellite nodes, and then use a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the microservice, and configure the server terminal according to the deployment strategy of the satellite node. In this way, resources can be configured for each satellite node according to the microservice's demand for server resources and the remaining resources of each satellite node, that is, microservices with different resource requirements are deployed to appropriate satellite nodes, thereby improving the resource utilization balance of the satellite nodes, reducing the call delay, and improving the configuration efficiency.

Optionally, the multi-agent strategy deployment model is obtained by:

Obtain agent sample parameters, wherein the agent sample parameters at least include the agent observation environment:

Acquire an intelligent network model, the intelligent network model comprising at least an actor network model and a critic network model;

Inputting the agent observation environment into the actor network model and outputting the deployment action of the agent;

Inputting the deployment action and global state of the agent into the critic network model, and outputting the action evaluation value;

Establish a replay pool based on the agent's current state information, action information, reward information, and state information at the next moment;

Adopting a multi-agent deep determination policy gradient algorithm, the network parameters in the actor network model and the critic network model are updated according to the current state information, action information, reward information and state information at the next moment obtained in the replay pool;

When the actor network model and the critic network model converge, the converged actor network model and the critic network model are determined as the multi-agent strategy deployment model.

Some embodiments of the present application convert the microservice deployment problem into a partially observable Markov decision process, and adopt a multi-agent reinforcement learning method to solve it in a centralized training and distributed execution manner. In the training phase, the container instance of the microservice, as an agent, needs to obtain global information to obtain the best deployment plan. In the execution phase, the microservice can complete the deployment only by relying on its own observation space, which greatly reduces the communication overhead between microservices.

Optionally, updating the network parameters in the actor network model and the critic network model includes:

Obtaining a first loss function of an actor network model and a second loss function of the critic network model;

Performing gradient calculation on the first loss function and the second loss function respectively;

The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

Some embodiments of the present application adopt a fixed network method, fix the target network and transfer the original network parameters to the target network at regular intervals to avoid constant changes in the update target and ensure the stability of training.

Optionally, the configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types and the capacity of heterogeneous resources.

Optionally, the resource utilization model is obtained in the following manner:

Obtain resource balance information of a first resource utilization model of different types of resources on the same satellite node, and node balance information of a second resource utilization model of the same type of resources on different satellite nodes;

The resource utilization model is determined according to the resource balance information and the weight value corresponding to the resource balance information, and the node balance information and the weight value corresponding to the node balance information.

Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model and a migration delay sub-model.

Some embodiments of the present application establish a resource utilization model and a delay model, minimize resource utilization variance and delay, and express the microservice deployment problem as a multi-objective optimization problem.

In a second aspect, some embodiments of the present application provide a satellite network microservice deployment device based on multi-agent reinforcement learning, including:

The acquisition module is used to obtain the resource demand information of microservices;

A first determination module, used to determine resource utilization information and delay information of the satellite node according to a pre-established resource utilization model and delay model, and configuration information of the satellite node;

A second determination module is used to use a pre-trained multi-agent strategy deployment model to determine a deployment strategy of a satellite node corresponding to the resource demand information of the microservice when the resource utilization information is less than a first preset value or the delay information is less than a second preset value, wherein the pre-trained multi-agent strategy deployment model is obtained by training various parameters of the agent network model using a multi-agent deep determination policy gradient algorithm;

The configuration module is used to configure the server terminal according to the deployment strategy of the satellite node.

Some embodiments of the present application establish a resource utilization model and a delay model based on the microservice architecture, determine the resource utilization information and delay information of the satellite nodes according to the configuration information of the satellite nodes, and then use a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the microservice, and configure the server terminal according to the deployment strategy of the satellite node. In this way, resources can be configured for each satellite node according to the microservice's demand for server resources and the remaining resources of each satellite node, that is, microservices with different resource requirements are deployed to appropriate satellite nodes, thereby improving the resource utilization balance of the satellite nodes, reducing the call delay, and improving the configuration efficiency.

Optionally, the device further comprises a model training module, wherein the model training module is used to:

Obtain agent sample parameters, wherein the agent sample parameters at least include the agent observation environment:

Acquire an intelligent network model, the intelligent network model comprising at least an actor network model and a critic network model;

Inputting the agent observation environment into the actor network model and outputting the deployment action of the agent;

Inputting the deployment action and global state of the agent into the critic network model, and outputting the action evaluation value;

Establish a replay pool based on the agent's current state information, action information, reward information, and state information at the next moment;

Adopting a multi-agent deep determination policy gradient algorithm, the network parameters in the actor network model and the critic network model are updated according to the current state information, action information, reward information and state information at the next moment obtained in the replay pool;

When the actor network model and the critic network model converge, the converged actor network model and the critic network model are determined as the multi-agent strategy deployment model.

Some embodiments of the present application convert the microservice deployment problem into a partially observable Markov decision process, and adopt a multi-agent reinforcement learning method to solve it in a centralized training and distributed execution manner. In the training phase, the container instance of the microservice, as an agent, needs to obtain global information to obtain the best deployment plan. In the execution phase, the microservice can complete the deployment only by relying on its own observation space, which greatly reduces the communication overhead between microservices.

Optionally, the model training module is used to:

Obtaining a first loss function of an actor network model and a second loss function of the critic network model;

Performing gradient calculation on the first loss function and the second loss function respectively;

The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

Some embodiments of the present application adopt a fixed network method, fix the target network and transfer the original network parameters to the target network at regular intervals to avoid constant changes in the update target and ensure the stability of training.

Optionally, the configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types and the capacity of heterogeneous resources.

Optionally, the model training module is used to:

Obtain resource balance information of a first resource utilization model of different types of resources on the same satellite node, and node balance information of a second resource utilization model of the same type of resources on different satellite nodes;

The resource utilization model is determined according to the resource balance information and the weight value corresponding to the resource balance information, and the node balance information and the weight value corresponding to the node balance information.

Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model and a migration delay sub-model.

Some embodiments of the present application establish a resource utilization model and a delay model, minimize resource utilization variance and delay, and express the microservice deployment problem as a multi-objective optimization problem.

In a third aspect, some embodiments of the present application provide an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the program, the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect can be implemented.

In a fourth aspect, some embodiments of the present application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect.

In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, it can implement the satellite network microservice deployment method based on multi-agent reinforcement learning as described in any embodiment of the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of some embodiments of the present application, the drawings required for use in some embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

FIG1 is a flow chart of a satellite network microservice deployment method based on multi-agent reinforcement learning provided in an embodiment of the present application;

FIG2 is a flow chart of another satellite network microservice deployment method based on multi-agent reinforcement learning provided in an embodiment of the present application;

FIG3 is a schematic diagram of a microservice deployment scenario provided in an embodiment of the present application;

FIG4 is a network structure diagram of the model training provided in an embodiment of the present application;

FIG5 is a schematic diagram of a flow chart of model training provided in an embodiment of the present application;

FIG6 is a schematic diagram of the structure of a satellite network microservice deployment device based on multi-agent reinforcement learning provided in an embodiment of the present application;

FIG. 7 is a schematic diagram of an electronic device provided in an embodiment of the present application.

DETAILED DESCRIPTION

The technical solutions in some embodiments of the present application will be described below in conjunction with the drawings in some embodiments of the present application.

It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

With the continuous evolution of software construction and operation mode, the traditional centralized architecture cannot meet various application requirements due to lack of flexibility, difficulty in expansion and migration, and has gradually evolved into a distributed microservice architecture. Microservice architecture is to split complex applications into several relatively independent small applications according to logical relationships. These microservices can be independently developed, updated, expanded and deployed without affecting each other. They use lightweight protocols for communication and can be deployed to different satellite edge nodes. Based on the microservice architecture, the engineering project has higher scalability, reliability and flexible distributed deployment capabilities.

At present, with the growing demand for communication services, the disadvantages of ground communication networks such as insufficient spectrum resources and small coverage area have emerged. Compared with ground communication, satellite communication has unique advantages, such as wide coverage, high system reliability, large communication capacity, and no impact from natural disasters such as earthquakes. However, each satellite has different resource amounts, different microservers have different requests for satellite resources, and the remaining resources of each satellite are also different. In view of this, some embodiments of the present application provide a satellite network microservice deployment method based on multi-agent reinforcement learning, the method comprising obtaining resource demand information of the microservice; determining the resource utilization information and delay information of the satellite node according to a pre-established resource utilization model and delay model, as well as the configuration information of the satellite node; when the resource utilization information is less than a first preset value, or the delay information is less than a second preset value, using a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice, wherein the pre-trained multi-agent strategy deployment model is used to deploy the satellite node. The agent strategy deployment model is obtained by training various parameters of the agent network model using a multi-agent deep determination policy gradient algorithm; the server terminal is configured according to the deployment strategy of the satellite node. The embodiment of the present application establishes a resource utilization model and a delay model based on the microservice architecture, determines the resource utilization information and delay information of the satellite node according to the configuration information of the satellite node, and then uses the pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite node corresponding to the microservice resource demand information, and configures the server terminal according to the deployment strategy of the satellite node. In this way, the resources of each satellite node can be configured according to the microservice demand for server resources and the remaining resources of each satellite node, that is, microservices with different resource requirements are deployed to appropriate satellite nodes, thereby improving the resource utilization balance of the satellite nodes, reducing the call delay, and improving the configuration efficiency.

As shown in FIG1 , an embodiment of the present application provides a satellite network microservice deployment method based on multi-agent reinforcement learning, the method comprising:

S101. Obtain resource demand information of microservices;

The server terminal is used to execute microservices. Each microservice corresponds to at least one container instance. Container technology is a lightweight resource virtualization technology that abstracts, transforms, and splits computing resources to present one or more computing resources. Among them, Docker is the most popular container technology, which is widely used in microservice deployment and cloud computing platforms.

The scheduling platform obtains the resource demand information of each microservice. For example, the resource demand information includes which microservice obtains the amount of resources of which satellite node, such as CPU, memory, and disk IO.

S102, determining resource utilization information and delay information of the satellite node according to a pre-established resource utilization model and delay model, and configuration information of the satellite node;

The configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types and the capacity of heterogeneous resources.

Specifically, the scheduling platform pre-establishes resource utilization models and delay models. The resource utilization model is represented by variance and is divided into two categories. One is the variance of different types of resources on the same node, which prevents excessive resources of a certain type from causing a short board effect and waste of resources; the other is the variance of the same type of resources on different nodes, which prevents satellite node resources from being idle.

The delay is divided into transmission delay, propagation delay and migration delay. Transmission delay can be expressed as the quotient of the data volume and the transmission rate. The transmission rate can be calculated by Shannon's formula. The propagation delay is proportional to the physical distance between nodes. The migration delay is determined by the migration frequency of the microservice.

The scheduling platform determines the resource utilization information and delay information of the satellite nodes according to the pre-established resource utilization model and delay model and the configuration information of the satellite nodes.

S103, when the resource utilization information is less than the first preset value, or the delay information is less than the second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice, wherein the pre-trained multi-agent strategy deployment model is obtained by training various parameters of the agent network model using a multi-agent deep determination policy gradient algorithm;

Specifically, the scheduling platform obtains global state information in advance. The global state information includes the resource occupancy of satellite nodes, the location information of satellites, and the deployment of containers. The satellite position changes over time. The intelligent agent will take corresponding actions according to the change in position, resulting in state changes. Under the condition that the variance and delay of resource utilization are as small as possible, a reward function is established. Based on the above information, an intelligent agent network model is established, and then the multi-agent deep deterministic policy gradient algorithm (Multi-agent Deep Deterministic Policy Gradient, MADDPG) is used to train the various parameters of the intelligent agent network model to obtain a multi-agent policy deployment model.

When the resource utilization information is less than the first preset value, or the delay information is less than the second preset value, a pre-trained multi-agent strategy deployment model is used to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice.

S104: Configure the server terminal according to the deployment strategy of the satellite node.

Specifically, the scheduling platform configures each microservice based on the deployment strategy of each satellite node. In this way, the satellite node executes the deployment strategy, improves the resource utilization balance of the satellite node, reduces the call delay, and improves the configuration efficiency.

Some embodiments of the present application establish a resource utilization model and a delay model based on the microservice architecture, determine the resource utilization information and delay information of the satellite nodes according to the configuration information of the satellite nodes, and then use a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the microservices, and configure the server terminal according to the deployment strategy of the satellite nodes. In this way, resources can be configured for each satellite node based on the microservice's demand for server resources and the remaining resources of each satellite node, that is, microservices with different resource requirements are deployed to appropriate satellite nodes, thereby improving the resource utilization balance of the satellite nodes, reducing the call delay, and improving the configuration efficiency.

Another embodiment of the present application further supplements the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiment.

FIG2 is a flow chart of another satellite network microservice deployment method based on multi-agent reinforcement learning provided in an embodiment of the present application. As shown in FIG2 , the satellite network microservice deployment method based on multi-agent reinforcement learning includes:

Step 1: Build a microservice deployment model;

Specifically, the network structure, number of satellite nodes and resource types of the satellite edge computing system are determined, and a microservice deployment model is constructed, as shown in FIG3 , including steps 101 to 104, as shown below:

Step 101: Consider a satellite edge computing scenario, as shown in Figure 1, which includes a set of satellite edge nodes S = {s ₁ ,s ₂ ,...,s _N }, where N is the number of satellite nodes. In the edge computing scenario, the total number of resource types is R (CPU, memory, disk IO, etc.), expressed as R = {r ₁ ,r ₂ ,...,r _R }. For node _si , its heterogeneous resource capacity is represented by a vector V _i = {V _i ¹ ,V _i ² ,...,V _i ^R }, where V _i ^j represents the available capacity of resource r _j on node _si .

Step 102: The set of microservices of the target deployed application in the satellite edge computing platform is MS = {ms ₁ ,ms ₂ ,...,ms _M }, where M is the number of microservices, and the microservices are deployed to the edge nodes in the form of containers. The resource requests of different microservices are set as vectors in Represents the number of requests from microservice ms _i to resource r _j .

Step 103: Each microservice can have multiple copies, that is, multiple containers deployed on different nodes. Let the number of containers for each microservice be Q = {q ₁ ,q ₂ ,...,q _M }, where q _i represents the number of container copies in microservice ms _i . Therefore, the total number of containers to be deployed is ∑Q, which can be expressed as a set Indicates the jth container replica corresponding to microservice ms _i . Defines the container instance scheduling decision variable when When it represents a container instance Deployed on node s _i , otherwise the value is 0.

Step 104: The calling relationship between microservices is represented by a directed acyclic graph. The adjacency matrix Y represents the calling relationship. When y _ij =1, it means that the next microservice called by microservice ms _i is ms _j , otherwise the value is 0.

Step 2: Optimize the problem representation, that is, establish a resource utilization model and a latency model. Minimize the resource utilization variance and latency, and represent the microservice deployment problem as a multi-objective optimization problem.

Optionally, the resource utilization model is obtained by:

Obtain resource balance information of a first resource utilization model of different types of resources on the same satellite node, and node balance information of a second resource utilization model of the same type of resources on different satellite nodes;

A resource utilization model is determined according to the resource balance information and the weight value corresponding to the resource balance information, and the node balance information and the weight value corresponding to the node balance information.

In this step, the resource utilization model is represented by variance, which is divided into two categories. One is the variance of different types of resources on the same node, which prevents excessive resources of a certain type from causing a short board effect and waste of resources; the other is the variance of the same type of resources on different nodes, which prevents satellite node resources from being idle.

Step 201: Establish a satellite node resource utilization model.

The utilization rate _uij of resource _rj on node _si can be expressed as:

Resource utilization models are divided into two categories. One is the resource utilization of different types of resources on the same node. Each microservice has different requests for resource types. When multiple microservices with the same resource type are deployed to the same node, other microservices will not be able to be deployed on the node, forming a "short board effect" and causing resource waste. Resource balance is expressed by standard deviation. The balance of all resources on node _{si can} be expressed as:

The other type is the resource utilization of the same type of resources on different nodes. When the number of _{microservices} increases, it is hoped that all satellite edge nodes can be utilized to prevent idle resources and waste of resources. It can be expressed as:

Therefore, the calculation formula for the cluster resource utilization U is, where α is the weight factor:

Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model and a migration delay sub-model.

Among them, the delay is divided into transmission delay, propagation delay and migration delay. Transmission delay can be expressed as the quotient of the data volume and the transmission rate. The transmission rate can be calculated by Shannon's formula. Propagation delay is proportional to the physical distance between nodes. Migration delay is determined by the migration frequency of microservices.

Step 202: Establish a delay model, which is divided into transmission delay, propagation delay and migration delay.

Specifically, according to the service deployment scenario, it is necessary to consider the communication link between the ground station and the satellite node and the communication link between the satellite nodes. According to Shannon's theorem, the data transmission rate from the ground station to the destination satellite node can be expressed as:

Where _{Wg_s} is the channel bandwidth, _pg is the transmit power of the ground station, _{gg_s} is the channel gain between the ground station and the target satellite, _N0 represents the background noise, Represents the sum of other noise interference powers from ground stations to satellites.

set up is the channel bandwidth of the intersatellite link between nodes _si and _sj , is the signal-to-noise ratio between the two nodes, then the data transmission rate between the two satellite nodes is:

According to the adjacency matrix, the complete scheduling chain data transmission delay can be calculated as:

Among them, d _{g_s} represents the amount of data transmitted from the ground station to the satellite, and d _i,j represents the amount of data transmitted by satellite nodes _si and _sj .

The information propagation rate is the speed of light c, and the distance between nodes _si and _sj is represented by dis _i,j . Then the propagation delay can be expressed as:

When the satellite moves out of the visible range of the service cell, it needs to make a migration action. When the agent makes a migration action, a migration delay will be generated. The migration action of the agent is modeled as a directed weight graph, and the weight is the migration delay from the original satellite to the destination satellite. The overall migration cost is the sum of the weights, expressed as

Tr represents the entire migration link under the action of the agent within this time period, and w _i,j represents the weight.

The overall delay D can be calculated as follows:

D＝τ _trans +τ _prop +τ _mig

Some embodiments of the present application establish a resource utilization model and a delay model, minimize resource utilization variance and delay, and express the microservice deployment problem as a multi-objective optimization problem.

Step 203: Optimize the problem representation.

Based on the above model, a joint optimization problem can be established to minimize the standard deviation of resource utilization U and minimize the overall delay D, which can be expressed as:

P:min(U),min(D)

Among them, C1 means that each microservice deploys at least one container instance to implement the function of the microservice, and C2 means that the amount of data requested by the microservice cannot be greater than the maximum resource capacity of the node.

Step 3: Represent the microservice deployment problem as a partially observable Markov decision process and solve it using a multi-agent reinforcement learning method.

As shown in Figure 4, since the agent cannot obtain all the state information, the problem is partially observable. Each agent has its own observation space. As time changes, the relative position of the satellite changes, which in turn affects the communication delay. Therefore, the state of the environment changes with time and the action of the agent.

Optionally, the multi-agent strategy deployment model is obtained by:

Obtain agent sample parameters, where the agent sample parameters at least include the agent observation environment:

Obtaining an intelligent network model, the intelligent network model at least includes an actor network model and a critic network model;

Input the agent's observed environment into the actor network model and output the agent's deployment action;

Input the agent's deployment actions and global state into the critic network model and output the action evaluation value;

Establish a replay pool based on the agent's current state information, action information, reward information, and state information at the next moment;

Adopting the multi-agent deep deterministic policy gradient algorithm, the network parameters in the actor network model and the critic network model are updated according to the current state information, action information, reward information and the state information of the next moment obtained from the replay pool;

When the actor network model and the critic network model converge, the converged actor network model and critic network model are determined as the multi-agent strategy deployment model.

Some embodiments of the present application convert the microservice deployment problem into a partially observable Markov decision process, and adopt a multi-agent reinforcement learning method to solve it in a centralized training and distributed execution manner. In the training phase, the container instance of the microservice, as an intelligent agent, needs to obtain global information to obtain the best deployment plan. In the execution phase, the microservice can complete the deployment only by relying on its own observation space, which greatly reduces the communication overhead between microservices.

Optionally, the network parameters in the actor network model and the critic network model are updated, including:

Obtain the first loss function of the actor network model and the second loss function of the critic network model;

Perform gradient calculations on the first loss function and the second loss function respectively;

The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

Some embodiments of the present application adopt a fixed network method, fix the target network and transfer the original network parameters to the target network at regular intervals to avoid constant changes in the update target and ensure the stability of training.

Specifically, in the embodiments of the present application, the partially observable Markov decision process representation is node preselection;

Step 301: State space representation. The global state information includes the resource occupancy of satellite nodes, the location information of satellites, and the deployment of containers, which can be represented as s = [u, p, c], where:

u＝[u _1,1 ,u _1,2 ,...,u _1,R ,u _2,1 ,u _2,2 ,...,u _2,R ,...,u _N,1 , u _N,2 ,...,u _N,R ]

p＝[x ₁ ,y ₁ ,z ₁ ,x ₂ ,y ₂ ,z ₂ ,...,x _N ,y _N ,z _N ]

u _i,j is the utilization rate of resource r _j on node _si , [x _i ,y _i , _zi ] is the location coordinate of node _si , The index number of the node where the container is deployed.

As an intelligent agent, the container cannot obtain global state information during the deployment process. The observation space of container instance j on microservice i can be expressed as

Step 302: Action space representation. Based on its own observation space, the action space of container instance j on microservice i is represented as k is the number of nodes that meet the resource requirements in the observation space. If it is, it indicates that the container is deployed on the node; otherwise, it is 0.

Step 303: State transfer function representation. The satellite position changes over time, and the agent will take corresponding actions according to the change in position, resulting in a change in state. The state transfer function can be represented as

Step 304: Reward function representation. We hope that the variance of resource utilization and latency are as small as possible, so the reward function can be expressed as

reward＝-(βU+(1-β)D)

Step 4: Use the multi-agent deep deterministic policy gradient algorithm MADDPG to train the parameters of the agent network model, including building a neural network and updating network parameters;

These include:

Build an agent network, as shown in Figure 4. Consider the container as an agent, each agent contains four networks, Actor network μ(o ⁱ ; θ ⁱ ), Target actor network t_μ(o ⁱ ; θ ⁱ ), Critic network c(s,a; ω ⁱ ), Targetcritic network t_c(s,a; ω ⁱ ). It includes steps 401 to 402. The fixed network method is adopted to fix the Target network and pass the original network parameters to the Target network at regular intervals to avoid the constant change of the update target and ensure the stability of training.

Step 401: Setting up two networks. The input of the Actor network, i.e., the actor network model, is the local observation information o ⁱ of the current agent, including the resource occupancy of the node, and the output is the deployment action a. The input of the Critic network, i.e., the critic network model, is the action and global state output by the Actor network, i.e., the global state information s and the action a, and the output is the corresponding Q value, which is used to judge the quality of the action performed by the agent in the current state.

Step 402: Network parameter transfer process. When updating parameters, if the update target keeps changing, it will cause update difficulties. Therefore, a fixed network method is adopted to fix the parameters of the Target network and transfer the original network parameters to the Target network at intervals to ensure the stability of training.

Step 5: Build an experience replay pool D. The agent randomly takes deployment actions according to the noise settings, generating a four-tuple, namely the record state, the agent's action, the next moment state and the reward, recorded as (s _t , a _t , r _t , s _t+1 ). Since the MADDPG algorithm is a heterogeneous strategy, the experience replay pool can be used to eliminate the correlation of historical experience and break it up. When training the neural network, a batch of experience data is randomly selected to make the neural network train better.

Step 6: Execute the MADDPG algorithm to update network parameters and perform centralized training. Randomly select a quadruple from the playback pool and update the Actor and Critic network parameters until convergence.

It is mainly reflected in the update process of two types of networks, including steps 601 to 602.

Step 601: Update the Actor network parameters. The loss function of the Actor network is -Q, which needs to be obtained by inputting the output action of the Actor network into the current Critic network. The smaller -Q, the better. Input the observation space o ⁱ of agent i in the playback pool into the Actor network μ(o ⁱ ; θ ⁱ ) of the agent to obtain the deployment action a _i , and then input the global state information s and a _i into the Critic network to obtain the Q value of the action. Use -Q as the loss function to update the network parameter θ ⁱ using gradient descent. Specifically, the loss function can be expressed as

Where x = (o ₁ ,o ₂ ,...,o _N ) represents the observation space of all agents, and a _i represents the action of agent i under its strategy μ _i . According to the chain rule, its gradient can be expressed as

Use gradient descent to update the parameters θ ⁱ

Step 602: Update the parameters of the Critic network. The Critic network needs to make the predicted Q value as accurate as possible, so its loss function is the difference between the output Q(s ₀ ,a ₀ ;ω ⁱ ) value of the Critic network (predicted value) and the output Q value of the Targetcritic network and the sum of the reward r ₁ +γQ(s ₁ ,a ₁ ;ω ⁱ ) (actual value). The smaller the difference, the better. The difference can be represented by MSE, and the network parameters ω ⁱ are updated using the gradient descent method. Specifically, the loss function can be expressed as

Computing Gradients

Update the parameters ω ⁱ using the gradient descent method

The two networks calculate the gradients according to their respective loss functions and use the gradient descent method to update the network parameters.

FIG5 is a schematic diagram of a flow chart of model training provided in an embodiment of the present application, as shown in FIG5 , including:

1) Initialize the network and satellite nodes to obtain a set of candidate nodes;

2) The container agent randomly generates actions and obtains a quaternion;

3) Store the quadruple into the experience replay pool;

4) Randomly extract a quadruple;

5) Update the Actor network and Critic network according to the LOSS function;

6) Update the Target network parameters, i.e. the target network parameters;

7) Output the trained strategy network, i.e., the multi-agent strategy deployment model.

Step 5: Policy network deployment.

By deploying the trained policy network to the container agent, microservices can independently make optimal decisions based on local observations.

The embodiment of the present application provides a satellite network microservice deployment method based on multi-agent reinforcement learning to solve the problem of microservice deployment. The neural network part adopts a fixed network method, which is divided into two categories: the original network and the target network. The target network is fixed first and the original network parameters are passed to the target network at intervals, avoiding the update difficulty caused by the constant change of the update target and ensuring the stability of the training. After the training is completed, the agent only needs to rely on its own observation space and use the strategy network to make the best action, reducing the overhead caused by the frequent interaction between microservices.

It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any manner without conflict, and this application is not limited thereto.

Another embodiment of the present application provides a satellite network microservice deployment device based on multi-agent reinforcement learning, which is used to execute the satellite network microservice deployment method based on multi-agent reinforcement learning provided by the above embodiment.

As shown in Figure 6, it is a schematic diagram of the structure of the satellite network microservice deployment device based on multi-agent reinforcement learning provided in an embodiment of the present application. The satellite network microservice deployment device based on multi-agent reinforcement learning includes an acquisition module 601, a first determination module 602, a second determination module 603 and a configuration module 604, wherein:

The acquisition module 601 is used to obtain resource demand information of microservices;

The first determination module 602 is used to determine the resource utilization information and delay information of the satellite node according to the pre-established resource utilization model and delay model, and the configuration information of the satellite node;

The second determination module 603 is used to use a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite node corresponding to the resource demand information of the microservice when the resource utilization information is less than the first preset value or the delay information is less than the second preset value, wherein the pre-trained multi-agent strategy deployment model is obtained by training various parameters of the agent network model using a multi-agent deep determination policy gradient algorithm;

The configuration module 604 is used to configure the server terminal according to the deployment strategy of the satellite node.

Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

Some embodiments of the present application establish a resource utilization model and a delay model based on the microservice architecture, determine the resource utilization information and delay information of the satellite nodes according to the configuration information of the satellite nodes, and then use a pre-trained multi-agent strategy deployment model to determine the deployment strategy of the satellite nodes corresponding to the resource demand information of the microservices, and configure the server terminal according to the deployment strategy of the satellite nodes. In this way, resources can be configured for each satellite node according to the microservice's demand for server resources and the remaining resources of each satellite node, that is, microservices with different resource requirements are deployed to appropriate satellite nodes, thereby improving the resource utilization balance of the satellite nodes, reducing the call delay, and improving the configuration efficiency.

Another embodiment of the present application further supplements the satellite network microservice deployment device based on multi-agent reinforcement learning provided in the above embodiment.

Optionally, the device further includes a model training module, and the model training module is used to:

Obtain agent sample parameters, where the agent sample parameters at least include the agent observation environment:

Obtain an intelligent network model, which includes at least an actor network model and a critic network model;

Input the agent's observed environment into the actor network model and output the agent's deployment action;

Input the agent's deployment actions and global state into the critic network model and output the action evaluation value;

Establish a replay pool based on the agent's current state information, action information, reward information, and state information at the next moment;

Adopting the multi-agent deep deterministic policy gradient algorithm, the network parameters in the actor network model and the critic network model are updated according to the current state information, action information, reward information and the state information of the next moment obtained from the replay pool;

When the actor network model and the critic network model converge, the converged actor network model and critic network model are determined as the multi-agent strategy deployment model.

Some embodiments of the present application convert the microservice deployment problem into a partially observable Markov decision process, and adopt a multi-agent reinforcement learning method to solve it in a centralized training and distributed execution manner. In the training phase, the container instance of the microservice, as an agent, needs to obtain global information to obtain the best deployment plan. In the execution phase, the microservice can complete the deployment only by relying on its own observation space, which greatly reduces the communication overhead between microservices.

Optionally, the model training module is used to:

Obtain the first loss function of the actor network model and the second loss function of the critic network model;

Perform gradient calculations on the first loss function and the second loss function respectively;

The gradient descent method is used to update the network parameters in the actor network model and the critic network model.

Some embodiments of the present application adopt a fixed network method, fix the target network and transfer the original network parameters to the target network at regular intervals to avoid constant changes in the update target and ensure the stability of training.

Optionally, the configuration information of the satellite nodes includes at least the number of satellite nodes, the total number of resource types and the capacity of heterogeneous resources.

Optionally, the model training module is used to:

Obtain resource balance information of a first resource utilization model of different types of resources on the same satellite node, and node balance information of a second resource utilization model of the same type of resources on different satellite nodes;

A resource utilization model is determined according to the resource balance information and the weight value corresponding to the resource balance information, and the node balance information and the weight value corresponding to the node balance information.

Optionally, the delay model includes at least a transmission delay sub-model, a propagation delay sub-model and a migration delay sub-model.

Some embodiments of the present application establish a resource utilization model and a delay model, minimize resource utilization variance and delay, and express the microservice deployment problem as a multi-objective optimization problem.

Regarding the device in this embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

It should be noted that each implementable method in this embodiment may be implemented separately, or may be implemented in combination in any manner without conflict, and this application is not limited thereto.

An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the operation of the method corresponding to any embodiment of the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiments can be implemented.

An embodiment of the present application also provides a computer program product, which includes a computer program, wherein when the computer program is executed by a processor, it can implement the operations corresponding to any embodiment of the satellite network microservice deployment method based on multi-agent reinforcement learning provided in the above embodiment.

As shown in Figure 7, some embodiments of the present application provide an electronic device 700, which includes: a memory 710, a processor 720, and a computer program stored in the memory 710 and executable on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and executes the program to implement a method of any embodiment of the satellite network microservice deployment method based on multi-agent reinforcement learning as described above.

Processor 720 can process digital signals and can include various computing structures, such as complex instruction set computer structure, reduced instruction set computer structure, or a structure that implements a combination of multiple instruction sets. In some examples, processor 720 can be a microprocessor.

The memory 710 may be used to store instructions executed by the processor 720 or data related to the execution of instructions. These instructions and/or data may include codes for implementing some or all functions of one or more modules described in the embodiments of the present application. The processor 720 of the disclosed embodiment may be used to execute instructions in the memory 710 to implement the method shown above. The memory 710 includes a dynamic random access memory, a static random access memory, a flash memory, an optical memory, or other memory known to those skilled in the art.

The above are only embodiments of the present application and are not intended to limit the scope of protection of the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

Claims (10)

1. The satellite network micro-service deployment method based on multi-agent reinforcement learning is characterized by comprising the following steps:

acquiring resource demand information of the micro service;

determining resource utilization rate information and time delay information of the satellite node according to a pre-established resource utilization rate model and time delay model and configuration information of the satellite node;

when the resource utilization rate information is smaller than a first preset value or the time delay information is smaller than a second preset value, a pre-trained multi-agent strategy deployment model is adopted to determine a deployment strategy of a satellite node corresponding to the resource demand information of the micro service, wherein the pre-trained multi-agent strategy deployment model is obtained by training each parameter of an agent network model by adopting a multi-agent depth determination strategy gradient algorithm;

and configuring the server terminal according to the deployment strategy of the satellite node.

2. The satellite network micro-service deployment method based on multi-agent reinforcement learning according to claim 1, wherein the multi-agent policy deployment model is obtained by:

obtaining an agent sample parameter, wherein the agent sample parameter at least comprises an agent observation environment:

Acquiring an intelligent network model, wherein the intelligent network model is at least an actor network model and a criticizer network model;

inputting the intelligent agent observation environment into the actor network model, and outputting deployment actions of intelligent agents;

inputting the deployment action and the global state of the intelligent agent into the criticizer network model, and outputting an action judgment value;

according to the current state information, action information, rewarding information and state information of the intelligent agent at the next moment, a playback pool is established;

a multi-agent depth determination strategy gradient algorithm is adopted, and network parameters in the actor network model and the criticizer network model are updated according to the current state information, the action information, the rewarding information and the state information of the next moment acquired in the playback pool;

and under the condition that the actor network model and the criticizer network model are converged, determining the converged actor network model and the criticizer network model as the multi-agent strategy deployment model.

3. The satellite network micro-service deployment method based on multi-agent reinforcement learning of claim 2, wherein the updating of network parameters in the actor network model and the criticizer network model comprises:

Acquiring a first loss function of an actor network model and a second loss function of the criticizer network model;

gradient calculation is carried out on the first loss function and the second loss function respectively;

and updating network parameters in the actor network model and the criticism network model by using a gradient descent method.

4. The satellite network micro-service deployment method based on multi-agent reinforcement learning according to claim 1, wherein the configuration information of the satellite nodes at least comprises the number of satellite nodes, the total amount of resource types and heterogeneous resource capacity.

5. The satellite network micro-service deployment method based on multi-agent reinforcement learning of claim 1, wherein the resource utilization model is obtained by:

acquiring resource balance information of a first resource utilization rate model of different types of resources on the same satellite node and node balance information of a second resource utilization rate model of the same type of resources on different satellite nodes;

and determining the resource utilization rate model according to the resource balance degree information and the weight value corresponding to the resource balance degree information, and the node balance degree information and the weight value corresponding to the node balance degree information.

6. The satellite network micro-service deployment method based on multi-agent reinforcement learning of claim 1, wherein the delay model comprises at least a transmission delay sub-model, a propagation delay sub-model, and a migration delay sub-model.

7. A satellite network micro-service deployment device based on multi-agent reinforcement learning, the device comprising:

the acquisition module is used for acquiring resource demand information of the micro service;

the first determining module is used for determining the resource utilization rate information and the time delay information of the satellite node according to a pre-established resource utilization rate model and a time delay model and configuration information of the satellite node;

the second determining module is configured to determine a deployment strategy of a satellite node corresponding to the resource demand information of the micro service by using a pre-trained multi-agent strategy deployment model when the resource utilization rate information is smaller than a first preset value or the time delay information is smaller than a second preset value, where the pre-trained multi-agent strategy deployment model is obtained by training each parameter of an agent network model by using a multi-agent depth determination strategy gradient algorithm;

And the configuration module is used for configuring the server terminal according to the deployment strategy of the satellite node.

8. The multi-agent reinforcement learning based satellite network micro-service deployment device of claim 7, further comprising a model training module for:

obtaining an agent sample parameter, wherein the agent sample parameter at least comprises an agent observation environment:

acquiring an intelligent network model, wherein the intelligent network model is at least an actor network model and a criticizer network model;

inputting the intelligent agent observation environment into the actor network model, and outputting deployment actions of intelligent agents;

inputting the deployment action and the global state of the intelligent agent into the criticizer network model, and outputting an action judgment value;

according to the current state information, action information, rewarding information and state information of the intelligent agent at the next moment, a playback pool is established;

a multi-agent depth determination strategy gradient algorithm is adopted, and network parameters in the actor network model and the criticizer network model are updated according to the current state information, the action information, the rewarding information and the state information of the next moment acquired in the playback pool;

And under the condition that the actor network model and the criticizer network model are converged, determining the converged actor network model and the criticizer network model as the multi-agent strategy deployment model.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor, when executing the program, implements the multi-agent reinforcement learning-based satellite network micro-service deployment method of any one of claims 1-6.

10. A computer readable storage medium, wherein a computer program is stored on the computer readable storage medium, and wherein the program is executed by a processor to implement the satellite network micro-service deployment method based on multi-agent reinforcement learning according to any one of claims 1 to 6.