WO2024152902A1 - 一种基于强化学习的数字孪生网络切片的方法和装置 - Google Patents

一种基于强化学习的数字孪生网络切片的方法和装置 Download PDF

Info

Publication number
WO2024152902A1
WO2024152902A1 PCT/CN2024/070341 CN2024070341W WO2024152902A1 WO 2024152902 A1 WO2024152902 A1 WO 2024152902A1 CN 2024070341 W CN2024070341 W CN 2024070341W WO 2024152902 A1 WO2024152902 A1 WO 2024152902A1
Authority
WO
WIPO (PCT)
Prior art keywords
network
pnf
node
reinforcement learning
vnf
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2024/070341
Other languages
English (en)
French (fr)
Inventor
徐安然
李文超
尹山
匡立伟
谢秋红
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fiberhome Telecommunication Technologies Co Ltd
Original Assignee
Fiberhome Telecommunication Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fiberhome Telecommunication Technologies Co Ltd filed Critical Fiberhome Telecommunication Technologies Co Ltd
Publication of WO2024152902A1 publication Critical patent/WO2024152902A1/zh
Priority to MX2024013573A priority Critical patent/MX2024013573A/es
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0893—Assignment of logical groups to network elements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
    • G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
    • G06F16/367—Ontology
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence

Definitions

  • the present invention relates to the field of network communications, and in particular to a method and device for digital twin network slicing based on reinforcement learning.
  • network slicing technology can flexibly allocate resources on a shared physical network and build multiple dedicated, isolated virtualized logical networks on demand, that is, multiple network slices, each of which is used to meet specific business needs.
  • Network slicing is achieved through network virtualization technology and consists of a set of virtual network functions (Virtual Network Function, abbreviated as VNF) that meet business needs and links connecting every two virtual network functions.
  • VNF Virtual Network Function
  • Network slicing can be regarded as a resource allocation process for VNFs and links based on a shared network infrastructure. How to flexibly and efficiently allocate limited resources to meet the service requirements of each network slice is a hot research issue in network slicing.
  • the present invention solves the problem of difficulty in policy adjustment during network slicing.
  • the present invention provides a method for digital twin network slicing based on reinforcement learning, specifically: the reinforcement learning module obtains the current network information triple and the network slice requirement triple in the knowledge graph, and each network slice corresponds to a set of network slice requirement triples, wherein the current network information triple includes all PNF connection relationships in the network and the resource attributes of each PNF, and the network slice requirement triple includes all VNF connection relationships of the slice requirements, the resource requirements of each VNF and the slice priority; the reinforcement learning module generates at least one decision action for the network slice, each decision action includes searching for the PNF node according to the node selection probability model and the knowledge graph, and mapping the resources allocated to each PNF node to the VNF according to the resource allocation model into a resource allocation vector, the knowledge graph makes a decision according to the decision action to obtain a decision result, and the decision result is the deployment location of the network slice VNF and the network slice PNF; the knowledge graph is updated according to the decision result, and the updated network status is transmitted back to the reinforcement learning module.
  • the reinforcement learning module generates at least one decision action for a network slice, specifically including: according to the slice requirements, based on the knowledge graph, finding the PNF with the highest probability in the node selection probability model as the first node; obtaining a resource evaluation vector based on the knowledge graph, inputting it into the resource allocation model to allocate resources to the first VNF node, and updating the knowledge graph; according to the position selected by the previous VNF node, performing knowledge reasoning based on the PNF connection relationship contained in the knowledge graph, finding the possible PNF position selection for the next VNF node, and searching in sequence until all nodes are deployed and allocated.
  • the method of finding the PNF with the highest probability in the node selection probability model as the first node specifically includes: finding all possible PNF positions in the action space of the first VNF node and selecting them as the deployment action space of the first VNF node, and obtaining a node state matrix; inputting the node state matrix into the node selection probability model, calculating the probability of each node being selected, and selecting the PNF with the highest probability as the first node; obtaining a resource evaluation vector of the first node based on the knowledge graph, inputting the evaluation vector into the resource allocation model, and allocating resources to the first VNF node.
  • the method of finding the possible PNF position selection of the next VNF node specifically includes: performing knowledge reasoning based on the PNF connection relationship contained in the knowledge graph according to the position selected by the previous VNF node, finding all possible PNF position selections of the next VNF node as the deployment action space of the next VNF node; obtaining the node state matrix based on the knowledge graph, and performing the deployment and resource allocation of the next node backward in sequence until all nodes are deployed and allocated, and completing the selection of the entire slice deployment strategy with the allocation results of all nodes.
  • the knowledge graph is updated according to the decision results, specifically including: the knowledge graph is updated according to the resource allocation strategy of reinforcement learning, and the feedback value is calculated according to the feedback function; and the parameters of the node selection probability model are updated using the gradient descent method in combination with the feedback value.
  • the node selection probability model specifically includes: mapping the distance between PNFs, the resource requirements of slices for VNF nodes, and the resource status on the PNF into the probabilities of possible selections, so as to facilitate the selection of the reinforcement learning module to obtain the PNF node with the highest probability for deployment.
  • the resource allocation model specifically includes: mapping the resource evaluation vector into a resource allocation vector through a function, so as to represent the computing, storage, and network resources allocated by the PNF node to the VNF.
  • the reinforcement learning module confirms that the network status is normal and also includes: when a network slice request is being deployed at a certain node, and a PNF with resources that meet the demand cannot be found because the previous network slice occupies too many resources, the resources of each VNF in the first few network slices are proportionally reduced according to the priority ranking of the network slices, without being lower than the minimum demand, and the size of the reduced resources is inversely proportional to the priority of the network slice.
  • the reinforcement learning module confirms that the network status is normal, and also includes: when the network status changes and the network status at the current moment in the knowledge graph module cannot meet the slicing requirements at the original moment, the reinforcement learning module returns to the unsatisfied slicing step to redeploy the decision.
  • the present invention provides a device for digital twin network slicing based on reinforcement learning, specifically: including at least one processor and a memory, at least one processor and the memory are connected through a data bus, the memory stores instructions that can be executed by at least one processor, and after the instructions are executed by the processor, they are used to complete the method for digital twin network slicing based on reinforcement learning in the first aspect.
  • the beneficial effects of the embodiments of the present invention are: using a reinforcement learning module to generate a network slice deployment strategy, quickly completing the decision of the time slice deployment plan in a dynamic network environment, obtaining the optimal slice deployment plan in the current network scenario, and better ensuring the efficient use of network resources.
  • FIG1 is a schematic diagram of a network structure model for executing network slicing
  • FIG2 is a flow chart of a method for digital twin network slicing based on reinforcement learning provided by an embodiment of the present invention
  • FIG3 is a flow chart of another method for digital twin network slicing based on reinforcement learning provided by an embodiment of the present invention.
  • FIG4 is a flow chart of another method for digital twin network slicing based on reinforcement learning provided by an embodiment of the present invention.
  • FIG5 is a schematic diagram of a reinforcement learning module architecture provided by an embodiment of the present invention.
  • Figure 6 is a schematic diagram of the device structure of a digital twin network slicing based on reinforcement learning provided in an embodiment of the present invention.
  • the present invention is an architecture of a specific functional system, so the specific embodiments mainly illustrate the functional logical relationship between the various structural modules, and do not limit the specific software and hardware implementation methods.
  • Embodiment 1 is a diagrammatic representation of Embodiment 1:
  • Reinforcement Learning is an artificial intelligence technology. The main idea is to obtain feedback based on continuous interaction with the environment and continuously optimize decisions. This patent combines knowledge graphs and reinforcement learning to implement network slice deployment decisions, which can better ensure the efficient use of network resources.
  • the method described in this patent implements network slicing through a network slice deployment strategy generation method based on reinforcement learning.
  • the generated network slice deployment strategy can better ensure the operating status of each network slice and the entire network.
  • the method for digital twin network slicing based on reinforcement learning has the following specific steps:
  • Step 101 The reinforcement learning module obtains the current network information triples and network slice requirement triples in the knowledge graph, and each network slice corresponds to a set of network slice requirement triples.
  • the input of the reinforcement learning module comes from the knowledge graph, which consists of "entity-relationship-entity" triples and entities and their related attribute-value pairs.
  • the triples can be expressed as (Thing1, Relation, Thing2), where Thing1 represents the first entity, Thing2 represents the second entity, and Relation represents the relationship between Thing1 and Thing2.
  • the method of this embodiment is used to calculate the deployment strategy of network slices based on the current network status, so network information triples and network slice demand triples are required.
  • the current network information triple includes all PNF connection relationships in the network, and the resource attributes of each PNF.
  • the resource attributes include the determined hardware resource attributes or available resource attributes of each PNF, such as computing, storage, and network.
  • Thing1 represents PNF
  • Relation represents the triple.
  • Resource type Thing2 represents the value of this type of resource required by Thing1.
  • a PNF represented as PNF1
  • PNF1 requires computing, storage, and network resources of 100, 200, and 300, respectively.
  • the PNF can be represented as: (PNF1, connection, PNF2), (PNF1, computing, 100), (PNF1, storage, 200), (PNF1, network, 300).
  • the network slice requirement triple includes all VNF connection relationships required by the slice, the resource requirements of each VNF, and the slice priority. Each network slice corresponds to a set of network slice requirement triples.
  • Thing1 represents a network slice or VNF
  • Relation represents the resource type of the triple
  • Thing2 represents the value of this type of resource required by Thing1.
  • the type of Relation is "required VNF" or priority, etc.
  • Thing2 is the VNF or priority value required by Thing1.
  • VNF1 is required in network slice 1
  • the triplet is (slice 1, required, VNF1).
  • Thing1 is a VNF, similar to the network information triplet
  • Relation is the type of resources such as computing, storage, and network allocated to the VNF
  • Thing2 is the value of the resource type.
  • the computing resource allocation value of VNF1 is 100
  • the computing resource triplet of VNF1 is (VNF1, computing, 100).
  • the reinforcement learning module will receive demands from multiple network slices at the same time.
  • Step 102 The reinforcement learning module generates at least one decision action for a network slice.
  • Each decision action includes searching for a PNF node based on a node selection probability model and a knowledge graph, and mapping the resources allocated to each PNF node by the VNF according to a resource allocation model into a resource allocation vector.
  • the knowledge graph makes a decision based on the decision action to obtain a decision result, which is the deployment location of the network slice VNF and the network slice PNF.
  • the output of the reinforcement learning module is the deployment strategy of the network slice, that is, on which PNF each VNF in a network slice is deployed and how many resources are allocated to each VNF.
  • the output of the reinforcement learning module will be passed to the knowledge graph, and the knowledge graph will be updated.
  • Step 103 The knowledge graph is updated according to the decision result, and the updated network status is transmitted back to the reinforcement learning module. After the reinforcement learning module confirms that the network status is normal, it issues the deployment strategy of the network slice.
  • step 102 through the decision of the reinforcement learning module, the VNF required by the network slice will change, and the computing, storage, network and other resources required by each VNF will also change. Therefore, it is necessary to update the triples in the knowledge graph accordingly based on the data after the decision. For example, originally only VNF1 was needed in network slice 1, expressed as (slice 1, required, VNF1). After the decision, it changes to requiring VNF1 and VNF2, and the triples are updated to (slice 1, required, VNF1), (slice 1, required, VNF2). For another example, the computing resource allocation value of VNF1 changes from 100 to 150, and the corresponding triples also need to be updated from (VNF1, computing, 100) to (VNF1, computing, 150).
  • a network slice requirement is converted into allocation requirements of multiple VNFs.
  • the allocation process of each VNF first confirms which PNF it is allocated to, and then allocates the corresponding resources according to the requirements.
  • Each time a VNF is allocated a change in the triplet occurs, and the next VNF needs to be checked according to the new triplet. Therefore, each time a VNF is allocated, there is an interaction with the environmental knowledge graph.
  • the knowledge graph transmits the updated network status back to the reinforcement learning module.
  • the reinforcement learning module evaluates the adjusted network status and, after confirming that the network status is normal after using the slicing strategy, transmits the deployment decision information to the control module for distribution.
  • the reinforcement learning module can be used to complete the optimal selection of network slicing deployment decisions based on the current network status, and the decision results can be confirmed to ensure that the network status is normal after the new slicing strategy is executed.
  • the network slice deployment decision based on reinforcement learning mainly consists of the following parts: environment, action, state, reward and decision model.
  • the environment of reinforcement learning is the network knowledge graph.
  • State refers to the current state of the network.
  • Action is a decision made by reinforcement learning in the current state, that is, the node selection plan and resource allocation plan of a network slice.
  • Reward refers to the feedback given by the environment after reinforcement learning makes an action, which reflects the quality of the slice deployment decision.
  • the decision model in this patent includes a node selection probability model and a resource allocation model, and reinforcement learning generates a slice deployment decision action based on the decision model.
  • the following steps can be used to complete the deployment strategy of the reinforcement learning module to generate network slices.
  • Step 201 According to the slicing requirements and based on the knowledge graph, find the PNF with the highest probability in the node selection probability model as the first node.
  • the node state matrix is obtained.
  • the node state matrix is input into the node selection probability model, the probability of each node being selected is calculated, and the PNF with the highest probability is selected as the first node.
  • the resource evaluation vector of the first node is obtained based on the knowledge graph, and the evaluation vector is input into the resource allocation model to allocate resources to the first VNF node.
  • Step 202 Obtain a resource evaluation vector based on the knowledge graph, input it into the resource allocation model to allocate resources to the first VNF node, and update the knowledge graph.
  • the knowledge graph is updated according to the resource allocation strategy of reinforcement learning, and the feedback value is calculated according to the feedback function.
  • This formula represents the current node selection probability model after the selection probability of each node in the node selection probability model is gradient decreased according to the feedback value.
  • Step 203 Based on the location selected by the previous VNF node, knowledge reasoning is performed based on the PNF connection relationship contained in the knowledge graph to find the possible PNF location selection for the next VNF node, and the search is carried out in sequence until all nodes are deployed and allocated.
  • a deployment decision for the next network slice is made.
  • step 102 After steps 201 to 203 provided in this embodiment, the decision of network slicing in step 102 can be completed, and the deployment strategies of all VNFs and PNFs can be obtained. Combined with the above steps, the complete process of network slicing and adjustment is shown in Figure 4.
  • the deployment decisions need to be passed to the knowledge graph module to update the knowledge graph module.
  • the knowledge graph then passes the updated network status back to the reinforcement learning module.
  • the decision accuracy is improved through the process of circular feedback, and the corresponding relationship in the knowledge graph can be consistent with the current environment when the network environment changes dynamically.
  • the reinforcement learning model can be constructed and deployed and fed back in the following ways.
  • the reinforcement learning module shown in FIG5 can be used.
  • the network operation status of the network slice specifically includes the location, resources and relationship information of the network slice PNF and the network slice VNF.
  • a network slice S s Represents a VNF node, Represents the virtual link between VNF nodes, for example Indicates the slice and For a virtual link between
  • the computing, storage, and network resources are represented as VNF nodes in a slice
  • the computing, storage, and network resources are represented as
  • a PNF (A VNF ) can be expressed using the following formula.
  • the availability of the entire slice is equal to the minimum availability of the PNFs and VNFs that make up the slice, which is used to evaluate the network operation status.
  • the computing, storage, and network resources should meet the following conditions:
  • the network operation state is defined according to the PNF, VNF availability and resource conditions, as follows: when slicing the network, select the VNF and the deployed PNF, and sort the PNFs according to the spatial position relationship.
  • the total number of PNFs is k, and the set of all PNFs is represented as PNF (PNF 1 , PNF 2 , ..., PNF k ); the total number of VNFs is r, and the set of all VNFs is represented as VNF (VNF 1 , VNF 2 , ..., VNF r ).
  • the initial state of a PNF/VNF consists of four parts: the first part indicates the previous PNF/VNF serial number, The first PNF/VNF has a first bit of 0.
  • the second part indicates the PNF number where the PNF/VNF is located.
  • the second part of the first PNF/VNF is 1.
  • the number of bits in the first two parts will vary according to the number of PNFs/VNFs. For example, if there are 4 PNFs in total, 2 bits are used. If there are 10 PNFs, the number of bits needs to be 4.
  • the third part indicates the computing, storage, and network resources in the PNF/VNF. For PNF, 1 indicates that the computing/storage/network resources are available, and 0 indicates that they are used up.
  • VNF For VNF, 1 indicates that the computing/storage/network resources are available on the current PNF, and 0 indicates that they are used up.
  • the fourth part indicates the operating status of the PNF/VNF, which is determined by the availability defined above. When the calculated availability is higher than the set availability threshold, it is 1 for normal operation and 0 for abnormal operation. For example, when there are 2 PNFs and 3 VNFs, VNF1 and VNF2 are deployed on PNF1, and VNF3 is deployed on PNF2. They are all running normally and have sufficient resources. For each status value, the maximum value of the first two parts is 3 (3 VNFs), which occupies 2 bits respectively, the third part occupies 3 bits, and the fourth part occupies 1 bit.
  • the initial status of PNF is 00011111, 01101111 respectively.
  • the initial status of VNF is 00011111, 01011111, 10101111 respectively.
  • the initial status of PNF is 00011111, starting from the first bit: the first part 00 represents the serial number of the previous PNF; the second part 01 represents the serial number of the PNF; each bit in the third part 111 corresponds to the computing, storage, and network resources of VNF respectively, and the bit is 1, which means that the corresponding resources are available; the fourth part 1 indicates that the operation status of VNF is normal.
  • the action space of network slice deployment is the set of node deployment strategies that a node in the slice can choose.
  • There are k optional actions for deploying a node, and the action space can be expressed as As (A 1 , A 2 ,..., Ak ).
  • the action space is constrained by network resource limitations.
  • the node selection probability model is used to select a node deployment action.
  • the node selection probability model maps the distance between PNFs, the resource requirements of slices for VNF nodes, and the resource status on PNFs into the probabilities of possible selections, so that the reinforcement learning module can select the PNF node with the highest probability for deployment.
  • the input of the node selection probability model is a node state matrix X.
  • Each row of the matrix is the node state vector of each PNF node in, Indicates the selectability of PNF node i, which is determined by whether the previous hops have been selected and whether they are reachable. It is 1 if it is selectable and 0 if it is not selectable. Indicates the physical distance of the shortest available link from PNF node i to the previous PNF node. If it is the first node, then are all 1. If there is no link reachable from PNF node i to the previous PNF node, then All are -1. They represent the satisfaction degree of computing, storage and network resources on PNF node i respectively.
  • This patent uses a neural network to parameterize the node selection probability model, where W is the neuron weight parameter and b is the model bias parameter, both of which are obtained by prior model training.
  • the input of the resource allocation model is a resource evaluation vector in Indicates the minimum requirements for VNF computing, storage, and network resources. Indicates the available amount of computing, storage, and network resources on the PNF node. Indicates the priority of the network slice.
  • the resource model maps the resource evaluation vector into a resource allocation vector through a function.
  • the resource allocation vector can be expressed as: in, They represent the computing, storage, and network resources allocated by the PNF node to the VNF, respectively. In a specific implementation, appropriate parameters can be selected as parameters of the resource allocation vector according to actual needs.
  • the feedback function is used to judge the quality of an action and deployment strategy and give rewards.
  • the reinforcement learning model will update the strategy model based on the reward obtained from the last action selection. Selecting the slice deployment strategy with the highest reward is the goal of reinforcement learning.
  • the feedback function consists of three parts, including deployment accuracy, slice demand satisfaction, and link efficiency. Deployment accuracy indicates whether the selected VNF, PNF, and link can work properly. For example, when the resource usage of the selected node of the VNF exceeds the resource limit of the PNF, the deployment accuracy is 0. When the selected VNF and PNF are not running properly, the deployment accuracy is 0.
  • Slice demand satisfaction refers to the degree of satisfaction of resources allocated to the slice. Consider two situations. One is when the computing, storage, and network resources allocated to the slice are insufficient to meet the business requirements. The other is when the resources allocated to the slice can meet the requirements, but too many resources may be allocated, resulting in low efficiency of network resource utilization. Therefore, the slice demand satisfaction can be expressed by the following formula.
  • ⁇ represents the feedback parameter when the demand is not met
  • x represents the number of unsatisfied resources of a node
  • ⁇ represents the feedback parameter when the resources are over-allocated
  • ud represents the resource utilization of a node VNF
  • ad represents the resources allocated to this node.
  • Link efficiency reflects the distance between the PNFs where two adjacent VNFs are deployed.
  • the PNF nodes where two adjacent VNFs are located should be as close as possible. If the distance is too far, the latency will increase. Link efficiency can be expressed by the following formula.
  • ⁇ and ⁇ are the proportional coefficients of slice demand satisfaction and link efficiency respectively.
  • r is the feedback value, which is calculated by weighted sum of demand satisfaction, r SATISFACATION and link efficiency, r EFFICIENCY , and then multiplied by deployment accuracy r ACCURACY . It can be seen from the formula that the higher the slice demand satisfaction and link efficiency, the higher the feedback value of the slice deployment strategy.
  • the parameters in the feedback function can be fitted using a neural network.
  • the resources of each VNF in the first few network slices are proportionally reduced according to the priority ranking of the network slices, provided that the minimum requirement is not lower than that.
  • the reduced resource size is inversely proportional to the priority of the network slice.
  • the reduced resource size v can be specifically calculated according to the following formula.
  • i is the slice priority order
  • z is the resource size previously allocated
  • h is the minimum slice requirement
  • the parameter ⁇ can take values in (0,1].
  • ⁇ is the greater the degree of resource reduction.
  • ⁇ can be 0.5 to 1.
  • the deployment strategy is passed to the knowledge graph module for detection to determine whether the current network state can meet the generated network slicing decision. If not, the reinforcement learning module returns to the unsatisfied slicing step and makes a new deployment decision; if satisfied, the network state is updated and the next deployment decision is made based on the updated network state.
  • the reinforcement learning-based digital twin network slicing method provided in this embodiment introduces network operation status, resource utilization, and link efficiency when performing node selection and resource allocation, and can generate a reasonable deployment strategy for network slices while ensuring the network operation status.
  • the deployment strategy of network slices is dynamically adjusted according to the priority of network slices and the status of network resources, so as to ensure that each network slice and the entire network can operate stably.
  • Embodiment 2 is a diagrammatic representation of Embodiment 1:
  • the reinforcement learning-based digital twin network slicing method provided in Example 1 can be implemented through the specific implementation methods in this embodiment, so that the user's task requirements are converted into network slicing requirements through the knowledge graph module.
  • PNFs there are 6 PNFs in the network, and their computing, storage, and network resources are PNF 1 (100, 200, 95), PNF 2 (100, 60, 70), PNF 3 (200, 100, 80), PNF 4 (100, 200, 95), PNF 5 (200, 200, 200), and PNF 6 (100, 100, 100).
  • PNF 1 is connected to PNF 3 and PNF 6
  • PNF 2 is connected to PNF 4 and PNF 5 , and they are physically reachable.
  • Network slice 1 requires three The minimum computing, storage, and network requirements for each VNF are The slice priority is 1.
  • Network slice 2 requires 4 VNFs, and the computing, storage, and network requirements of each VNF are The slice priority is 2.
  • the superscript of the VNF indicates the network slice in which it is located, and the subscript is the sequence number of the VNF itself in each network slice.
  • the knowledge graph module builds the current network knowledge graph based on the previous network status and available resources, and uses The user's slice requirements and network knowledge graph information are passed to the reinforcement learning module in the form of triplets.
  • the network information triplets are: (PNF 1 , connection, PNF 3 ), (PNF 1 , computing, 100), (PNF 1 , storage, 100), (PNF 1 , network, 90), (PNF 2 , computing, 100), (PNF 2 , storage, 60), (PNF 2 , network, 70), (PNF 3 , computing, 200), (PNF 3 , storage, 100), (PNF 3 , network, 80), (PNF 4 , computing, 100), (PNF 4 , storage, 200), (PNF 4 , network, 95), and the remaining triplets can be established according to the same rules.
  • a set of slice requirement triplets for network slice 1 is: (slice 1, priority is,1), (slice 1, needs, ),(slice 1,need, ),(slice 1,need, ),( Calculate, 10), ( Storage, 10),), ( Network, 10), ( Calculate, 10), ( Storage, 20), ( Network, 10), ( Calculate, 20), ( Storage, 10), ( Network, 20).
  • a set of slice requirement triplets for network slice 2 are (slice 2, priority is,2), (slice 2, needs, ),(Slice 2,need, ),(Slice 2,need, ),(Slice 2,need, ),( Calculate, 20), ( Storage, 20),), ( Network, 20), ( Calculate, 20), ( Storage, 20), ( Network, 10), ( Calculate, 10), ( Storage, 10), ( Network, 20), ( Calculation, 30), ( Storage, 10), ( Network, 10).
  • the knowledge graph transmits the above network information triples and network slice requirement triples, as well as network information and network slice requests, to the reinforcement learning module.
  • the reinforcement learning module first makes slice deployment decisions for network slice 1 and network slice 2 respectively.
  • the node deployment constructs the node state matrix X.
  • Each row of the matrix is the node state vector of each PNF node in, Indicates the selectability of PNF node i, which is determined by whether the previous hops have been selected and whether they are reachable. It is 1 if it is selectable and 0 if it is not selectable. Indicates the physical distance of the shortest available link from PNF node i to the previous PNF node. If it is the first node, then Both are 1.
  • PNF 1 100, 90
  • PNF 2 100, 60, 70
  • X 2 [1, 1, 90, 60, 50].
  • the model first performs a weighted calculation on the node state matrix to obtain the evaluation score of each PNF node, and then Then, the probability of each PNF node being selected is calculated based on the evaluation score of each PNF.
  • the selection probability of each PNF is calculated to be [0.9, 0.7, ...], among which the probability of PNF 1 is the largest, 0.9, so it is selected.
  • the resource allocation model maps the resource evaluation vector to the resource allocation vector through a function. The mapping function is obtained in advance through neural network fitting.
  • the calculated output is (85,85,80), which is The computing, storage, and network resources are allocated to 85, 85, and 80 respectively.
  • the amount of resources allocated depends on the resource allocation model. If the model is well trained, the allocation result will be more reasonable and no excessive adjustments will be required in the future. If the model is not well trained, the most reasonable sharding cannot be obtained and further adjustments will be required in the future.
  • the environmental knowledge graph After completing the deployment of a node, the environmental knowledge graph will be updated, such as the available resources of PNF 1 are reduced to (15, 15, 10).
  • the deployment accuracy, slice demand satisfaction and link efficiency of the current slice are calculated according to the calculation formula mentioned above, and then the reward of this network slice resource allocation action is calculated comprehensively according to the feedback function formula.
  • the deployment accuracy is calculated to be 1.
  • This node is the first node, so the link efficiency is calculated to be 1.
  • the parameters of the node selection probability model are updated once using the gradient descent method in combination with the feedback value. Since this network slice resource allocation action successfully allocated resources to a network slice and obtained a reward, it shows that the mapping direction of the resource allocation model is correct this time. Therefore, the result of the parameter update is to encourage the model to continue to optimize in this mapping direction.
  • the node state matrix X is constructed based on the node deployment. Assume that there are PNF 3 (200,100,80), PNF 4 : (100,200,95), the physical distance from PNF 3 to PNF 1 is 3, and PNF 4 is unreachable to PNF 1.
  • the node state matrix is constructed, it is input into the node selection probability model to calculate the selection probability of each PNF as [...0.8,0.6,...], where the maximum probability of PNF 3 is 0.8, so PNF 3 is selected. Deployed on PNF 3.
  • construct the resource evaluation vector x r (10, 20, 10, 200, 100, 80) and input it into the resource allocation model to calculate the output (150, 80, 50), then The computing, storage, and network resources are allocated as 150, 80, and 50 respectively.
  • the environmental knowledge graph is updated, such as the available resources of PNF 3 are reduced to (50, 20, 30). Then the reward is calculated based on the feedback function to update the parameters of the node selection probability model. The calculation is as follows:
  • w is the model parameter
  • ⁇ is the update coefficient
  • r is the feedback function value
  • the resource allocation model allocates computing, storage, and network resources to it as 60, 50, and 60, respectively, completing the deployment decision for the first network slice request.
  • the reinforcement learning module After completing the slice deployment decision for network slice 1, the reinforcement learning module begins to make the deployment decision for the second network slice.
  • the resource allocation adjustment formula (9) to calculate the required reduction of resources (42,42,40) according to the priority of the first network slice.
  • the resource allocation of PNF 1 is reduced from (85,85,80) to (43,43,40). Now the available resources on PNF 1 change from (15,15,10) to (57,57,50), which can satisfy So we continue to follow the previous steps to make deployment decisions for the second network slice.
  • the deployment decisions for the two slices are: S1: (PNF 1 , PNF 3 , PNF 5 ), S2: (PNF 1 ,PNF 4 ,PNF 7 ,PNF 9 ),
  • the deployment decision is passed to the knowledge graph module, and the knowledge graph module is updated.
  • the knowledge graph transmits the updated network information back to the reinforcement learning module, obtains the mean time to failure and the mean time to repair of each PNF from the network information, and calculates the availability of each PNF according to formula (1). Assuming that the calculated availability of PNF 1 is 0.99, the availability of PNF 2 is 0.98, the availability of PNF 3 is 0.96, the availability of PNF 4 is 0.97, and the availability of PNF 6 is 0.98, and the set availability threshold is 0.95, so all PNFs are available. Then, based on the availability of PNF and the resource conditions of PNF and VNF, the network operation status of the slice can be obtained.
  • the PNF status of slice S1 is 0000001111, 0000101111, 0101001111
  • the VNF initial status of slice S1 is 000001111, 010101111, 101001111
  • the PNF status of slice S2 is 0000011111, 0010111111, 0111011111
  • the VNF initial status of slice S2 is 000011111, 010111111, 101011111, 111011111. Therefore, the network status is evaluated as normal, and then the deployment decision information is transmitted to the control module for issuance to complete the user's network slicing request.
  • Example 1 It can be seen from the above specific examples that the method for digital twin network slicing based on reinforcement learning provided in Example 1 can quickly and effectively complete the digital twin network slicing, better ensure the network resources. Efficient use of resources and ensuring that the network status remains normal when the network changes dynamically.
  • Embodiment 3 is a diagrammatic representation of Embodiment 3
  • the present invention also provides a device for digital twin network slicing based on reinforcement learning that can be used to implement the above-mentioned method, as shown in the figure, which is a schematic diagram of the device architecture of an embodiment of the present invention.
  • the device for digital twin network slicing based on reinforcement learning of this embodiment includes one or more processors 11 and a memory 12. Among them, one processor 11 is taken as an example in FIG6.
  • the processor 11 and the memory 12 may be connected via a bus or other means, and FIG6 takes the connection via a bus as an example.
  • the memory 12 is a non-volatile computer-readable storage medium for a method of digital twin network slicing based on reinforcement learning, and can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the method of digital twin network slicing based on reinforcement learning in Examples 1 and 2.
  • the processor 11 executes various functional applications and data processing of the device of digital twin network slicing based on reinforcement learning by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, implementing the method of digital twin network slicing based on reinforcement learning in Examples 1 and 2.
  • the memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device.
  • the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the processor 11 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • the program instructions/modules are stored in the memory 12, and when executed by one or more processors 11, the reinforcement learning-based digital twin network slicing method in the above-mentioned embodiments 1 to 2 is executed, for example, the various steps shown in Figures 2 and 3 described above are executed.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Software Systems (AREA)
  • Animal Behavior & Ethology (AREA)
  • Medical Informatics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

本发明涉及网络通信领域，特别是涉及一种基于强化学习的数字孪生网络切片的方法和装置。主要包括：强化学习模块获取知识图谱中当前网络信息三元组和网络切片需求三元组；强化学习模块生成至少一个网络切片的决策动作，每个决策动作包含根据节点选择概率模型和知识图谱查找PNF节点以及根据资源分配模型对每个PNF节点分配给VNF的资源映射为资源分配向量，知识图谱根据决策动作进行决策以获得决策结果，决策结果为网络切片VNF与网络切片PNF的部署位置；知识图谱根据决策结果进行更新，并将更新后的网络状态传回给强化学习模块，强化学习模块下发网络切片的部署策略。本发明可以在动态网络环境下快速获得当前网络场景中最优的切片部署方案。

Description

一种基于强化学习的数字孪生网络切片的方法和装置
相关申请的交叉参考
本申请要求于如下六件专利申请的优先权：
(1)、2023年1月17日提交中国专利局，申请号为202310089923.8，名称为“一种基于强化学习的数字孪生网络切片的方法和装置”的中国专利申请。
技术领域
本发明涉及网络通信领域，特别是涉及一种基于强化学习的数字孪生网络切片的方法和装置。
背景技术
随着5G和云技术的发展，越来越多的新种类业务逐渐涌现出来，而不同的业务或用户对网络的服务质量要求不尽相同。为了满足用户们的多样化需求，提供个性化差异服务，网络切片技术应运而生。如图1所示，网络切片技术可以在一张共享的物理网络上灵活调配资源，按需构建多个专用的、互相隔离的虚拟化逻辑网络，即多个网络切片，每个网络切片用以满足特定的业务需求。网络切片是通过网络虚拟化技术实现的，由一组满足业务需求的虚拟网络功能(Virtual Network Function,简写为VNF)以及连接每两个虚拟网络功能的链路组成。对网络进行切片可以看作是基于共享的网络基础设施进行VNF以及链路的资源分配过程。如何对有限的资源进行灵活高效的分配，以实现各个网络切片的服务需求是网络切片的热点研究问题。
当前网络切片的部署大多仅考虑节点资源能力大小，没有综合考虑资源能力、链路长度以及物理网络功能(physical network function，简写为PNF)运行状态，难以保证不同网络需求和网络资源的高效利用。并且，物理网络资源是在时刻变化的，并且同时会产生多个网络切片需求，当前的网络切片的部署和编排方法大多没有考虑到物理网络状态的动态变化以及多网络切片需求下切片部署策略的调整。
鉴于此，如何克服现有技术所存在的缺陷，解决网络切片时策略调整困难的现象，是本技术领域待解决的问题。
申请内容
针对现有技术的以上缺陷或改进需求，本发明解决了网络切片时策略调整困难的问题。
本发明实施例采用如下技术方案：
第一方面，本发明提供了一种基于强化学习的数字孪生网络切片的方法，具体为：强化学习模块获取知识图谱中当前网络信息三元组和网络切片需求三元组，每个网络切片对应一组网络切片需求三元组，其中，当前网络信息三元组包括网络中所有PNF连接关系和每个PNF的资源属性，网络切片需求三元组包括切片需求的所有VNF连接关系、每个VNF的资源需求和切片优先级；强化学习模块生成至少一个网络切片的决策动作，每个决策动作包含根据节点选择概率模型和知识图谱查找PNF节点以及根据资源分配模型对每个PNF节点分配给VNF的资源映射为资源分配向量，知识图谱根据决策动作进行决策以获得决策结果，决策结果为网络切片VNF与网络切片PNF的部署位置；知识图谱根据决策结果进行更新，并将更新后的网络状态传回给强化学习模块，强化学习模块确认网络状态正常后，下发网络切片的部署策略。
优选的，所述强化学习模块生成至少一个网络切片的决策动作，具体包括：根据切片需求，基于知识图谱，找到节点选择概率模型中概率最大的PNF作为第一个节点；基于知识图谱得到资源评估向量，输入到资源分配模型中为第一VNF节点分配资源，并更新知识图谱；根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到可能的下一个VNF节点的PNF位置选择，依次查找直至所有节点都部署分配完毕。
优选的，所述找到节点选择概率模型中概率最大的PNF作为第一个节点，具体包括：找到第一个VNF节点的动作空间中所有可能的PNF位置选择作为第一VNF节点的部署动作空间，并得到节点状态矩阵；将节点状态矩阵输入到节点选择概率模型中，计算每个节点被选择的概率，选择其中概率最大的PNF作为第一个节点；基于知识图谱得到第一个节点的资源评估向量，将评估向量输入到资源分配模型中，为第一VNF节点分配资源。
优选的，所述找到可能的下一个VNF节点的PNF位置选择，具体包括：根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到所有可能的下一个VNF节点的PNF位置选择，作为下一个VNF节点的部署动作空间；基于知识图谱得到节点状态矩阵，依次向后进行下一个节点的部署和资源分配，直至所有节点都部署分配完毕，以所有节点的分配结果完成一整个切片部署策略的选择。
优选的，所述知识图谱根据决策结果进行更新，具体包括：知识图谱根据强化学习的资源分配策略进行更新，并根据反馈函数计算反馈值；结合反馈值利用梯度下降法更新节点选择概率模型的参数。
优选的，所述节点选择概率模型具体包括：将PNF之间的距离、切片对于VNF节点的资源需求和PNF上的资源状态映射成可能选择的概率，以便于选择强化学习模块获取概率最大的PNF节点进行部署。
优选的，所述资源分配模型具体包括：通过函数将资源评估向量映射成资源分配向量，以便于表示PNF节点分配给VNF的计算、存储、网络资源。
优选的，所述强化学习模块确认网络状态正常，还包括：当网络切片请求在进行某个节点部署，由于之前的网络切片占用过多资源而无法找到资源满足需求的PNF时，根据网络切片的优先级排序，在不低于最低需求的条件下，按比例缩小前几个网络切片中每个VNF的资源，缩小的资源大小与网络切片的优先级成反比。
优选的，所述强化学习模块确认网络状态正常，还包括：当网络状态发生改变，知识图谱模块中当前时刻的网络状态无法满足原时刻的切片需求时，则强化学习模块退回到不满足的切片步骤重新进行部署决策。
另一方面，本发明提供了一种基于强化学习的数字孪生网络切片的装置，具体为：包括至少一个处理器和存储器，至少一个处理器和存储器之间通过数据总线连接，存储器存储能被至少一个处理器执行的指令，指令在被处理器执行后，用于完成第一方面中的基于强化学习的数字孪生网络切片的方法。
与现有技术相比，本发明实施例的有益效果在于：使用强化学习模块生成网络切片部署策略，在动态网络环境下快速完成时间切片部署方案的决策，获得当前网络场景中最优的切片部署方案，更好地保证网络资源的高效利用。
附图说明
为了更清楚地说明本发明实施例的技术方案，下面将对本发明实施例中所需要使用的附图作简单地介绍。显而易见地，下面所描述的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动的前提下，还可以根据这些附图获得其他的附图。
图1为执行网络切片的网络结构模型示意图；
图2为本发明实施例提供的一种基于强化学习的数字孪生网络切片的方法流程图；
图3为本发明实施例提供的另一种基于强化学习的数字孪生网络切片的方法流程图；
图4为本发明实施例提供的另一种基于强化学习的数字孪生网络切片的方法流程图；
图5为本发明实施例提供的强化学习模块架构示意图；
图6为本发明实施例提供的一种基于强化学习的数字孪生网络切片的装置结构示意图。
具体实施方式
为了使本发明的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本发明进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本发明，并不用于限定本发明。
本发明是一种特定功能系统的体系结构，因此在具体实施例中主要说明各结构模组的功能逻辑关系，并不对具体软件和硬件实施方式做限定。
此外，下面所描述的本发明各个实施方式中所涉及到的技术特征只要彼此之间未构成冲突就可以相互组合。下面就参考附图和实施例结合来详细说明本发明。
实施例1：
强化学习(Reinforcement Learning，简写为RL)是一种人工智能技术，主要思想是基于和环境的持续交互，获得反馈，并据此不断优化决策。本专利通过结合知识图谱和强化学习来实现网络切片部署决策，可以更好地保证网络资源的高效利用。
针对以上问题，本专利所描述的方法通过一种基于强化学习的网络切片部署策略生成方法实现网络切片，生成的网络切片的部署策略可以更好地保证每个网络切片以及整个网络的运行状态。
如图2所示，本发明实施例提供的基于强化学习的数字孪生网络切片的方法具体步骤如下：
步骤101：强化学习模块获取知识图谱中当前网络信息三元组和网络切片需求三元组，每个网络切片对应一组网络切片需求三元组。
强化学习模块的输入是来自知识图谱的，知识图谱由“实体-关系-实体”三元组以及实体及其相关属性-值对组成，三元组可表示为(Thing1，Relation，Thing2)，Thing1表示第一个实体，Thing2表示第二个实体，Relation表示Thing1和Thing2之间的关系。本实施例的方法用于根据当前网络状态计算网络切片的部署策略，因此需要使用网络信息三元组和网络切片需求三元组。当前网络信息三元组包括网络中所有PNF连接关系，和每个PNF的资源属性，资源属性包括每个PNF的计算、存储、网络等已确定的硬件资源属性或可使用的资源属性。网络信息三元组中，Thing1表示PNF，Relation表示该三元组的 资源类型，Thing2表示Thing1所需的该类型资源的值。例如在某个具体场景中，某个PNF(表示为PNF1)需要的计算、存储、网络资源分别为100、200、300，PNF1与PNF2之间存在连接关系，该PNF可表示为：(PNF1，连接，PNF2)、(PNF1，计算，100)、(PNF1，存储，200)、(PNF1，网络，300)。网络切片需求三元组包括切片需求的所有VNF连接关系、每个VNF的资源需求和切片优先级。每个网络切片对应一组网络切片需求三元组。网络切片需求三元组中，Thing1表示网络切片或VNF，Relation表示该三元组的资源类型，Thing2表示Thing1所需的该类型资源的值。进一步的，当Thing1为网络切片时，Relation的类型为“所需的VNF”或优先级等，Thing2为Thing1需要的VNF或优先级值，例如，网络切片1中需要VNF1，三元组为(切片1，需要，VNF1)。当Thing1为VNF时，与网络信息三元组类似，Relation为VNF分配的计算、存储、网络等资源类型，Thing2为该资源类型的值。例如，VNF1的计算资源分配值为100，VNF1的计算资源三元组为(VNF1，计算，100)。在实际使用中，强化学习模块同时会收到多个网络切片的需求。
步骤102：强化学习模块生成至少一个网络切片的决策动作，每个决策动作包含根据节点选择概率模型和知识图谱查找PNF节点以及根据资源分配模型对每个PNF节点分配给VNF的资源映射为资源分配向量，知识图谱根据决策动作进行决策以获得决策结果，决策结果为网络切片VNF与网络切片PNF的部署位置。
强化学习模块的输出是网络切片的部署策略，即一个网络切片中每个VNF部署在哪个PNF上，每个VNF被分配多少资源。强化学习模块的输出会传到知识图谱，知识图谱进行更新。
步骤103：知识图谱根据决策结果进行更新，并将更新后的网络状态传回给强化学习模块，强化学习模块确认网络状态正常后，下发网络切片的部署策略。
步骤102中，通过强化学习模块的决策，网络切片需要的VNF会改变，每个VNF所需的计算、存储、网络等资源也会出现变化，因此，需要根据决策后的数据对知识图谱中的三元组进行相应更新。例如，网络切片1中原本只需要VNF1，表示为(切片1，需要，VNF1)，决策后变化为需要VNF1和VNF2，三元组更新为(切片1，需要，VNF1)、(切片1，需要，VNF2)。又例如，VNF1的计算资源分配值由100变为150，相应的三元组也需要由(VNF1，计算，100)更新为(VNF1，计算，150)。
本实施例提供的方法中，将一个网络切片需求转化为多个VNF的分配需求，每个VNF的分配过程，都是先确认分配在哪个PNF上，然后再根据需求分配相应的资源。每分配完成一个VNF都会出现一次三元组的变化，需要根据新的三元组对下一个VNF进行却侧，因此，每次分配完成一个VNF后，均有一次与环境知识图谱的交互。
知识图谱将更新后的网络状态传回强化学习模块，强化学习模块对调整后的网络状态做出评估，确认使用该切片策略后网络状态正常后，将部署决策信息传至控制模块进行下发。
经过本实施例中提供的步骤101-步骤103后，即可根据当前网络状态，使用强化学习模块完成网络切片的部署决策最优化选择，并对决策结果进行确认，确保执行新的切片策略后网络状态正常。
基于强化学习进行网络切片部署决策主要由以下几部分组成：环境、动作、状态、奖励和决策模型。强化学习的环境为网络知识图谱。状态是指网络当前的状态。动作是强化学习在当前状态下做出的一个决策，即一个网络切片的节点选择方案和资源分配方案。奖励是指强化学习做出一个动作后影响环境，环境所给的反馈，反映出切片部署决策的好坏。决策模型在本专利中包括节点选择概率模型和资源分配模型，强化学习根据决策模型产生一个切片部署的决策动作。
如图3所示，可以使用以下步骤完成强化学习模块生成网络切片的部署策略。
步骤201：根据切片需求，基于知识图谱，找到节点选择概率模型中概率最大的PNF作为第一个节点。
根据切片需求，基于知识图谱，找到第一个VNF节点的动作空间中所有可能的的PNF位置选择作为第一VNF节点的部署动作空间，并得到节点状态矩阵。将节点状态矩阵输入到节点选择概率模型中，计算每个节点被选择的概率，选择其中概率最大的PNF作为第一个节点。再基于知识图谱得到第一个节点的资源评估向量，将评估向量输入到资源分配模型中，为第一VNF节点分配资源。
步骤202：基于知识图谱得到资源评估向量，输入到资源分配模型中为第一VNF节点分配资源，并更新知识图谱。
完成一个节点的部署动作后，知识图谱根据强化学习的资源分配策略进行更新，并根据反馈函数计算反馈值。
在进行更新时，还需要结合反馈值利用梯度下降法更新节点选择概率模型的参数。具体的计算公式如下。
该公式表示，节点选择概率模型中，每个节点的选择概率根据反馈值进行梯度下降后，当前的节点选择概率模型。
步骤203：根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到可能的下一个VNF节点的PNF位置选择，依次查找直至所有节点都部署分配完毕。
完成第一个VNF节点选择后，还需要依次向后获取所有VNF节点的PNF位置。根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到所有可能的下一个VNF节点的PNF位置选择，作为下一个VNF节点的部署动作空间。再基于知识图谱得到节点状态矩阵，重复前面步骤，依次向后进行下一个节点的部署和资源分配，直至所有节点都部署分配完毕，以所有节点的分配结果完成一整个切片部署策略的选择。
在实际决策过程中，若在进行第k个节点的部署策略选择时，发现没有可供选择的节点位置选项，则退回到第k-1个节点，选择概率第二大的选项再进行下一步。
当需要进行多个网络切片时，对一个网络切片请求生成部署策略后，进行下一个网络切片的部署决策。
经过本实施例中提供的步骤201-步骤203后，即可完成步骤102中网络切片的决策，获取到所有VNF和PNF的部署策略。与前述步骤相结合后，网络切片和调整的完整过程如图4所示。
进一步的，根据上述步骤完成所有网络切片部署决策后，还需要将部署决策传到知识图谱模块，对知识图谱模块进行更新。知识图谱再将更新后的网络状态传回强化学习模块。通过循环反馈的过程提高决策准确度，并使知识图谱中的对应关系能够在网络环境出现动态变化时与当前环境相符。
在实际实施场景中，可以通过以下方式构建强化学习模型，并完成部署和反馈。具体实施中，可以使用如图5所示的强化学习模块。
网络切片的网络运行状态具体包括网络切片PNF和网络切片VNF的位置、资源和关系信息。本实施例中，使用以下数学模型对网络运行状态进行描述： 表示物理集群层中的PNF。表示物理集群层中的PNF与PNF之间的物理链路，例如，表示切片中与之间的物理链路。网络中存在的网络切片用S＝{S1,S2,…,Sb}表示，总计有b个网络切片。在一个网络切片Ss中，表示VNF节点，表示VNF节点之间的虚拟链路，例如表示切片中与之间的虚拟链路。对于的计算、存储、网络资源分别表示为切片中VNF节点的计算、存储、网络资源分别表示为
每个PNF或VNF可靠性由失效平均时间(Mean Time To Failure，简写为MTTF)衡量，可服务性由平均修复时间(Mean Time To Restoration，简写为MTTR)衡量，则PNF(VNF)的可用性APNF(AVNF)可以使用以下公式表示。
整个切片的可用性等于构成切片PNF和VNF的可用性最小值，用于评估网络运行状态。
VNF和PNF的位置关系用可以用以下公式表示：
上述公式表示：如果在上，则否则为0。
和的计算、存储和网络资源应满足以下条件：
上述公式表示：根据PNF、VNF可用性以及资源条件定义网络运行态，具体如下：对网络切片时，选择VNF并选择所部署的PNF，按照空间位置关系对PNF排序。
本实施例的一个具体实施场景中，PNF的总数为k个，所有PNF的集合表示为PNF(PNF1,PNF2,…,PNFk)；VNF的总数为r个，所有VNF的集合表示为VNF(VNF1,VNF2,…,VNFr)。
PNF/VNF初始状态由四部分组成：第一部分表示前一个PNF/VNF序号， 第一个PNF/VNF首位为0。第二部分表示PNF/VNF所在PNF序号，第一个PNF/VNF第二部分表示为1。前两部分的比特数会根据PNF/VNF的数量大小变化，例如总共有4个PNF，则用2个比特表示，若有10个PNF，比特数则需要为4。第三部分表示PNF/VNF中计算、存储、网络资源。对于PNF，1表示计算/存储/网络资源有剩余，0表示已用完；对于VNF，1表示在当前PNF上计算/存储/网络资源有剩余，0表示已用完。第四部分表示，PNF/VNF的运行状态，通过上面定义的可用性来判断，当计算得到的可用性高于设定的可用性阈值时，则是正常运行为1，非正常运行为0。例如，当有2个PNF、3个VNF时，其中VNF1，VNF2部署在PNF1上，VNF3部署在PNF2上，均运行正常，资源充足。对于每个状态值，前两部分最大值为3(3个VNF)分别占用2个bit，第三部分占用3个bit，第四部分占用1个bit。则PNF初始状态表示分别为00011111,01101111。VNF初始状态分别为00011111,01011111,10101111。例如，PNF初始状态00011111，自首位开始：第一部分00表示前一个PNF的序号；第二部分01表示所在PNF的序号；第三部分111中的每一位分别对应VNF的计算、存储、网络资源，该位为1表示对应资源有剩余；第四部分1表示VNF的运行状态正常。
强化学习模块中，网络切片部署的动作空间即切片中一个节点可以选择的节点部署策略集合。一个动作表示一个节点部署策略，可以表示为A＝(a1,a2)。其中a1表示选择一个PNF节点，a2表示该节点分配的资源数量。部署一个节点的可选的动作有k个，则动作空间可以表示为As＝(A1,A2,…,Ak)。动作空间会受到网络资源限制的约束。
节点选择概率模型用于选择一个节点部署动作。节点选择概率模型将PNF之间的距离、切片对于VNF节点的资源需求和PNF上的资源状态映射成可能选择的概率，以便于选择强化学习模块获取概率最大的PNF节点进行部署。
节点选择概率模型的输入是一个节点状态矩阵X。矩阵的每一行为每个PNF节点的节点状态向量其中，表示PNF节点i的可选性，根据前几跳是否被选择过以及是否可达来判断，可选为1，不可选为0。表示PNF节点i到上一个PNF节点的可用的最短链路的物理距离，若为第一个节点，则都为1，若PNF节点i到上一个PNF节点没有链路可达，则都为-1。分别表示PNF节点i上计算、存储和网络资源对需求的满足程度，通过计算PNF节点i上的可用资源与切片对该节点的最小资源需求的 差值得到。假设一个节点的资源需求是(10,10,10)，PNFi的可用资源为(100,100,90)，前几跳没有被选用过，PNFi到上一个PNF节点的物理距离为3，则PNFi对应的节点状态向量表示为Xi＝[1,3,90,90,80]。
节点概率选择模型先对节点状态矩阵进行加权计算：Y＝WX+b得到每一个PNF节点是否可以作为下一个PNF节点的评价分数Y＝(y1,y2,…yn)，然后根据评价分数计算每个PNF节点作为被选择的概率本专利使用神经网络来参数化节点选择概率模型，其中W为神经元权重参数，b为模型偏置参数，均由事先的模型训练得到。
资源分配模型的输入是一个资源评估向量其中表示VNF计算、存储、网络资源的最小需求，表示PNF节点上计算、存储、网络资源的可用量,表示网络切片的优先级。资源模型通过函数将资源评估向量映射成资源分配向量，资源分配向量可以表示为：其中，分别表示PNF节点分配给VNF的计算、存储、网络资源，在具体实施中，可以根据实际需要选择合适的参数作为资源分配向量的参数。资源分配模型表示为yr＝f(xr)，映射函数f(·)可以通过神经网络拟合。
反馈函数用于评判一个动作及部署策略的好坏，给出奖励。强化学习模型在选择策略动作时，会根据上次动作选择得到的奖励更新策略模型。选择奖励最高的切片部署策略就是强化学习的目标。反馈函数由三部分组成，包括部署精度、切片需求满足度和链路效率。部署精度表示所选择的VNF、PNF和链路是否能正常工作。例如，当VNF选取节点资源使用已超过所在PNF资源限制时，部署精度为0，当选取的VNF、PNF运行不正常时，部署精度为0。
切片需求满足度指分配给切片的资源满足程度。考虑两种情况，一种情况是当分配给切片的计算、存储和网络资源不足以满足业务要求。另一种情况是分配给切片的资源可以满足要求，但可能分配的资源过多导致网络资源利用效率低。因此切片需求满足程度可以由以下公式表示。
其中，α表示不满足需求时的反馈参数，x表示一个节点不满足的资源的数量，γ表示资源分配过多时的反馈参数，ud表示一个节点VNF的资源利用率，ad表示分配给这个节点的资源。
链路效率反应的是两个相邻的VNF所部署的PNF的距离远近。两个相邻VNF所在PNF节点应该尽可能相邻，若距离过远时延会增加。链路效率可以用以下公式表示。
综合三部分后反馈函数可以定义为：
r＝rACCURACY(ηrSATISFACATION+ξrEFFICIENCY)    (8)
其中，η，ξ分别为切片需求满足度和链路效率的比例系数。r为反馈值，计算方式为需求满足度与rSATISFACATION和链路效率rEFFICIENCY加权求和，再与部署精度rACCURACY相乘。由该公式可见，当切片需求满足度和链路效率越高，该切片部署策略的反馈值越高。反馈函数中的参数可以用神经网络进行拟合。
使用上述强化学习模块，并配合相应的反馈函数，即可根据获取到的当前网络状态，完成网络切片中每一个VNF和PNF的部署。
进一步的，对于数字孪生强化学习模块下多网络切片，可能会面临网络资源冲突的情况，此时需要进行策略调整。以下提供两种常见情况下的策略调整方式，在具体实施场景中，可以根据实际需要参考以下方式进行策略调整。
(1)当网络切片请求在进行某个节点部署，由于之前的网络切片占用过多资源而无法找到资源满足需求的PNF时，根据网络切片的优先级排序，在不低于最低需求的条件下，按比例缩小前几个网络切片中每个VNF的资源，缩小的资源大小与网络切片的优先级成反比。缩小的资源大小v具体可以按如下公式计算。
其中，i为切片优先级顺序，z为之前被分配到的资源大小，h为切片最低需求，参数θ可以在(0,1]中取值。θ越大，资源缩减程度越大。优选方案中，当网络中切片需求较多时，θ可以取0.5～1。
(2)当网络状态发生改变，知识图谱模块中当前时刻的网络状态无法满足原时刻的切片需求时，为了避免此资源冲突情况的发生，在完成单个网络切片部署策略生成后，就将部署策略传给知识图谱模块进行检测，判断当前时刻的网络状态能否满足已经生成的网络切片决策。如果不满足，则强化学习模块退回到不满足的切片步骤重新进行部署决策；如果满足，则更新网络状态，基于更新后的网络状态继续进行下一次部署决策。
通过上述策略调整方式，可以在决策失误时快速有效的进行调整，确保下发的部署策略能够维持网络状态正常。
本实施例提供的基于强化学习的数字孪生网络切片的方法，在进行节点选择和资源分配时引入了网络运行状态、资源利用率以及链路效率，可以为网络切片生成合理的部署策略的同时保证网络运行状态。在多网络切片需求下，根据网络切片的优先级和网络资源状态对网络切片的部署策略进行动态调整，以此保证每个网络切片和整个网络能稳定运行。
实施例2：
基于实施例1提供的基于强化学习的数字孪生网络切片的方法，在不同的具体应用场景中，在某些具体实施方式中，可以通过本实施例中的具体实施方式对实施例1中提供的基于强化学习的数字孪生网络切片方法进行实现，使用户的任务需求通过知识图谱模块转换成网络切片需求。
在某个具体实施场景中，网络中存在6个PNF，它们的计算、存储、网络资源分别为PNF1(100，200，95)，PNF2(100，60，70)，PNF3(200，100，80)，PNF4(100，200，95)，PNF5(200，200，200)，PNF6(100，100，100)，其中PNF1和PNF3、PNF6有连接关系，PNF2和PNF4、PNF5有连接关系，物理可达。
用户发起两个切片请求：网络切片1需要三个每个VNF的计算、存储、网络最低需求分别为切片优先级为1。网络切片2需要4个VNF，每个VNF的计算、存储、网络需求分别为切片优先级为2。其中，VNF的上标表示其所在的网络切片，下标为每个网络切片中VNF自身的序号。
知识图谱模块根据前网络状态及可用资源构建当前网络知识图谱，并将用 户的切片需求与网络知识图谱信息以三元组的形式传递给强化学习模块。其中，网络信息三元组为：(PNF1，连接，PNF3)，(PNF1，计算，100)，(PNF1，存储，100)，(PNF1，网络，90)，(PNF2，计算，100)，(PNF2，存储，60)，(PNF2，网络，70)，(PNF3，计算，200)，(PNF3，存储，100)，(PNF3，网络，80)，(PNF4，计算，100)，(PNF4，存储，200)，(PNF4，网络，95)，其余三元组可依同样的规则建立。
网络切片1的一组切片需求三元组为：(切片1，优先级是，1)，(切片1，需要，)，(切片1，需要，)，(切片1，需要，)，(计算，10)，(存储，10)，)，(网络，10)，(计算，10)，(存储，20)，(网络，10)，(计算，20)，(存储，10)，(网络，20)。
网络切片2的一组切片需求三元组有(切片2，优先级是，2)，(切片2，需要，)，(切片2，需要，)，(切片2，需要，)，(切片2，需要，)，(计算，20)，(存储，20)，)，(网络，20)，(计算，20)，(存储，20)，(网络，10)，(计算，10)，(存储，10)，(网络，20)，(计算，30)，(存储，10)，(网络，10)。
按照步骤101，知识图谱将上述网络信息三元组和网络切片需求三元组，以及网络信息和网络切片请求，传到强化学习模块。强化学习模块依次分别先进行网络切片1和网络切片2的切片部署决策。
按照步骤102，基于知识图谱，先为的节点部署构建节点状态矩阵X。矩阵的每一行为每个PNF节点的节点状态向量其中，表示PNF节点i的可选性，根据前几跳是否被选择过以及是否可达来判断，可选为1，不可选为0。表示PNF节点i到上一个PNF节点的可用的最短链路的物理距离，若为第一个节点，则都为1。分别表示PNF节点i上计算、存储和网络资源对需求的满足程度，通过计算PNF节点i上的可用资源与切片对该节点的最小资源需求的差值得到。假设有PNF1:(100,100,90),PNF2:(100,60,70)，对于第一个VNF都可选，所以对应的节点状态向量分别为X1＝[1,1,90,90,80,],X2＝[1,1,90,60,50]。构建好节点状态矩阵后，输入到节点选择概率模型中。节点概率选择模型是一个神经网络模型。模型首先对节点状态矩阵进行加权计算，得到每一个PNF节点的评价分数，然 后根据每个PNF的评价分数大小计算每个PNF节点被选择的概率。计算得到每个PNF的选择概率为[0.9,0.7,…]，其中PNF1的概率最大为0.9，所以选择部署在PNF1上。然后，构建资源评估向量为xr＝(10,10,10,100,100,90)输入到资源分配模型中计算需要分配的资源。资源分配模型通过函数将资源评估向量映射成资源分配向量，映射函数是事先通过神经网络拟合得到。计算得到输出为(85,85,80)，则为分配计算、存储、网络资源分别为85、85、80。分配的资源是多少取决于资源分配模型，若模型训练得较好，得到的分配结果会更合理，后续不需要过多的调整；若模型训练的不够好，则无法得到最合理的分片，后续还需要进一步进行调整。
完成一个节点的部署动作后，环境知识图谱会进行更新，如PNF1的可用资源减少为(15,15,10)。接下来，根据前文提到的计算公式分别计算当前切片的部署精度、切片需求满足度以及链路效率，再根据反馈函数公式综合计算得到此次网络切片资源分配动作的奖励。其中，由于该节点部署成功，所以部署精度计算为1。切片需求满足度，假设反馈系数为2，根据节点需求和被分配的资源结合公式(5)可计算得2x(10/85+10/85+10/80)/3＝0.24。
该节点为第一个节点，所以链路效率计算为1。假设切片需求满足度和链路效率的比例系数分别为0.5，则根据公式(7)计算可得反馈奖励值为1x(0.5*0.24+0.5*1)＝0.62。然后，结合反馈值利用梯度下降法更新一遍节点选择概率模型的参数。由于此次网络切片资源分配动作成功为一个网络切片分配资源并得到奖励，表明资源分配模型此次映射的方向是正确的，因此参数更新的结果是鼓励模型继续朝这个映射方向优化。
接下来根据第一个VNF节点选取的位置，PNF1，来为的节点部署构建节点状态矩阵X。假设有PNF3(200,100,80),PNF4:(100,200,95)，PNF3到PNF1的物理距离是3，PNF4到PNF1不可达，则对应的节点状态向量分别为X3＝[1,3,190,80,70,],X4＝[0,-1,90,180,85]。构建好节点状态矩阵后，输入到节点选择概率模型中计算得到每个PNF的选择概率为[...0.8,0.6,…]，其中PNF3的概率最大为0.8，所以选择部署在PNF3上。然后，构建资源评估向量为xr＝(10,20,10,200,100,80)输入到资源分配模型中计算得到输出为(150,80,50)，则为分配计算、存储、网络资源分别为150、80、50。
完成第二个节点的部署动作后，环境知识图谱进行更新，如PNF3的可用资源减少为(50,20,30)。再根据反馈函数计算奖励更新节点选择概率模型的参数。 计算方式如下：
其中w为模型参数，η为更新系数，r为反馈函数值。
重复上述操作进行第三个节点的部署，节点选择概率模型选择部署在PNF6上，资源分配模型为其分配计算、存储、网络资源分别为60、50、60，即完成第一个网络切片请求的部署决策。完成网络切片1的切片部署决策后，强化学习模块开始进行第二个网络切片的部署决策。当为第二个网络切片的第一个节点进行部署决策时发现只有PNF1可选，但PNF1由于在第一个网络切片时为分配过多资源导致资源不够。此时利用资源分配调整公式(9)，根据第一个网络切片的优先级计算需要缩小资源(42,42,40)。然后调整的资源分配从(85,85,80)减少为(43,43,40)。现在PNF1上的可用资源从(15,15,10)变为(57,57,50)，可以满足的资源需求。于是继续按照先前步骤为第二个网络切片进行部署决策。
对第二个网络切片的每个节点进行部署决策后，这两个切片的部署决策为：S1:(PNF1,PNF3,PNF5),S2:(PNF1,PNF4,PNF7,PNF9), 将部署决策传到知识图谱模块，知识图谱模块进行更新。
按照步骤103，知识图谱将更新后的网络信息传回强化学习模块，从网络信息中获取到各PNF的失效平均时间和平均修复时间后根据公式(1)计算各PNF的可用性，假设计算得到的PNF1的可用性为0.99，PNF2的可用性为0.98，PNF3的可用性为0.96，PNF4的可用性为0.97，PNF6的可用性为0.98，设定的可用性阈值为0.95，因此所有PNF都可用。再根据PNF的可用性以及PNF和VNF的资源条件可以得到切片的网络运行状态。其中，切片S1的PNF状态分别为0000001111、0000101111、0101001111，切片S1的VNF初始状态分别为000001111、010101111、101001111；切片S2的PNF状态分别为0000011111、0010111111、0111011111，切片S2的VNF初始状态分别为000011111、010111111、101011111、111011111。于是网络状态评估为正常，然后将部署决策信息传至控制模块进行下发，完成用户的网络切片请求。
由上述具体实例可见，根据实施例1中提供的基于强化学习的数字孪生网络切片的方法，能够快速有效的完成数字孪生的网络切片，更好地保证网络资 源的高效利用，并确保网络动态变化的情况下能够保持网络状态正常。
实施例3：
在上述实施例1至实施例2提供的基于强化学习的数字孪生网络切片的方法的基础上，本发明还提供了一种可用于实现上述方法的基于强化学习的数字孪生网络切片的装置，如图基于强化学习的数字孪生网络切片所示，是本发明实施例的装置架构示意图。本实施例的基于强化学习的数字孪生网络切片的装置包括一个或多个处理器11以及存储器12。其中，图6中以一个处理器11为例。
处理器11和存储器12可以通过总线或者其他方式连接，图6中以通过总线连接为例。
存储器12作为一种基于强化学习的数字孪生网络切片的方法非易失性计算机可读存储介质，可用于存储非易失性软件程序、非易失性计算机可执行程序以及模块，如实施例1至实施例2中的基于强化学习的数字孪生网络切片的方法。处理器11通过运行存储在存储器12中的非易失性软件程序、指令以及模块，从而执行基于强化学习的数字孪生网络切片的装置的各种功能应用以及数据处理，即实现实施例1至实施例2的基于强化学习的数字孪生网络切片的方法。
存储器12可以包括高速随机存取存储器，还可以包括非易失性存储器，例如至少一个磁盘存储器件、闪存器件、或其他非易失性固态存储器件。在一些实施例中，存储器12可选包括相对于处理器11远程设置的存储器，这些远程存储器可以通过网络连接至处理器11。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
程序指令/模块存储在存储器12中，当被一个或者多个处理器11执行时，执行上述实施例1至实施例2中的基于强化学习的数字孪生网络切片的方法，例如，执行以上描述的图2和图3所示的各个步骤。
本领域普通技术人员可以理解实施例的各种方法中的全部或部分步骤是可以通过程序来指令相关的硬件来完成，该程序可以存储于一计算机可读存储介质中，存储介质可以包括：只读存储器(Read Only Memory，简写为：ROM)、随机存取存储器(Random Access Memory，简写为：RAM)、磁盘或光盘等。
以上所述仅为本发明的较佳实施例而已，并不用以限制本发明，凡在本发明的精神和原则之内所作的任何修改、等同替换和改进等，均应包含在本发明的保护范围之内。

Claims (22)

  1. 一种基于强化学习的数字孪生网络切片的方法，其特征在于，具体包括：
    强化学习模块获取知识图谱中当前网络信息三元组和网络切片需求三元组，每个网络切片对应一组网络切片需求三元组，其中，当前网络信息三元组包括网络中所有PNF连接关系和每个PNF的资源属性，网络切片需求三元组包括切片需求的所有VNF连接关系、每个VNF的资源需求和切片优先级；
    强化学习模块生成至少一个网络切片的决策动作，每个决策动作包含根据节点选择概率模型和知识图谱查找PNF节点以及根据资源分配模型对每个PNF节点分配给VNF的资源映射为资源分配向量，知识图谱根据决策动作进行决策以获得决策结果，决策结果为网络切片VNF与网络切片PNF的部署位置；
    知识图谱根据决策结果进行更新，并将更新后的网络状态传回给强化学习模块，强化学习模块确认网络状态正常后，下发网络切片的部署策略。
  2. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块生成至少一个网络切片的决策动作，具体包括：
    根据切片需求，基于知识图谱，找到节点选择概率模型中概率最大的PNF作为第一个节点；
    基于知识图谱得到资源评估向量，输入到资源分配模型中为第一VNF节点分配资源，并更新知识图谱；
    根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到可能的下一个VNF节点的PNF位置选择，依次查找直至所有节点都部署分配完毕。
  3. 根据权利要求2所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述找到节点选择概率模型中概率最大的PNF作为第一个节点，具体包括：找到第一个VNF节点的动作空间中所有可能的PNF位置选择作为第一VNF节点的部署动作空间，并得到节点状态矩阵；
    将节点状态矩阵输入到节点选择概率模型中，计算每个节点被选择的概率，选择其中概率最大的PNF作为第一个节点；
    基于知识图谱得到第一个节点的资源评估向量，将评估向量输入到资源分配模型中，为第一VNF节点分配资源。
  4. 根据权利要求2所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述找到可能的下一个VNF节点的PNF位置选择，具体包括：
    根据上一个VNF节点选取的位置，基于知识图谱里包含的PNF连接关系进行知识推理，找到所有可能的下一个VNF节点的PNF位置选择，作为下一个VNF节点的部署动作空间；
    基于知识图谱得到节点状态矩阵，依次向后进行下一个节点的部署和资源分配，直至所有节点都部署分配完毕，以所有节点的分配结果完成一整个切片部署策略的选择。
  5. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述知识图谱根据决策结果进行更新，具体包括：
    知识图谱根据强化学习的资源分配策略进行更新，并根据反馈函数计算反馈值；结合反馈值利用梯度下降法更新节点选择概率模型的参数。
  6. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述节点选择概率模型具体包括：
    将PNF之间的距离、切片对于VNF节点的资源需求和PNF上的资源状态映射成可能选择的概率，以便于选择强化学习模块获取概率最大的PNF节点进行部署。
  7. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述资源分配模型具体包括：
    通过函数将资源评估向量映射成资源分配向量，以便于表示PNF节点分配给VNF的计算、存储、网络资源。
  8. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块确认网络状态正常，还包括：
    当网络切片请求在进行某个节点部署，由于之前的网络切片占用过多资源而无法找到资源满足需求的PNF时，根据网络切片的优先级排序，在不低于最低需求的条件下，按比例缩小前几个网络切片中每个VNF的资源，缩小的资源大小与网络切片的优先级成反比。
  9. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块确认网络状态正常，还包括：
    当网络状态发生改变，知识图谱模块中当前时刻的网络状态无法满足原时刻的切片需求时，则强化学习模块退回到不满足的切片步骤重新进行部署决策。
  10. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法包括：
    所述网络信息三元组中，第一个实体表示PNF，关系表示该三元组的资源类型，第二个实体表示第一个实体所需的该类型资源的值；
    所述网络切片需求三元组中，第一个实体表示网络切片或VNF，关系表示该三元组的资源类型，第二个实体表示第一个实体所需的该类型资源的值。
  11. 根据权利要求10所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法还包括：
    所述网络切片需求三元组中，当第一个实体为网络切片时，关系的类型为所需的VNF或优先级，第二个实体为第一个实体需要的VNF或优先级的值。
  12. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述知识图谱根据决策结果进行更新，并将更新后的网络状态传回给强化学习模块，具体包括：
    将一个网络切片需求转化为至少一个VNF的分配需求，对于每个VNF的分配过程，确认分配在哪个PNF上，并根据需求分配相应的资源；
    每分配完成一个VNF后，对知识图谱中的网络信息三元组进行更新，以便根据更新后的网络信息三元组对下一个VNF进行预测。
  13. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块确认网络状态正常后，下发网络切片的部署策略，具体包括：
    知识图谱将更新后的网络状态传回强化学习模块，强化学习模块对调整后的网络状态做出评估，确认使用该切片策略后网络状态正常后，将部署决策信息传至控制模块进行下发。
  14. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法还包括：
    完成所有网络切片部署决策后，将部署决策传到知识图谱模块，对知识图谱模块进行更新；
    知识图谱将更新后的网络状态传回强化学习模块，以便使知识图谱中的对应关系能够在网络环境出现动态变化时与当前环境相符。
  15. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块具体包括：
    使用以下数学模型对网络运行状态进行描述：
    表示物理集群层中的PNF；
    表示物理集群层中的PNF与PNF之间的物理链路；
    S＝{S1,S2,…,Sb}表示网络中存在的网络切片用，总计有b个网络切片；
    在一个网络切片Ss中，表示VNF节点，表示VNF节点之间的虚拟链路；
    对于的计算、存储、网络资源分别表示为
    切片中VNF节点的计算、存储、网络资源分别表示为
  16. 根据权利要求15所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块还包括：
    每个PNF或VNF可靠性由失效平均时间衡量，可服务性由平均修复时间衡量；PNF(VNF)的可用性APNF(AVNF)使用以下公式表示：
    其中，整个切片的可用性等于构成切片PNF和VNF的可用性最小值，用于评估网络运行状态。
  17. 根据权利要求15所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块还包括：
    VNF和PNF的位置关系用使用以下公式表示：
    其中，如果在上，则否则为0。
  18. 根据权利要求15所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述强化学习模块还包括：
    和的计算、存储和网络资源满足以下条件：
    其中，根据PNF、VNF可用性以及资源条件定义网络运行态，对网络切片时，选择VNF并选择所部署的PNF，按照空间位置关系对PNF排序。
  19. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法还包括：
    节点概率选择模型对节点状态矩阵进行加权计算：Y＝WX+b，以得到每一个PNF节点是否可以作为下一个PNF节点的评价分数Y＝(y1,y2,…yn)；
    根据评价分数计算每个PNF节点作为被选择的概率
    其中，W为神经元权重参数，b为模型偏置参数，均由事先的模型训练得到。
  20. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法还包括：
    奖励更新策略模型的反馈函数使用以下公式表示：
    r＝rACCURACY(ηrSATISFACATION+ξrEFFICIENCY)；
    其中，η，ξ分别为切片需求满足度和链路效率的比例系数，r为反馈值，计算方式为需求满足度与rSATISFACATION和链路效率rEFFICIENCY加权求和，再与部署精度rACCURACY相乘。
  21. 根据权利要求1所述的基于强化学习的数字孪生网络切片的方法，其特征在于，所述方法还包括：
    当网络切片请求在过多资源而无法找到资源满足需求的PNF时，缩小的资源大小v使用如下公式计算：
    其中，i为切片优先级顺序，z为之前被分配到的资源大小，h为切片最低需求，参数θ可以在(0,1]中取值。θ越大，资源缩减程度越大。
  22. 一种基于强化学习的数字孪生网络切片的装置，其特征在于：
    包括至少一个处理器和存储器，所述至少一个处理器和存储器之间通过数据总线连接，所述存储器存储能被所述至少一个处理器执行的指令，所述指令在被所述处理器执行后，用于完成权利要求1-21中任一项所述的基于强化学习的数字孪生网络切片的方法。
PCT/CN2024/070341 2023-01-17 2024-01-03 一种基于强化学习的数字孪生网络切片的方法和装置 Ceased WO2024152902A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
MX2024013573A MX2024013573A (es) 2023-01-17 2024-11-01 Metodo y aparato de segmentacion de red gemela digital a base de aprendizaje de refuerzo

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202310089923.8 2023-01-17
CN202310089923.8A CN116389266A (zh) 2023-01-17 2023-01-17 一种基于强化学习的数字孪生网络切片的方法和装置

Publications (1)

Publication Number Publication Date
WO2024152902A1 true WO2024152902A1 (zh) 2024-07-25

Family

ID=86968291

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2024/070341 Ceased WO2024152902A1 (zh) 2023-01-17 2024-01-03 一种基于强化学习的数字孪生网络切片的方法和装置

Country Status (4)

Country Link
CN (1) CN116389266A (zh)
CL (1) CL2024003560A1 (zh)
MX (1) MX2024013573A (zh)
WO (1) WO2024152902A1 (zh)

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119402967A (zh) * 2024-11-04 2025-02-07 国网山东省电力公司营销服务中心(计量中心) 一种基于5g通信分级分类的电力业务调度方法及系统
CN119450526A (zh) * 2024-11-12 2025-02-14 重庆邮电大学 一种联合vnf分区与混合备份共享的网络切片部署方法
CN119891197A (zh) * 2025-03-24 2025-04-25 北京北方永达智能电气有限公司 一种基于数字孪生的预装式设备舱配电控制方法及系统
CN120219124A (zh) * 2025-03-28 2025-06-27 湖北凯纳特科技有限公司 一种智慧校园数据管理方法及系统
CN120547082A (zh) * 2025-07-31 2025-08-26 西北工业大学 一种基于优先级驱动的自适应网络切片重构方法及系统
CN120645237A (zh) * 2025-08-20 2025-09-16 上海鸿亦昌实业有限公司 基于深度强化学习的机器人决策方法
CN120723958A (zh) * 2025-09-01 2025-09-30 浙江大学 基于孪生网络的非结构化数据连接查询优化系统及方法
CN120977551A (zh) * 2025-10-21 2025-11-18 北京冠新医卫软件科技有限公司 基于医疗智能体中间件的全-专科协同诊疗方法及系统
CN120993994A (zh) * 2025-10-17 2025-11-21 铭科智能装备科技(南通)有限公司 一种基于数字孪生的模具温度场协同调控方法
CN121218362A (zh) * 2025-11-19 2025-12-26 广东联想懂的通信有限公司 一种基于5g切片的资源动态分配方法及系统
CN121454973A (zh) * 2026-01-06 2026-02-03 新之航传媒科技集团有限公司 一种基于物联网的智能展馆控制系统

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN116389266A (zh) * 2023-01-17 2023-07-04 烽火通信科技股份有限公司 一种基于强化学习的数字孪生网络切片的方法和装置
CN116828507B (zh) * 2023-07-27 2026-04-17 西安电子科技大学 一种基于非完美数字孪生的用于多无人机网络的决策方法
CN116886548A (zh) * 2023-07-31 2023-10-13 西北工业大学 一种5g端到端切片部署方法及系统
CN118785302B (zh) * 2024-07-17 2025-09-09 北京工业大学 一种基于通算功能切片的无人机集群网络动态路由方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113098714A (zh) * 2021-03-29 2021-07-09 南京邮电大学 一种基于深度强化学习的低时延网络切片的方法
US20210345357A1 (en) * 2018-10-11 2021-11-04 Koninklijke Kpn N.V. Dynamic slice priority handling
CN114666220A (zh) * 2022-03-23 2022-06-24 中国电信股份有限公司 网络切片的资源配置方法及装置、存储介质、电子设备
CN115603850A (zh) * 2021-06-28 2023-01-13 中兴通讯股份有限公司（Cn） 虚拟网络的优化方法、装置及计算机存储介质
CN116389266A (zh) * 2023-01-17 2023-07-04 烽火通信科技股份有限公司 一种基于强化学习的数字孪生网络切片的方法和装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11956763B2 (en) * 2019-01-25 2024-04-09 Northeastern University Methods for the enforcement of network slicing policies in virtualized cellular networks
CN113891401B (zh) * 2021-11-12 2024-10-15 中国电子科技集团公司第七研究所 一种基于深度强化学习的异构网络切片调度方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210345357A1 (en) * 2018-10-11 2021-11-04 Koninklijke Kpn N.V. Dynamic slice priority handling
CN113098714A (zh) * 2021-03-29 2021-07-09 南京邮电大学 一种基于深度强化学习的低时延网络切片的方法
CN115603850A (zh) * 2021-06-28 2023-01-13 中兴通讯股份有限公司（Cn） 虚拟网络的优化方法、装置及计算机存储介质
CN114666220A (zh) * 2022-03-23 2022-06-24 中国电信股份有限公司 网络切片的资源配置方法及装置、存储介质、电子设备
CN116389266A (zh) * 2023-01-17 2023-07-04 烽火通信科技股份有限公司 一种基于强化学习的数字孪生网络切片的方法和装置

Cited By (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN119402967A (zh) * 2024-11-04 2025-02-07 国网山东省电力公司营销服务中心(计量中心) 一种基于5g通信分级分类的电力业务调度方法及系统
CN119450526A (zh) * 2024-11-12 2025-02-14 重庆邮电大学 一种联合vnf分区与混合备份共享的网络切片部署方法
CN119891197A (zh) * 2025-03-24 2025-04-25 北京北方永达智能电气有限公司 一种基于数字孪生的预装式设备舱配电控制方法及系统
CN120219124A (zh) * 2025-03-28 2025-06-27 湖北凯纳特科技有限公司 一种智慧校园数据管理方法及系统
CN120547082A (zh) * 2025-07-31 2025-08-26 西北工业大学 一种基于优先级驱动的自适应网络切片重构方法及系统
CN120645237A (zh) * 2025-08-20 2025-09-16 上海鸿亦昌实业有限公司 基于深度强化学习的机器人决策方法
CN120723958A (zh) * 2025-09-01 2025-09-30 浙江大学 基于孪生网络的非结构化数据连接查询优化系统及方法
CN120993994A (zh) * 2025-10-17 2025-11-21 铭科智能装备科技(南通)有限公司 一种基于数字孪生的模具温度场协同调控方法
CN120977551A (zh) * 2025-10-21 2025-11-18 北京冠新医卫软件科技有限公司 基于医疗智能体中间件的全-专科协同诊疗方法及系统
CN121218362A (zh) * 2025-11-19 2025-12-26 广东联想懂的通信有限公司 一种基于5g切片的资源动态分配方法及系统
CN121454973A (zh) * 2026-01-06 2026-02-03 新之航传媒科技集团有限公司 一种基于物联网的智能展馆控制系统

Also Published As

Publication number Publication date
CN116389266A (zh) 2023-07-04
MX2024013573A (es) 2024-12-06
CL2024003560A1 (es) 2025-02-21

Similar Documents

Publication Publication Date Title
WO2024152902A1 (zh) 一种基于强化学习的数字孪生网络切片的方法和装置
CN110365514B (zh) 基于强化学习的sdn多级虚拟网络映射方法和装置
CN113225377B (zh) 物联网边缘任务卸载方法及装置
US11201815B2 (en) Method and system for selecting least-loaded route based on naive Bayes classifier
CN112738820A (zh) 一种服务功能链的动态部署方法、装置及计算机设备
CN108684046B (zh) 一种基于随机学习的接入网服务功能链部署方法
CN117041330B (zh) 一种基于强化学习的边缘微服务细粒度部署方法及系统
CN113835899B (zh) 针对分布式图学习的数据融合方法及装置
EP3793206B1 (en) Physical optical network virtualization mapping method and apparatus, and controller and storage medium
CN108111335B (zh) 一种调度和链接虚拟网络功能的方法及系统
CN110247795B (zh) 一种基于意图的云网资源服务链编排方法及系统
CN110995619B (zh) 一种服务质量感知的虚拟网络映射方法和装置
CN119110348B (zh) 一种移动边缘计算的协同任务卸载和区块挖掘方法
CN116541106A (zh) 计算任务卸载方法、计算设备及存储介质
CN104506576B (zh) 一种无线传感器网络及其节点任务迁移方法
CN116827796A (zh) 一种基于vnf资源需求预测的sfc智能部署方法及系统
CN119225921A (zh) 一种传输路径的确定方法及计算设备
Bensalem et al. Towards optimal serverless function scaling in edge computing network
JP5626685B2 (ja) 仮想網制御方法及び管理ノード装置
CN120469982B (zh) 基于拓扑感知与深度强化学习的协同式数据迁移调度方法
CN119728107A (zh) 基于注意力机制及残差网络的密码资源分配方法及系统
CN116436978B (zh) 面向云计算的内存分配方法、内存获取方法、装置和设备
CN118301072A (zh) 一种多源多播服务功能链部署方法及系统
Mohamed et al. Fragmentation-aware VNF placement: A deep reinforcement learning approach
CN113596109A (zh) 业务请求运行方法、系统、装置、设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24744078

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 12024552325

Country of ref document: PH

WWE Wipo information: entry into national phase

Ref document number: MX/A/2024/013573

Country of ref document: MX

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24744078

Country of ref document: EP

Kind code of ref document: A1