CN113923128B - Intelligent coding caching method based on federal reinforcement learning in fog wireless access network - Google Patents
Intelligent coding caching method based on federal reinforcement learning in fog wireless access network Download PDFInfo
- Publication number
- CN113923128B CN113923128B CN202111258088.3A CN202111258088A CN113923128B CN 113923128 B CN113923128 B CN 113923128B CN 202111258088 A CN202111258088 A CN 202111258088A CN 113923128 B CN113923128 B CN 113923128B
- Authority
- CN
- China
- Prior art keywords
- access point
- fog
- local
- global
- fog access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 36
- 230000002787 reinforcement Effects 0.000 title claims abstract description 20
- 230000005540 biological transmission Effects 0.000 claims abstract description 18
- 238000012549 training Methods 0.000 claims abstract description 9
- 239000003595 mist Substances 0.000 claims description 13
- 238000013528 artificial neural network Methods 0.000 claims description 6
- 230000010354 integration Effects 0.000 claims description 4
- 239000003795 chemical substances by application Substances 0.000 claims description 3
- 239000000284 extract Substances 0.000 claims description 3
- 230000004931 aggregating effect Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 230000009471 action Effects 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000010295 mobile communication Methods 0.000 description 2
- 238000004088 simulation Methods 0.000 description 2
- 230000003139 buffering effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Mobile Radio Communication Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network, which comprises the following steps: 1. the cloud center server builds a global prediction model and initializes the global prediction model, and a plurality of distributed learning agents are arranged at the fog access points and the local prediction model is initialized; 2. observing the global state by the system; 3. each fog access point observes a local state; 4. the system uses a multicast coding transmission mode to meet the request received by each network edge node according to the implemented content placement decision; 5. each fog access point calculates theoretical time delay and virtual feedback according to local request conditions, and stores local learning experience; 6. each fog access point independently performs local model training by using local learning experience; 7. and uploading the local model to the cloud center server every a plurality of time slots by each fog access point to update the global model. The invention can reduce the forward load of the system, reduce the transmission delay and protect the privacy of users.
Description
Technical Field
The invention belongs to the field of intelligent caching of an edge network in a mobile communication system, and particularly relates to an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network.
Background
With the widespread popularity of smart devices and the rapid development of mobile applications, mobile communication networks will be subjected to increasingly large traffic loads. In order to handle these surge data traffic, a Fog wireless Access network is proposed as a novel network architecture, wherein Fog wireless Access points (F-APs) are arranged near the network edges of users and use their edge storage capability and edge computing capability to reduce the traffic pressure of a cloud center server and improve the user experience. Meanwhile, the coding cache is used as a new cache mode, the local cache and the multicast transmission are effectively combined, and the limited storage space of the edge equipment can be more fully utilized.
However, most of the current research on non-uniform popularity code caching methods is focused on upper and lower bound analysis of theoretical performance of a fixed content placement strategy, so that the non-optimal content placement strategy cannot fully utilize the potential of code caching to reduce the forwarding load. In addition, the current non-uniform popularity coding caching method assumes ideal conditions of fixed popularity of the content, ignores the fact that the popularity of the content changes with time, and causes that the performances of forwarding load, time delay and the like cannot reach ideal states in practice. Therefore, there is a need for a method for encoding and caching in the case of time-varying popularity, which reduces the content transmission delay and improves the stability at the same time, thereby providing higher quality and more reliable communication services for mobile users.
Disclosure of Invention
The invention aims to provide an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network, which aims to solve the technical problems of reducing forward load, reducing request time delay and protecting user privacy.
In order to solve the technical problems, the specific technical scheme of the invention is as follows:
a code caching method based on federal reinforcement learning in a fog wireless access network comprises the following steps:
step 1, constructing a global model theta at a cloud center server G And initializing, arranging a learning agent at each fog access point and initializing its local model θ k ；
Step 2, at the end of the time slot t, the system observes the global state s (t), and each mist access point observes its local state s k (t)；
Step 3, the system predicts and implements the actual coding cache content placement decision a (t) by using a global model according to the observed global state s (t), and each fog access point simultaneously carries out the local state s according to the observation k (t) predicting and recording virtual content placement decisions a k (t)；
Step 4, in the time slot t+1, the system uses a multicast coding transmission mode to meet the request received by each fog access point according to the implemented content placement decision a (t);
step 5, at time slot t+1 end, local state becomes s k (t+1) each fog access point counts the content request condition of its service user in time slot t+1 and makes a decision a according to its virtual placement k (t) calculating theoretical delay and virtual feedback r k (t) storing the local learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T ；
Step 6, each fog access point randomly samples the local learning experience of the fog access point to train so as to update the local model;
step 7, every T A Uploading the local model of each fog access point to a cloud center server for integration, and downloading an integrated global model for replacing the local model;
and 8, repeatedly executing the steps 2 to 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5%.
Further, the step 1 specifically includes the following steps:
step 1.1, constructing a neural network Q (s, a; theta) by a cloud center server G ) Where s is the current global state vector, a is the global content placement decision vector, θ G Is a global network model parameter and is randomly initialized;
step 1.2 forFor the fog access point index set, K is the number of fog access points, and the fog access point K constructs a neural network Q (s k ，a k ；θ k ) Wherein s is k A is the current local state vector, a k Placing a decision vector, θ, for local content k Is a local network model parameter and is randomly initialized.
Further, the step 2 specifically includes the following steps:
step 2.1, at the end of time slot t, the system observes the global stateWherein a (t-1) is a global caching decision made by the t-1 slot system,/for a-1 slot system>The frequency vector is requested for statistics of all N files in the t time slot system;
step 2.2, at the same time, forFog access point k observes its local state +.>Wherein a is k (t-1) local buffer decision recorded for t-1 time slot,/i>The frequency vector is requested for all N files within slot tboot access point k.
Further, the step 3 specifically includes the following steps:
step 3.1, at the end of time slot t, the system predicts and implements actual encoded cache content placement decisions using global model based on observed global state s (t)Wherein N is c (t) represents the number of files in the code cache, c n (t) =1 means that file n is selected, c n (t) =0 indicates that file n is not selected;
step 3.2 forFog access point k based on observed local state s k (t) predicting and recording virtual coded cache content placement decisions +.>Wherein->Representing the number of files encoded in the cache,/->Indicating that file n is selected,/->It indicates that file n is not selected.
Further, the step 3.1 specifically includes the following steps:
step 3.1.1, at the end of the time slot t, the system predicts the actual encoded cache content placement decision using the global model according to the observed global state s (t):
step 3.1.2, implementing a content placement strategy: by usingRepresenting a subset of foggy access points, wherein +.>Is the number of elements L t Is a fog access point subset, L t Is a variable and L t ＝KM/N c (t), M is the cache size of the fog access point, |ζ| represents the number of collection elements taken; splitting File n into +.>Sub-files of the same size +.>Wherein-> Representing the selected set of file indexes, shaped as +.>Representing the number of combinations of B elements taken from among A different elements; for->The placement content of the fog access point k is as follows:
further, the step 3.2 specifically includes the following steps:
step 3.2.1 forFog access point k based on observed local state s k (t) randomly selecting virtual content placement decisions with a probability of ε according to a greedy action selection policy, predicting virtual content placement decisions with a probability of 1- ε using its local model:
step 3.2.2 record content placement strategy a k (t) but not implemented;
further, the step 4 specifically includes the following steps:
step 4.1, each fog access point receives a user request;
step 4.2, for cached requests, usingRepresenting its index set; use->Representing a fog access point set in which the request file is cached; let->Representing a set of fog access point subsets, wherein +.>For the fog access point subset, the cloud center server is +.>The contents of the multicast transmission are:
wherein,representing a bit exclusive or operation;
step 4.3, for uncached requests, the cloud center server goes toUnicast transmission requests content.
Further, the step 5 specifically includes:
step 5.1 forThe fog access point K equally divides the received V requests into K parts;
step 5.2, aggregating each request by the fog access pointIs received in a mist-access manner, where k 1 ，k 2 ，...，k K-1 Is a virtual fog access point that virtually exists;
step 5.3, for fog access point set stationThe i-th request of all K mist access points in the network is represented by +.>Wherein->The i-th request representing the virtual foggy access point k' is file n,/or->The ith request representing virtual fog access point k' is not file n so that it is loaded from the forwarding link asWherein->Is a variable and->Form min (a, B) represents the smaller value selected from the two values of A, B;
in step 5.4, in time slot t, the theoretical delay of the fog access point k is expressed asWherein d is f Delay for completely transmitting one file to fog access point for cloud center server, d a A delay for complete transmission of a file from the foggy access point to the user;
step 5.5, virtual feedback is as follows:
wherein the method comprises the steps ofμ 1 +μ 2 ＝1，0＜μ 1 ＜μ 2 ＜1；
Step 5.6, learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T Stored in a local experience playback pool.
Further, the step 6 specifically includes the following steps:
step 6.1 forMist access point k randomly extracts experiences s from a local experience playback pool k (j)，a k (j)，r k (j)，s k (j+1)] T ；
Step 6.2 training the local model θ using gradient descent k And updating.
The intelligent coding caching method based on federal reinforcement learning in the fog wireless access network has the following advantages:
1. aiming at the more challenging scene of time-varying content popularity in the fog wireless access network, the invention uses federal reinforcement learning to track the time-varying popularity, adaptively makes content placement decisions, can effectively reduce forward load, reduce transmission delay and keep performance stable, and is suitable for scenes closer to reality.
2. According to the invention, a virtual coding caching method is used, and theoretical time delay and virtual feedback of a virtual content placement strategy are calculated by assuming a virtual fog access point, so that the problem of local training data collection caused by the fact that a single fog access point cannot execute coding caching is solved, and distributed local training is realized.
3. According to the method, federal learning is used, the edge computing capability of the mist access node is fully utilized, the global prediction model of the coding cache content placement strategy is obtained through distributed training and model integration, and repeated uploading of training data to a cloud center server is avoided, so that bandwidth resource waste of a forward link is reduced, and privacy leakage risk of a user is reduced.
Drawings
Fig. 1 is a schematic flow chart of a code caching method based on federal reinforcement learning in a foggy radio access network according to the present invention;
fig. 2 is a graph of average time delay performance simulation results of a code caching method based on federal reinforcement learning in a foggy radio access network according to the present invention.
Detailed Description
In order to better understand the purpose, structure and function of the present invention, the following describes in detail an intelligent coding caching method based on federal reinforcement learning in a mist radio access network with reference to the accompanying drawings.
The embodiment provides a method for buffering asynchronous request codes of a fog wireless access network, which is shown in fig. 1 and comprises the following steps:
step 1, constructing a global model theta at a cloud center server G And initializing, arranging a learning agent at each fog access point and initializing its local model θ k 。
The step 1 specifically comprises the following steps:
step 1.1, constructing a neural network Q (s, a; theta) by a cloud center server G ) Where s is the current global state vector, a is the global content placement decision vector, θ G Is a global network model parameter and is randomly initialized;
step 1.2 forFor the fog access point index set, K is the number of fog access points, and the fog access point K constructs a neural network Q (s k ，a k ；θ k ) Wherein s is k A is the current local state vector, a k Placing a decision vector, θ, for local content k Is a local network model parameter and is randomly initialized.
Step 2, at the end of the time slot t, the system observes the global state s (t), and each mist access point observes its local state s k (t)。
The step 2 specifically comprises the following steps:
step 2.1, at the end of time slot t, the system observes the global stateWherein a (t-1) is a global caching decision made by the t-1 slot system,/for a-1 slot system>The frequency vector is requested for statistics of all N files in the t time slot system;
step 2.2, at the same time, forFog access point k observes its local state +.>Wherein a is k (t-1) local buffer decision recorded for t-1 time slot,/i>The frequency vector is requested for all N files within slot tboot access point k.
Step 2.3, the system predicts and implements the actual code cache content placement decision a (t) using the global model based on the observed global state s (t), while each mist access point is based on its observed local state s k (t) predicting and recording virtual content placement decisions a k (t)。
The step 3 specifically comprises the following steps:
step 3.1, at the end of the time slot t, the system predicts the actual encoded cache content placement decision using the global model based on the observed global state s (t)
Wherein N is c (t) represents the number of files in the code cache, c n (t) =1 means that file n is selected, c n (t) =0 indicates that file n is not selected. The content placement policy is then implemented: by usingRepresenting a subset of foggy access pointsMiddle->Is the number of elements L t Is a fog access point subset, L t Is a variable and L t ＝KM/N c (t), M is the cache size of the fog access point, and the shape of M is represented by I and II, and the number of the collection elements is taken; splitting File n into +.>Sub-files of the same sizeWherein-> Representing the selected set of file indexes, shaped as +.>Representing the number of combinations of B elements taken from among A different elements. For->The placement content of the fog access point k is as follows:
step 3.2, at the same time, forFog access point k based on observed local state s k (t) randomly selecting virtual content placement decisions with epsilon probabilities according to a greedy action selection policy, predicting virtual content placement decisions with 1-epsilon probabilities using their local models ++>
Recording the content placement policy a k (t) but is not implemented.
And 4, in the time slot t+1, the system uses a multicast coding transmission mode to meet the request received by each fog access point according to the content placement decision a (t).
The step 4 specifically comprises the following steps:
step 4.1, each fog access point receives a user request;
step 4.2, for cached requests, usingRepresenting its index set. Use->Representing a set of foggy access points in which the request file is cached. Let->Representing a set of fog access point subsets, wherein +.>For the fog access point subset, the cloud center server is +.>The contents of the multicast transmission are:
wherein,representing a bit exclusive or operation;
step 4.3,For uncached requests, the cloud center server goes toUnicast transmission requests content.
Step 5, at the end of the time slot t+1, the local state becomes s k (t+1) each fog access point counts the content request condition of its service user in time slot t+1 and makes a decision a according to its virtual placement k (t) calculating theoretical delay and virtual feedback r k (t) storing the local learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T 。
The step 5 specifically comprises the following steps:
step 5.2 forThe fog access point K equally divides the received V requests into K parts;
step 5.2, assume that each request is aggregated by a foggy access pointIs received in a mist-access manner, where k 1 ，k 2 ，...，k K-1 Is a virtual fog access point that virtually exists;
step 5.3 for foggy Access Point aggregationThe i-th request of all K mist access points in the network is represented by +.>Wherein->The i-th request representing the virtual foggy access point k' is file n,/or->Then represent a virtual fog access pointThe ith request of k' is not file n, so it is loaded from the forwarding link asWherein->Is a variable and->Form min (a, B) represents the smaller value selected from the two values of A, B;
in step 5.4, in time slot t, the theoretical delay of the fog access point k is expressed asWherein d is f Delay for completely transmitting one file to fog access point for cloud center server, d a A delay for complete transmission of a file from the foggy access point to the user;
step 5.5, virtual feedback is as follows:
wherein the method comprises the steps ofμ 1 +μ 2 ＝1，0＜μ 1 ＜μ 2 ＜1；
Step 5.6, learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T Stored in a local experience playback pool;
and 6, each fog access point randomly samples the local learning experience of the fog access point to train, and local model updating is achieved.
The step 6 specifically comprises the following steps:
step 6.1 forMist access point k randomly extracts experiences s from a local experience playback pool k (j)，a k (j)，r k (j)，s k (j+1)] T ；
Step 6.2 training the local model θ using gradient descent k And updating.
Step 7, every T A And uploading the local model of each cloud access point to a cloud center server for integration, and downloading the integrated global model for replacing the local model.
The step 7 specifically comprises the following steps:
step 7.1 forThe fog access point k will model its local model θ k Uploading to a cloud center server;
step 7.2, the cloud center server integrates:
wherein D is k At T for fog access point k A The number of learning experiences drawn in total in training of the individual time slots;
step 7.3 forMist access point k downloads updated global model θ G And let theta k ＝θ G 。
And 8, repeatedly executing the steps 2 to 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5%.
In the simulation result of fig. 2, LFU (Least frequently Used) is a conventional non-coding caching method, NUCC (Coded Caching Under Non-Uniform Content Popularity Distributions with Multiple Requests published by Abdollah Ghaffari Sheshjavani et al in 2020IEEE Wireless Communications and Networking Conference) and APCC (Coded Caching Under Arbitrary Popularity Distributions published by Jinbei Zhang et al in IEEE Transactions on Information Theory in 2018) are two coding caching methods designed for non-uniform popularity, centralised is a Centralized extension method of the present invention, and propend is the method of the present patent. Compared with LFU, NUCC, APCC and other methods, the method of the invention realizes more stable and lower-delay content transmission, and has no obvious performance loss compared with a centralized extension method.
It will be understood that the invention has been described in terms of several embodiments, and that various changes and equivalents may be made to these features and embodiments by those skilled in the art without departing from the spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.
Claims (9)
1. The code caching method based on federal reinforcement learning in the fog wireless access network is characterized by comprising the following steps:
step 1, constructing a global model theta at a cloud center server G And initializing, arranging a learning agent at each fog access point and initializing its local model θ k ；
Step 2, at the end of the time slot t, the system observes the global state s (t), and each mist access point observes its local state s k (t)；
Step 3, the system predicts and implements the actual content placement decision a (t) by using the global model according to the observed global state s (t), and each fog access point simultaneously predicts and implements the actual content placement decision a (t) according to the observed local state s k (t) predicting and recording virtual content placement decisions a k (t)；
In the time slot t+1, the system uses a multicast coding transmission mode to meet the request received by each fog access point according to the implemented actual content placement decision a (t);
step 5, at the end of the time slot t+1, the local state becomes s k (t+1) each fog access point counts the content request condition of its service user in time slot t+1 and makes decision a according to its virtual content placement k (t) calculating theoretical delay and virtual feedback r k (t) storing the local learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T ；
Step 6, each fog access point randomly samples the local learning experience of the fog access point to train so as to update the local model;
step 7, every T A Uploading the local model of each fog access point to a cloud center server for integration, and downloading an integrated global model for replacing the local model;
and 8, repeatedly executing the steps 2 to 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5%.
2. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 1, wherein the step 1 specifically comprises the following steps:
step 1.1, constructing a neural network Q (s, a; theta) by a cloud center server G ) Where s is the current global state vector, a is the global content placement decision vector, θ G Is a global network model parameter and is randomly initialized;
step 1.2 forFor the fog access point index set, K is the number of fog access points, and the fog access point K constructs a neural network Q (s k ，a k ；θ k ) Wherein s is k A is the current local state vector, a k Placing a decision vector, θ, for local content k Is a local network model parameter and is randomly initialized.
3. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 2, wherein the step 2 specifically comprises the following steps:
step 2.1, at the end of time slot t, the system observes the global stateWherein a (t-1) is a global caching decision made by the t-1 slot system,/for a-1 slot system>The frequency vector is requested for statistics of all N files in the t time slot system;
step 2.2, at the same time, forFog access point k observes its local state +.>Wherein a is k (t-1) local buffer decision recorded for t-1 time slot,/i>The frequency vector is requested for all N files within slot tboot access point k.
4. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 3, wherein the step 3 specifically comprises the following steps:
step 3.1, at the end of time slot t, the system predicts and implements actual content placement decisions using global models based on the observed global state s (t)Wherein N is c (t) represents the number of files in the code cache, c n (t) =1 means that file n is selected, c n (t) =0 indicates that file n is not selected;
step 3.2 forFog access point k based on observed local state s k (t) predicting and recording virtual content placement decisions using its local model>Wherein->Representing the number of files encoded in the cache,/->Indicating that file n is selected,/->It indicates that file n is not selected.
5. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 4, wherein the step 3.1 specifically comprises the steps of:
step 3.1.1, at the end of the time slot t, the system predicts the actual content placement decision using the global model according to the observed global state s (t):
step 3.1.2, implementing a content placement strategy: by usingRepresenting a subset of foggy access points, wherein +.>Is the number of elements L t Is a fog access point subset, L t Is a variable and L t ＝KM/N c (t), M is the cache size of the fog access point,representing the number of collection elements; splitting File n into +.>Sub-files of the same size +.>Wherein the method comprises the steps of Representing the selected set of file indexes, shaped as +.>Representing the number of combinations of B elements taken from among A different elements; for->The placement content of the fog access point k is as follows:
6. the code caching method based on federal reinforcement learning in a foggy radio access network according to claim 5, wherein the step 3.2 specifically comprises the steps of:
step 3.2.1 forFog access point k based on observed local state s k (t) according to greedyAction selection strategy, randomly selecting virtual content placement decisions with epsilon probability, and predicting virtual content placement decisions with 1-epsilon probability by using local model:
step 3.2.2 recording virtual content placement decision a k (t) but is not implemented.
7. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 6, wherein the step 4 specifically comprises the steps of:
step 4.1, each fog access point receives a user request;
step 4.2, for cached requests, usingRepresenting its index set; use->Representing a fog access point set in which the request file is cached; let->Representing a set of fog access point subsets, wherein +.>For the fog access point subset, the cloud center server is +.>The contents of the multicast transmission are:
wherein,representing a bit exclusive or operation, s\ { k } is the set subtraction of the fog access point subset S and the fog access point set { k };
step 4.3, for uncached requests, the cloud center server goes toUnicast transmission request content,/->Index set for fog access point->Mist access point set with request file cached +.>Is a set subtraction of (a).
8. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 7, wherein the step 5 specifically comprises:
step 5.1 forThe fog access point K equally divides the received V requests into K parts;
step 5.2, aggregating each request by the fog access pointWherein k is a fog access point reception 1 ，k 2 ，...，k K-1 Is a virtual fog access point that virtually exists;
step 5.3 for foggy Access Point aggregationIth request of all K foggy access points in the network, definitionRepresenting the number of requests buffered, wherein +.>The i-th request representing the virtual foggy access point k' is file n,/or->The i-th request indicating the virtual fog access point k' is not file n and thus its forward link load is +.>Wherein->Is a variable and->Form min (a, B) represents the smaller value selected from the two values of A, B;
in step 5.4, in time slot t, the theoretical delay of the fog access point k is expressed asWherein d is f Delay for completely transmitting one file to fog access point for cloud center server, d a A delay for complete transmission of a file from the foggy access point to the user;
step 5.5, virtual feedback is as follows:
wherein the method comprises the steps ofμ 1 +μ 2 ＝1，0＜μ 1 ＜μ 2 ＜1；
Step 5.6, learning experience [ s ] k (t)，a k (t)，r k (t)，s k (t+1)] T Stored in a local experience playback pool.
9. The code caching method based on federal reinforcement learning in a foggy radio access network according to claim 8, wherein the step 6 specifically comprises the steps of:
step 6.1 forMist access point k randomly extracts experiences s from a local experience playback pool k (j)，a k (j)，r k (j)，s k (j+1)] T ；
Step 6.2 training the local model θ using gradient descent k And updating.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111258088.3A CN113923128B (en) | 2021-10-27 | 2021-10-27 | Intelligent coding caching method based on federal reinforcement learning in fog wireless access network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111258088.3A CN113923128B (en) | 2021-10-27 | 2021-10-27 | Intelligent coding caching method based on federal reinforcement learning in fog wireless access network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113923128A CN113923128A (en) | 2022-01-11 |
| CN113923128B true CN113923128B (en) | 2024-02-13 |
Family
ID=79243126
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111258088.3A Active CN113923128B (en) | 2021-10-27 | 2021-10-27 | Intelligent coding caching method based on federal reinforcement learning in fog wireless access network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113923128B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108184252A (en) * | 2017-11-28 | 2018-06-19 | 东南大学 | A kind of asynchronous request code cache method of mist wireless access network |
| CN111314862A (en) * | 2020-02-19 | 2020-06-19 | 东南大学 | Caching method with recommendation under deep reinforcement learning in fog wireless access network |
| CN111340277A (en) * | 2020-02-19 | 2020-06-26 | 东南大学 | Popularity prediction model and method based on federal learning in fog wireless access network |
| CN111935784A (en) * | 2020-08-12 | 2020-11-13 | 重庆邮电大学 | Content caching method based on federal learning in fog computing network |
| CN112579544A (en) * | 2020-12-18 | 2021-03-30 | 北京邮电大学 | File caching method and device, electronic equipment and storage medium |
| CN113255004A (en) * | 2021-06-16 | 2021-08-13 | 大连理工大学 | Safe and efficient federal learning content caching method |
| CN113382059A (en) * | 2021-06-08 | 2021-09-10 | 东南大学 | Collaborative caching method based on federal reinforcement learning in fog wireless access network |
-
2021
- 2021-10-27 CN CN202111258088.3A patent/CN113923128B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108184252A (en) * | 2017-11-28 | 2018-06-19 | 东南大学 | A kind of asynchronous request code cache method of mist wireless access network |
| CN111314862A (en) * | 2020-02-19 | 2020-06-19 | 东南大学 | Caching method with recommendation under deep reinforcement learning in fog wireless access network |
| CN111340277A (en) * | 2020-02-19 | 2020-06-26 | 东南大学 | Popularity prediction model and method based on federal learning in fog wireless access network |
| CN111935784A (en) * | 2020-08-12 | 2020-11-13 | 重庆邮电大学 | Content caching method based on federal learning in fog computing network |
| CN112579544A (en) * | 2020-12-18 | 2021-03-30 | 北京邮电大学 | File caching method and device, electronic equipment and storage medium |
| CN113382059A (en) * | 2021-06-08 | 2021-09-10 | 东南大学 | Collaborative caching method based on federal reinforcement learning in fog wireless access network |
| CN113255004A (en) * | 2021-06-16 | 2021-08-13 | 大连理工大学 | Safe and efficient federal learning content caching method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113923128A (en) | 2022-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110312231B (en) | Content caching decision and resource allocation optimization method based on MEC in Internet of vehicles | |
| CN109218747A (en) | Video traffic classification caching method in super-intensive heterogeneous network based on user mobility | |
| CN112995950B (en) | Resource joint allocation method based on deep reinforcement learning in Internet of vehicles | |
| Dai et al. | Edge intelligence for adaptive multimedia streaming in heterogeneous internet of vehicles | |
| WO2018218850A1 (en) | Decentralized coded caching placement method and system for files with heterogeneous sizes | |
| CN113012013B (en) | Collaborative edge caching method based on deep reinforcement learning in Internet of vehicles | |
| CN113411826B (en) | Edge network equipment caching method based on attention mechanism reinforcement learning | |
| CN113255004A (en) | Safe and efficient federal learning content caching method | |
| CN115633380A (en) | Multi-edge service cache scheduling method and system considering dynamic topology | |
| CN113993168B (en) | Collaborative caching method based on multi-agent reinforcement learning in fog wireless access network | |
| CN109672626B (en) | Service aggregation method based on queuing delay utilization | |
| KR101966588B1 (en) | Method and apparatus for receiving video contents | |
| CN113490239B (en) | Heterogeneous wireless link concurrent transmission control method based on adaptive network coding | |
| CN113923128B (en) | Intelligent coding caching method based on federal reinforcement learning in fog wireless access network | |
| CN114374949A (en) | Power control mechanism based on information freshness optimization in Internet of vehicles | |
| CN115587266A (en) | Air-space-ground integrated internet intelligent edge caching method | |
| CN113783933B (en) | Double-layer network communication method, device and medium based on coding cache | |
| CN113473408B (en) | User association method and system for realizing video transmission in Internet of vehicles | |
| CN115756873A (en) | Mobile edge computing unloading method and platform based on federal reinforcement learning | |
| CN111124298B (en) | Mist computing network content cache replacement method based on value function | |
| Somuyiwa et al. | Multicast-aware proactive caching in wireless networks with deep reinforcement learning | |
| Sivakumar et al. | Tree social relations optimization-based ReLU-BiLSTM framework for improving video quality in video compression | |
| CN115484314A (en) | Edge cache optimization method for recommending performance under mobile edge computing network | |
| CN115129387A (en) | Calculation unloading method based on multi-strategy self-adaptive bat algorithm | |
| CN113992770A (en) | Cooperative caching method based on policy-based federal reinforcement learning in fog wireless access network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |