CN113923128A - Intelligent coding caching method based on federal reinforcement learning in fog wireless access network - Google Patents

Intelligent coding caching method based on federal reinforcement learning in fog wireless access network Download PDF

Info

Publication number
CN113923128A
CN113923128A CN202111258088.3A CN202111258088A CN113923128A CN 113923128 A CN113923128 A CN 113923128A CN 202111258088 A CN202111258088 A CN 202111258088A CN 113923128 A CN113923128 A CN 113923128A
Authority
CN
China
Prior art keywords
fog
access point
local
global
fog access
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202111258088.3A
Other languages
Chinese (zh)
Other versions
CN113923128B (en
Inventor
蒋雁翔
陈颖琦
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Southeast University
Original Assignee
Southeast University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Southeast University filed Critical Southeast University
Priority to CN202111258088.3A priority Critical patent/CN113923128B/en
Publication of CN113923128A publication Critical patent/CN113923128A/en
Application granted granted Critical
Publication of CN113923128B publication Critical patent/CN113923128B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/147—Network analysis or design for predicting network behaviour
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Data Mining & Analysis (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Mobile Radio Communication Systems (AREA)
  • Information Transfer Between Computers (AREA)

Abstract

The invention discloses an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network, which comprises the following steps: 1. the cloud center server builds a global prediction model and initializes the global prediction model, and a plurality of distributed learning agents are arranged at the fog access point and initializes the local prediction model; 2. the system observes the global state; 3. each fog access point observes local conditions; 4. the system uses a multicast coding transmission mode to meet the request received by each network edge node according to the implemented content placement decision; 5. each fog access point calculates theoretical time delay and virtual feedback according to local request conditions and stores local learning experience; 6. each fog access point independently performs local model training by using local learning and remembering experience; 7. and each fog access point uploads a local model to the cloud center server every a plurality of time slots to update the global model. The invention can reduce the forward load of the system, reduce the transmission time delay and protect the privacy of users.

Description

Intelligent coding caching method based on federal reinforcement learning in fog wireless access network
Technical Field
The invention belongs to the field of intelligent caching of edge networks in mobile communication systems, and particularly relates to an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network.
Background
With the wide popularization of intelligent devices and the rapid development of mobile applications, mobile communication networks will bear increasingly large traffic loads. In order to handle these increased data traffic, a Fog wireless Access network (Fog Access Point, F-AP) is proposed as a new network architecture, where the Fog wireless Access Point (Fog Access Point, F-AP) is disposed near the network edge of the user and uses its edge storage capability and edge computing capability to reduce the traffic pressure of the cloud center server, thereby improving the user experience. Meanwhile, the coding cache is used as a new cache mode, the local cache and the multicast transmission are effectively combined, and the limited storage space of the edge equipment can be more fully utilized.
However, most of the existing researches on the inconsistent popularity coding caching method are focused on the upper and lower bound analysis of the theoretical performance of the fixed content placement strategy, so that the potential of the coding caching cannot be fully utilized by the non-optimal content placement strategy to reduce the forward load. In addition, the current non-uniform popularity coding and caching method assumes an ideal condition that the popularity of the content is fixed, and ignores the fact that the popularity of the content changes along with time, so that the performance of forwarding load, time delay and the like cannot reach an ideal state in practice. Therefore, there is a need for a coding caching method under the condition of time-varying popularity, which reduces the content transmission delay and improves the stability at the same time, and provides a higher-quality and more reliable communication service for mobile users.
Disclosure of Invention
The invention aims to provide an intelligent coding caching method based on federal reinforcement learning in a fog wireless access network, and aims to solve the technical problems of reducing forward load, reducing request time delay and protecting user privacy.
In order to solve the technical problems, the specific technical scheme of the invention is as follows:
a code caching method based on federal reinforcement learning in a fog wireless access network comprises the following steps:
step 1, constructing a global model theta on a cloud center serverGAnd initializing, arranging a learning agent at each fog access point and initializing its local model θk；
Step 2, at the end of the time slot t, the system observes the global state s (t), and each fog access point observes the local state s thereofk(t)；
Step 3, the system uses the global model to predict and implement the actual code cache content placement decision a (t) according to the observed global state s (t), and each fog access point simultaneously uses the observed local state sk(t) predicting and recording virtual content placement decisions ak(t)；
Step 4, in the time slot t +1, the system places a decision a (t) according to the implemented content and uses a multicast coding transmission mode to meet the request received by each fog access point;
step 5, at the end of time slot t +1, the local state is changed into sk(t +1), each fog access point counts the content request condition of the service user in the time slot t +1 and makes a decision according to the virtual placement thereofk(t) calculating theoretical time delay and virtual feedback rk(t) storing local learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]T；
Step 6, randomly sampling local learning experience of each fog access point for training to realize local model updating;
step 7, every TAIn each time slot, all the fog access points upload the local models to the cloud center server for integration, and download the integrated global model for replacing the local models;
and 8, repeatedly executing the step 2 to the step 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5 percent.
Further, the step 1 specifically includes the following steps:
step 1.1, the cloud center server constructs a neural network Q (s, a; theta)G) Where s is the current global state vector, a is the global content placement decision vector, θGAre global network model parameters and are randomly initialized;
step 1.2, for
Figure BDA0003324748840000021
For the fog access point index set, K is the number of fog access points, and the fog access points K construct a neural network Q(s)k，ak；θk) Wherein s iskFor the current local state vector, akPlacement of decision vectors, θ, for local contentkAre local network model parameters and are randomly initialized.
Further, the step 2 specifically includes the following steps:
step 2.1, at the end of the time slot t, the system observes the global state
Figure BDA0003324748840000031
Where a (t-1) is the global buffering decision made by the t-1 slot system,
Figure BDA0003324748840000032
requesting frequency vectors for statistics of all N files in the t time slot system;
step 2.2, at the same time, for
Figure BDA0003324748840000033
Fog access point k observes its local state
Figure BDA0003324748840000034
Wherein a isk(t-1) local caching decisions for t-1 time slot records,
Figure BDA0003324748840000035
the requested frequency vectors for all N files in the slot tmouch access point k.
Further, the step 3 specifically includes the following steps:
step 3.1, at the end of the time slot t, the system uses the global model to predict and implement the actual coding cache content placement decision according to the observed global state s (t)
Figure BDA0003324748840000036
Wherein N isc(t) represents the number of files of the encoding buffer, cn(t) < 1 > indicates that file n is selected, cnWhen (t) is 0, it means that the file n is not writtenSelecting;
step 3.2, for
Figure BDA00033247488400000312
Fog access point k based on observed local state sk(t) predicting and recording virtual code cache content placement decisions using its local model
Figure BDA0003324748840000037
Wherein
Figure BDA0003324748840000038
Indicating the number of files of the encoding buffer,
Figure BDA0003324748840000039
the presentation file n is selected to indicate that,
Figure BDA00033247488400000310
it indicates that file n is not selected.
Further, the step 3.1 specifically includes the following steps:
step 3.1.1, at the end of the time slot t, the system predicts the actual coding cache content placement decision by using a global model according to the observed global state s (t):
Figure BDA00033247488400000311
step 3.1.2, implementing a content placement strategy: by using
Figure BDA0003324748840000041
Represents a subset of fog access points, wherein
Figure BDA0003324748840000042
Is the number of elements is LtOf fog access points, LtIs a variable and Lt＝KM/Nc(t), M is the buffer size of the fog access point, | □ | represents the number of the collection elements; splitting a file n into
Figure BDA0003324748840000043
Subfiles of the same size
Figure BDA0003324748840000044
Wherein
Figure BDA0003324748840000045
Figure BDA0003324748840000046
Representing a selected set of file indexes, shaped as
Figure BDA0003324748840000047
A combination number representing the number of B elements taken out of A different elements; for the
Figure BDA0003324748840000048
The placement content of the fog access point k is as follows:
Figure BDA0003324748840000049
further, the step 3.2 specifically includes the following steps:
step 3.2.1, for
Figure BDA00033247488400000419
Fog access point k based on observed local state sk(t) according to a greedy action selection strategy, randomly selecting a virtual content placement decision with a probability of epsilon, and predicting the virtual content placement decision with a local model thereof with a probability of 1-epsilon:
Figure BDA00033247488400000410
step 3.2.2, record content Placement strategy ak(t) but not practiced;
further, the step 4 specifically includes the following steps:
step 4.1, each fog access point receives a user request;
step 4.2, for the cached request, use
Figure BDA00033247488400000411
Representing its index set; by using
Figure BDA00033247488400000412
Representing a fog access point set cached with the request file; let
Figure BDA00033247488400000413
Represents a set of mist access point subsets, wherein
Figure BDA00033247488400000414
For the fog access point subset, the cloud center server is connected to
Figure BDA00033247488400000415
The content of the multicast transmission is:
Figure BDA00033247488400000416
wherein the content of the first and second substances,
Figure BDA00033247488400000417
representing a bit exclusive or operation;
step 4.3, for the request which is not cached, the cloud center server sends the request to the server
Figure BDA00033247488400000418
The request content is unicast transmitted.
Further, the step 5 specifically includes:
step 5.1, for
Figure BDA0003324748840000051
The fog access point K averagely divides the received V requests into K parts;
step 5.2, making each request collected by fog access point
Figure BDA0003324748840000052
Is received in fog, where k1，k2，...，kK-1Is a virtually existing virtual fog access point;
step 5.3, collecting station for fog access points
Figure BDA0003324748840000053
The number of buffered requests of the ith request of all K fog access points is expressed as
Figure BDA0003324748840000054
Wherein
Figure BDA0003324748840000055
The ith request representing a virtual fog access point k' is file n,
Figure BDA0003324748840000056
the ith request, representing a virtual fog access point k', is not file n and thus it is loaded from the forward link by
Figure BDA0003324748840000057
Wherein
Figure BDA0003324748840000058
Is a variable and
Figure BDA0003324748840000059
the form min (A, B) represents that the smaller value is selected from A, B;
step 5.4, in the time slot t, the theoretical time delay of the fog access point k is expressed as
Figure BDA00033247488400000510
Wherein d isfTime delay for complete transmission of one file to the cloud center serveraA time delay for complete transmission of a file from the fog access point to the user;
and 5.5, the virtual feedback comprises the following steps:
Figure BDA00033247488400000511
wherein
Figure BDA00033247488400000512
μ1+μ2＝1，0＜μ1＜μ2＜1；
Step 5.6, learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]TStored in a local experience playback pool.
Further, the step 6 specifically includes the following steps:
step 6.1, for
Figure BDA00033247488400000513
Fog access point k randomly draws experience s from within local experience playback poolk(j)，ak(j)，rk(j)，sk(j+1)]T；
Step 6.2, training by using gradient descent and carrying out theta on the local modelkAnd (6) updating.
The intelligent coding caching method based on federal reinforcement learning in the fog wireless access network has the following advantages:
1. aiming at the more challenging scene of the time-varying content popularity in the fog wireless access network, the invention tracks the time-varying popularity by using federal reinforcement learning and adaptively makes a content placement decision, can effectively reduce the forward load, reduce the transmission delay and keep the stable performance, and is closer to the reality in the applicable scene.
2. The invention uses a virtual coding cache method, calculates the theoretical time delay and virtual feedback of a virtual content placement strategy by assuming a virtual fog access point, thereby solving the problem of local training data collection caused by the fact that a single fog access point cannot execute coding cache and realizing distributed local training.
3. The method uses federated learning, fully utilizes the edge computing capability of the fog access node, obtains the global prediction model of the coding cache content placement strategy through distributed training and model integration, and avoids repeatedly uploading training data to the cloud center server, thereby reducing the bandwidth resource waste of a fronthaul link and reducing the privacy disclosure risk of a user.
Drawings
Fig. 1 is a schematic flow chart of a code caching method based on federal reinforcement learning in a fog wireless access network according to the present invention;
fig. 2 is a simulation result diagram of average delay performance of the code caching method based on federal reinforcement learning in the fog wireless access network.
Detailed Description
In order to better understand the purpose, structure and function of the present invention, the following describes an intelligent code caching method based on federal reinforcement learning in a fog radio access network in detail with reference to the accompanying drawings.
The embodiment provides a fog radio access network asynchronous request code caching method, as shown in fig. 1, which includes the following steps:
step 1, constructing a global model theta on a cloud center serverGAnd initializing, arranging a learning agent at each fog access point and initializing its local model θk。
The step 1 specifically comprises the following steps:
step 1.1, the cloud center server constructs a neural network Q (s, a; theta)G) Where s is the current global state vector, a is the global content placement decision vector, θGAre global network model parameters and are randomly initialized;
step 1.2, for
Figure BDA0003324748840000071
For the fog access point index set, K is the number of fog access points, and the fog access points K construct a neural network Q(s)k，ak；θk) Wherein s iskFor the current local state vector, akPlacement of decision vectors, θ, for local contentkIs a local network model parameter andis randomly initialized.
Step 2, at the end of the time slot t, the system observes the global state s (t), and each fog access point observes the local state s thereofk(t)。
The step 2 specifically comprises the following steps:
step 2.1, at the end of the time slot t, the system observes the global state
Figure BDA0003324748840000072
Where a (t-1) is the global buffering decision made by the t-1 slot system,
Figure BDA0003324748840000073
requesting frequency vectors for statistics of all N files in the t time slot system;
step 2.2, at the same time, for
Figure BDA0003324748840000074
Fog access point k observes its local state
Figure BDA0003324748840000075
Wherein a isk(t-1) local caching decisions for t-1 time slot records,
Figure BDA0003324748840000076
the requested frequency vectors for all N files in the slot tmouch access point k.
Step 2.3, the system uses the global model to predict and implement the actual code cache content placement decision a (t) according to the observed global state s (t), and each fog access point simultaneously uses the observed local state sk(t) predicting and recording virtual content placement decisions ak(t)。
The step 3 specifically comprises the following steps:
step 3.1, at the end of the time slot t, the system uses a global model to predict the actual code cache content placement decision according to the observed global state s (t)
Figure BDA0003324748840000081
Figure BDA0003324748840000082
Wherein N isc(t) represents the number of files of the encoding buffer, cn(t) < 1 > indicates that file n is selected, cnIf (t) ═ 0, then file n is not selected. The content placement policy is then enforced: by using
Figure BDA0003324748840000083
Represents a subset of fog access points, wherein
Figure BDA0003324748840000084
Is the number of elements is LtOf fog access points, LtIs a variable and Lt＝KM/Nc(t), M is the buffer size of the fog access point, and the form | □ | represents the number of the collection elements; splitting a file n into
Figure BDA0003324748840000085
Subfiles of the same size
Figure BDA0003324748840000086
Wherein
Figure BDA0003324748840000087
Figure BDA0003324748840000088
Representing a selected set of file indexes, shaped as
Figure BDA0003324748840000089
The number of combinations of B elements taken out of a different elements is indicated. For the
Figure BDA00033247488400000810
The placement content of the fog access point k is as follows:
Figure BDA00033247488400000811
step 3.2, at the same time, for
Figure BDA00033247488400000812
Fog access point k based on observed local state sk(t) selecting a strategy according to a greedy action, randomly selecting a virtual content placement decision with a probability of epsilon, and predicting the virtual content placement decision with a local model thereof with a probability of 1-epsilon
Figure BDA00033247488400000813
Figure BDA00033247488400000814
Record the content placement policy ak(t) but not implemented.
And 4, in the time slot t +1, the system uses a multicast coding transmission mode to meet the request received by each fog access point according to the content placement decision a (t).
The step 4 specifically comprises the following steps:
step 4.1, each fog access point receives a user request;
step 4.2, for the cached request, use
Figure BDA00033247488400000815
Representing its index set. By using
Figure BDA00033247488400000816
Representing a set of fog access points that have cached the request file. Let
Figure BDA0003324748840000091
Represents a set of mist access point subsets, wherein
Figure BDA0003324748840000092
For the fog access point subset, the cloud center server is connected to
Figure BDA0003324748840000093
The content of the multicast transmission is:
Figure BDA0003324748840000094
wherein the content of the first and second substances,
Figure BDA0003324748840000095
representing a bit exclusive or operation;
step 4.3, for the request which is not cached, the cloud center server sends the request to the server
Figure BDA0003324748840000096
The request content is unicast transmitted.
Step 5, at the end of time slot t +1, the local state is changed into sk(t +1), each fog access point counts the content request condition of the service user in the time slot t +1 and makes a decision according to the virtual placement thereofk(t) calculating theoretical time delay and virtual feedback rk(t) storing local learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]T。
The step 5 specifically comprises the following steps:
step 5.2, for
Figure BDA0003324748840000097
The fog access point K averagely divides the received V requests into K parts;
step 5.2, assume that each request is assembled by a fog access point
Figure BDA0003324748840000098
Is received in fog, where k1，k2，...，kK-1Is a virtually existing virtual fog access point;
step 5.3, for fog Access Point set
Figure BDA0003324748840000099
The ith request of all K fog access points in the system, a cached request number tableShown as
Figure BDA00033247488400000910
Wherein
Figure BDA00033247488400000911
The ith request representing a virtual fog access point k' is file n,
Figure BDA00033247488400000912
the ith request, representing a virtual fog access point k', is not file n and thus it is loaded from the forward link by
Figure BDA00033247488400000913
Wherein
Figure BDA00033247488400000914
Is a variable and
Figure BDA00033247488400000915
the form min (A, B) represents that the smaller value is selected from A, B;
step 5.4, in the time slot t, the theoretical time delay of the fog access point k is expressed as
Figure BDA00033247488400000916
Wherein d isfTime delay for complete transmission of one file to the cloud center serveraA time delay for complete transmission of a file from the fog access point to the user;
and 5.5, the virtual feedback comprises the following steps:
Figure BDA0003324748840000101
wherein
Figure BDA0003324748840000102
μ1+μ2＝1，0＜μ1＜μ2＜1；
Step 5.6, learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]TStoring in a local experience playback pool;
and 6, randomly sampling local learning experience of each fog access point for training to realize local model updating.
The step 6 specifically comprises the following steps:
step 6.1, for
Figure BDA0003324748840000103
Fog access point k randomly draws experience s from within local experience playback poolk(j)，ak(j)，rk(j)，sk(j+1)]T；
Step 6.2, training by using gradient descent and carrying out theta on the local modelkAnd (6) updating.
Step 7, every TAAnd in each time slot, all the fog access points upload the local models to the cloud center server for integration, and download the integrated global model for replacing the local models.
The step 7 specifically comprises the following steps:
step 7.1, for
Figure BDA0003324748840000104
The fog access point k will have its local model θkUploading to a cloud center server;
step 7.2, the cloud center server integrates:
Figure BDA0003324748840000105
wherein DkFor fog access point k at TAThe total number of the extracted learning experiences in the training of the time slots;
step 7.3, for
Figure BDA0003324748840000106
The fog access point k downloads the updated global model thetaGAnd make thetak＝θG。
And 8, repeatedly executing the step 2 to the step 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5 percent.
In the simulation results shown in FIG. 2, LFU (Least frequency used) is a conventional Non-coding Caching method, NUCC (Abdollah Ghaffari Sheshjavani et al, authored in 2020 Wireless Communications and network Conference), Coded Cable Under Non-Uniform Content delivery distribution with Multiple Requests, and APCC (Coded Cable architecture Power distribution, authored in 2018, IEEE Transactions on Information Theory) are two coding methods designed for Non-Uniform Popularity, CentralizedCentralizedmethod is a Centralized extension method of the present invention method, and Propoad is the present patent method. Compared with LFU, NUCC, APCC and other methods, the method realizes more stable and lower-delay content transmission, and has no obvious performance loss compared with a centralized extension method.
It is to be understood that the present invention has been described with reference to certain embodiments, and that various changes in the features and embodiments, or equivalent substitutions may be made therein by those skilled in the art without departing from the spirit and scope of the invention. In addition, many modifications may be made to adapt a particular situation or material to the teachings of the invention without departing from the essential scope thereof. Therefore, it is intended that the invention not be limited to the particular embodiment disclosed, but that the invention will include all embodiments falling within the scope of the appended claims.

Claims (9)

1. A code caching method based on federal reinforcement learning in a fog wireless access network is characterized by comprising the following steps:
step 1, constructing a global model theta on a cloud center serverGAnd initializing, arranging a learning agent at each fog access point and initializing its local model θk；
Step 2, at the end of the time slot t, the system observes the global state s (t), and each fog access point observes the local state thereofsk(t)；
Step 3, the system uses the global model to predict and implement the actual code cache content placement decision a (t) according to the observed global state s (t), and each fog access point simultaneously uses the observed local state sk(t) predicting and recording virtual content placement decisions ak(t)；
Step 4, in the time slot t +1, the system places a decision a (t) according to the implemented content and uses a multicast coding transmission mode to meet the request received by each fog access point;
step 5, at the end of time slot t +1, the local state is changed into sk(t +1), each fog access point counts the content request condition of the service user in the time slot t +1 and makes a decision according to the virtual placement thereofk(t) calculating theoretical time delay and virtual feedback rk(t) storing local learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]T；
Step 6, randomly sampling local learning experience of each fog access point for training to realize local model updating;
step 7, every TAIn each time slot, all the fog access points upload the local models to the cloud center server for integration, and download the integrated global model for replacing the local models;
and 8, repeatedly executing the step 2 to the step 7 until the predicted performance fluctuation of the global model in the continuous time slot is less than 5 percent.
2. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 1 specifically comprises the following steps:
step 1.1, the cloud center server constructs a neural network Q (s, a; theta)G) Where s is the current global state vector, a is the global content placement decision vector, θGAre global network model parameters and are randomly initialized;
step 1.2, for
Figure FDA0003324748830000021
Figure FDA0003324748830000022
For the fog access point index set, K is the number of fog access points, and the fog access points K construct a neural network Q(s)k，ak；θk) Wherein s iskFor the current local state vector, akPlacement of decision vectors, θ, for local contentkAre local network model parameters and are randomly initialized.
3. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 2 specifically comprises the following steps:
step 2.1, at the end of the time slot t, the system observes the global state
Figure FDA0003324748830000023
Where a (t-1) is the global buffering decision made by the t-1 slot system,
Figure FDA0003324748830000024
requesting frequency vectors for statistics of all N files in the t time slot system;
step 2.2, at the same time, for
Figure FDA0003324748830000025
Fog access point k observes its local state
Figure FDA0003324748830000026
Wherein a isk(t1) local caching decisions for t-1 slot records,
Figure FDA0003324748830000027
the requested frequency vectors for all N files in the slot tmouch access point k.
4. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 3 specifically comprises the following steps:
step 3.1, at the end of the time slot t, the system uses the global model to predict and implement the actual coding cache content placement decision according to the observed global state s (t)
Figure FDA0003324748830000028
Wherein N isc(t) represents the number of files of the encoding buffer, cn(t) < 1 > indicates that file n is selected, cn(t) ═ 0 then indicates that file n has not been selected;
step 3.2, for
Figure FDA0003324748830000029
Fog access point k based on observed local state sk(t) predicting and recording virtual code cache content placement decisions using its local model
Figure FDA00033247488300000210
Wherein
Figure FDA00033247488300000211
Indicating the number of files of the encoding buffer,
Figure FDA00033247488300000212
the presentation file n is selected to indicate that,
Figure FDA00033247488300000213
it indicates that file n is not selected.
5. The code caching method based on federal reinforcement learning in a fog wireless access network as claimed in claim 4, wherein the step 3.1 specifically comprises the following steps:
step 3.1.1, at the end of the time slot t, the system predicts the actual coding cache content placement decision by using a global model according to the observed global state s (t):
Figure FDA0003324748830000031
step 3.1.2, implementing a content placement strategy: by using
Figure FDA0003324748830000032
Represents a subset of fog access points, wherein
Figure FDA0003324748830000033
Is the number of elements is LtOf fog access points, LtIs a variable and Lt＝KM/Nc(t), M is the buffer size of the fog access point, | □ | represents the number of the collection elements; splitting a file n into
Figure FDA0003324748830000034
Subfiles of the same size
Figure FDA0003324748830000035
Wherein
Figure FDA0003324748830000036
Figure FDA0003324748830000037
Representing a selected set of file indexes, shaped as
Figure FDA0003324748830000038
A combination number representing the number of B elements taken out of A different elements; for the
Figure FDA0003324748830000039
The placement content of the fog access point k is as follows:
Figure FDA00033247488300000310
6. the code caching method based on federal reinforcement learning in a fog radio access network as claimed in claim 4, wherein said step 3.2 specifically comprises the following steps:
step 3.2.1, for
Figure FDA00033247488300000311
Fog access point k based on observed local state sk(t) according to a greedy action selection strategy, randomly selecting a virtual content placement decision with a probability of epsilon, and predicting the virtual content placement decision with a local model thereof with a probability of 1-epsilon:
Figure FDA00033247488300000312
step 3.2.2, record content Placement strategy ak(t) but not implemented.
7. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 4 specifically comprises the following steps:
step 4.1, each fog access point receives a user request;
step 4.2, for the cached request, use
Figure FDA0003324748830000041
Representing its index set; by using
Figure FDA0003324748830000042
Representing a fog access point set cached with the request file; let
Figure FDA0003324748830000043
Represents a set of mist access point subsets, wherein
Figure FDA0003324748830000044
As fog access pointSubset, cloud-centric Server to
Figure FDA0003324748830000045
The content of the multicast transmission is:
Figure FDA0003324748830000046
wherein the content of the first and second substances,
Figure FDA0003324748830000047
representing a bit exclusive or operation;
step 4.3, for the request which is not cached, the cloud center server sends the request to the server
Figure FDA0003324748830000048
The request content is unicast transmitted.
8. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 5 specifically comprises:
step 5.1, for
Figure FDA0003324748830000049
The fog access point K averagely divides the received V requests into K parts;
step 5.2, making each request collected by fog access point
Figure FDA00033247488300000410
Is received in fog, where k1，k2，...，kK-1Is a virtually existing virtual fog access point;
step 5.3, for fog Access Point set
Figure FDA00033247488300000411
The number of buffered requests of the ith request of all K fog access points is expressed as
Figure FDA00033247488300000412
Wherein
Figure FDA00033247488300000413
The ith request representing a virtual fog access point k' is file n,
Figure FDA00033247488300000414
the ith request, representing a virtual fog access point k', is not file n and thus it is loaded from the forward link by
Figure FDA00033247488300000415
Wherein
Figure FDA00033247488300000416
Is a variable and
Figure FDA00033247488300000417
the form min (A, B) represents that the smaller value is selected from A, B;
step 5.4, in the time slot t, the theoretical time delay of the fog access point k is expressed as
Figure FDA00033247488300000418
Wherein d isfTime delay for complete transmission of one file to the cloud center serveraA time delay for complete transmission of a file from the fog access point to the user;
and 5.5, the virtual feedback comprises the following steps:
Figure FDA0003324748830000051
wherein
Figure FDA0003324748830000052
μ1+μ2＝1，0＜μ1＜μ2＜1；
Step 5.6, learning experience [ s ]k(t)，ak(t)，rk(t)，sk(t+1)]TStored in a local experience playback pool.
9. The code caching method based on federal reinforcement learning in a fog radio access network according to claim 1, wherein the step 6 specifically comprises the following steps:
step 6.1, for
Figure FDA0003324748830000053
Fog access point k randomly draws experience s from within local experience playback poolk(j)，ak(j)，rk(j)，sk(j+1)]T；
Step 6.2, training by using gradient descent and carrying out theta on the local modelkAnd (6) updating.
CN202111258088.3A 2021-10-27 2021-10-27 Intelligent coding caching method based on federal reinforcement learning in fog wireless access network Active CN113923128B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202111258088.3A CN113923128B (en) 2021-10-27 2021-10-27 Intelligent coding caching method based on federal reinforcement learning in fog wireless access network

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202111258088.3A CN113923128B (en) 2021-10-27 2021-10-27 Intelligent coding caching method based on federal reinforcement learning in fog wireless access network

Publications (2)

Publication Number Publication Date
CN113923128A true CN113923128A (en) 2022-01-11
CN113923128B CN113923128B (en) 2024-02-13

Family

ID=79243126

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202111258088.3A Active CN113923128B (en) 2021-10-27 2021-10-27 Intelligent coding caching method based on federal reinforcement learning in fog wireless access network

Country Status (1)

Country Link
CN (1) CN113923128B (en)

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108184252A (en) * 2017-11-28 2018-06-19 东南大学 A kind of asynchronous request code cache method of mist wireless access network
CN111314862A (en) * 2020-02-19 2020-06-19 东南大学 Caching method with recommendation under deep reinforcement learning in fog wireless access network
CN111340277A (en) * 2020-02-19 2020-06-26 东南大学 Popularity prediction model and method based on federal learning in fog wireless access network
CN111935784A (en) * 2020-08-12 2020-11-13 重庆邮电大学 Content caching method based on federal learning in fog computing network
CN112579544A (en) * 2020-12-18 2021-03-30 北京邮电大学 File caching method and device, electronic equipment and storage medium
CN113255004A (en) * 2021-06-16 2021-08-13 大连理工大学 Safe and efficient federal learning content caching method
CN113382059A (en) * 2021-06-08 2021-09-10 东南大学 Collaborative caching method based on federal reinforcement learning in fog wireless access network

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108184252A (en) * 2017-11-28 2018-06-19 东南大学 A kind of asynchronous request code cache method of mist wireless access network
CN111314862A (en) * 2020-02-19 2020-06-19 东南大学 Caching method with recommendation under deep reinforcement learning in fog wireless access network
CN111340277A (en) * 2020-02-19 2020-06-26 东南大学 Popularity prediction model and method based on federal learning in fog wireless access network
CN111935784A (en) * 2020-08-12 2020-11-13 重庆邮电大学 Content caching method based on federal learning in fog computing network
CN112579544A (en) * 2020-12-18 2021-03-30 北京邮电大学 File caching method and device, electronic equipment and storage medium
CN113382059A (en) * 2021-06-08 2021-09-10 东南大学 Collaborative caching method based on federal reinforcement learning in fog wireless access network
CN113255004A (en) * 2021-06-16 2021-08-13 大连理工大学 Safe and efficient federal learning content caching method

Also Published As

Publication number Publication date
CN113923128B (en) 2024-02-13

Similar Documents

Publication Publication Date Title
CN110312231B (en) Content caching decision and resource allocation optimization method based on MEC in Internet of vehicles
CN111031102B (en) Multi-user, multi-task mobile edge computing system cacheable task migration method
Baştuğ et al. Proactive small cell networks
CN109218747B (en) Video service classification caching method based on user mobility in super-dense heterogeneous network
CN111935784B (en) Content caching method based on federal learning in fog computing network
CN105979274B (en) The distributed caching laying method of dynamic self-adapting video stream media
CN110580199B (en) Service migration method based on particle swarm in edge computing environment
CN112954385A (en) Self-adaptive shunt decision method based on control theory and data driving
CN113255004A (en) Safe and efficient federal learning content caching method
CN114863683B (en) Heterogeneous Internet of vehicles edge computing unloading scheduling method based on multi-objective optimization
CN113012013A (en) Cooperative edge caching method based on deep reinforcement learning in Internet of vehicles
Huang et al. A more refined mobile edge cache replacement scheme for adaptive video streaming with mutual cooperation in multi-mec servers
CN115720237A (en) Caching and resource scheduling method for edge network self-adaptive bit rate video
CN109672626B (en) Service aggregation method based on queuing delay utilization
Wang et al. An efficient content popularity prediction of privacy preserving based on federated learning and wasserstein gan
Liu et al. Mobility-aware video prefetch caching and replacement strategies in mobile-edge computing networks
CN113993168A (en) Multi-agent reinforcement learning-based cooperative caching method in fog wireless access network
CN113923128A (en) Intelligent coding caching method based on federal reinforcement learning in fog wireless access network
Kim et al. Impacts of device caching of content fractions on expected content quality
CN115756873B (en) Mobile edge computing and unloading method and platform based on federation reinforcement learning
CN115587266A (en) Air-space-ground integrated internet intelligent edge caching method
KR101966588B1 (en) Method and apparatus for receiving video contents
CN115484314A (en) Edge cache optimization method for recommending performance under mobile edge computing network
CN115883601A (en) Method, system, equipment and medium for allocating cooperative resources of Internet of vehicles
CN111124298B (en) Mist computing network content cache replacement method based on value function

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant