CN113055229B - Wireless network self-selection protocol method based on DDQN - Google Patents

Abstract

The invention relates to a DDQN-based wireless network self-selection protocol method, which aims at the situations of complex current wireless network environment and integration of a plurality of protocols. The method comprises the following steps: acquiring current network environment quality parameters and determining node service types in real time through an environment agent module; noise reduction and normalization are carried out on the data on the basis of the step 1), the node service type is determined through an analytic hierarchy process, and feature extraction is carried out; based on the step 2), data is input into a DDQN decision network for real-time training, and an execution result is applied to enable the network state to tend to be stable. According to the method, the data is directly subjected to feature extraction without preprocessing, the obtained historical data is used as training data, and the strong advantage of deep learning is utilized, so that the learning speed and decision performance of the reinforcement learning algorithm are effectively improved.

Description

A method of wireless network self-selection protocol based on DDQN

Technical field

The invention relates to a method for self-selecting network protocols in heterogeneous wireless networks, aiming at the current complex wireless network environment and the integration of many protocols.

Background technique

With the continuous development of network technology, network technologies widely used in the world today have a large amount of overlap. In the current network environment, WLAN and cellular networks are the most common heterogeneous network combinations, and they also play an important role in modern information communications. Operators have also deployed their own WLAN hotspots in user-dense areas such as shopping malls, schools, and office buildings to disperse the pressure caused by cellular networks.

The next generation heterogeneous network is a network that integrates multiple protocols but has a complex environment. It needs to provide reliable network services to users at any time and anywhere. But before this can be realized, the network environment needs to mature, and functions such as wireless network coverage, network self-configuration, and automatic management of network equipment must be solved. In the existing network environment, it is still difficult to implement the above configuration with a single network protocol. However, some algorithms can be used to achieve comprehensive resource scheduling of the current heterogeneous network. Efficient use of heterogeneous network resource switching will gradually become a research topic. hot spots. With the further development of wireless communications, certain requirements will be placed on the scalability and flexibility of heterogeneous networks.

As a tool that can make decisions that meet the requirements of the development environment in a non-deterministic environment, reinforcement learning can make targeted adjustments according to dynamic changes in the network, making heterogeneous wireless networks a system that automatically adapts to changes in user scenarios. program to optimize the network environment. Reinforcement learning is a type of machine learning. It mainly uses agents to continuously adjust in the environment, and ultimately can maximize a specific indicator (Reward). In wireless networks, due to the movement of nodes, , and the mutual interference between nodes make the network environment complex. Compared with traditional machine learning algorithms, deep reinforcement learning has greater potential and higher accuracy, and can directly extract features without preprocessing the data. , using the obtained historical data as training data, and taking advantage of the powerful advantages of deep learning to effectively improve the learning speed and decision-making performance of the reinforcement learning algorithm.

Contents of the invention

In view of the above characteristics of existing networks, the present invention provides a wireless network self-selecting protocol method based on DDQN (Deep Reinforcement Learning with Double Q-learning). Including: network quality data processing solution; feature extraction solution based on deep learning; network protocol selection solution based on DDQN. The object of the present invention is achieved through the following technical solutions.

A method for wireless network self-selection protocol based on DDQN, which method includes the following steps:

1) Obtain the current network environment quality parameters in real time and determine the node business type through the environment agent module;

2) On the basis of 1), perform noise reduction and normalization processing on the data, determine the node business type through the analytic hierarchy process, and perform feature extraction;

3) On the basis of 2), input the data into the DDQN decision-making network for real-time training, and apply the execution results to stabilize the network state.

1. A method for wireless network self-selection protocol based on DDQN, which is characterized by including the following steps:

Step 1: Obtain the current network environment quality parameters and node business type in real time through the environment agent module to determine the status, action and reward value;

State space definition: The definition of the state space S of a terminal at time t is s _mn ∈S, which represents the state when the terminal m is connected to the nth network and interacts with information in the network; its state space is:

S＝s ₁ ,s ₂ ,…,s _mn (1)

State definition: Use average throughput T, delay D, signal strength P, and node distance W to describe the network state. The network quality Φ is expressed as:

Φ＝T×D×P×W (2)

Action space definition: An action space needs to be set for the agent to choose. The definition of action space is:

A＝a ₁ ,a ₂ ,…, _an (3)

where a _n indicates that a node uses the nth network protocol;

Access service network parameters consist of QoS parameters. Establish a decision matrix for network QoS and solve for parameter weights:

The decision matrix is as shown above, where each element represents the importance of the QoS parameters, as defined in the following table. At the same time, the decision matrix should satisfy m _ij >0; m _ji =1/m _ij ; m _ij =1;

2, 4, 6, and 8 that do not appear in the table are used to represent the intermediate values of adjacent judgments; because in the process of defining reward values, business types are divided into 4 categories, and throughput, delay, and signal strength are considered. attribute, the decision matrix should be defined as a 3*3 matrix, that is, _Mi ∈R _3×3 , where i=1, 2, 3, and 4 respectively represent the four business types of category 1, category 2, category 3, and category 4. , and then establish decision matrices for the four types of services according to the requirements of different service QoS parameters;

According to the current network service type classification standard RFC2474, DSCP is used to determine the attribute value in the service level; DSCP uses the used 6 bits and the unused 2 bits in the service category TOS identification byte in the IP header of each data packet. The IP priority is determined by encoding the value; the IP priority field can be applied to traffic classification. The larger the value, the higher the priority. The value ranges from 0 to 63, which can match 64 levels. According to the size of the level, every seven levels are divided into In the first category, the relationship between business attributes and parameters can be determined by sending the DSCP field in the IP data packet;

For these four types of business, i takes the values 1, 2, 3, and 4 in sequence; the eigenvector corresponding to the largest eigenvalue is normalized, that is Each value in the normalized feature vector is the weight of the corresponding network QoS parameter; in the above four cases, there will be differences in the network parameter requirements of different business types, and these differences will be produced later in the division of reward value weights. Impact; Treat the entire network as a whole, and the ultimate goal will be to optimize the overall network quality by selecting nodes to use protocols. The reward value is a function that is strongly related to the network;

V _t =v ₁ ,v ₂ ,…,v _n (5)

t represents the state information of the network at time t, and V _t is a subset Φ of the network state space. Therefore, for a specific business B, the network space state V _t , the reward function R is expressed as, and will be solved in the next step:

R＝f _B (V _t ) (6)

The access of nodes will affect changes in network parameters. After an action is executed, the network status needs to be measured and corresponding rewards fed back; when the action performed leads to an increase in network throughput, a reduction in latency, and an increase in signal strength, it is an effective action. ; On the contrary, when the action performed results in a decrease in network throughput, delay, and signal strength, it is an invalid action; therefore, the average throughput α _avg , average delay β _avg , and signal strength γ are considered when calculating the reward;

Step 2: Normalize the data based on 1), determine the node business type, and determine the reward function;

Use min-max standardization to eliminate the impact of different data units:

Use the above equations for normalization to obtain the normalized network average throughput f _t (α) _avg , average delay f _t (β) _avg , and signal strength f _t (γ);

Combining the above formulas, we get the reward function:

R＝ω ₁ f _t (α) _avg +ω ₂ f _t (β) _avg +ω ₃ f _t (γ) (8)

Among them, ω ₁ , ω ₂ , and ω ₃ are the weights of the average network throughput, delay, and signal strength corresponding to the eigenvectors after normalization of the decision matrix;

Step 3: Based on 2), input the data into the DDQN decision-making network for real-time training, and apply the execution results to stabilize the network state;

First initialize the state S and action space A, initialize the Q matrix to a zero matrix, initialize the Q-MainNet network and Q-target network with random parameters θ, θ is the network parameter, Q-MainNet θ is randomly set during initialization, Q-targetθ ^- =0, t represents the current time state. The agent module reads the current network state information S _t and inputs it into the Q-MainNet network. The Q values of different actions in the S _t state are output through the Q-MainNet network; according to ε- Greedy strategy, the Q-MainNet network randomly selects an action a _t ∈ A with probability ε, or selects an action with probability 1-ε The terminal performs corresponding actions in the heterogeneous wireless network. After obtaining the network data, the data is processed into the format required by the algorithm and handed over to the control layer for processing; thereby obtaining the throughput α, delay β, and signal strength γ; and then they are Normalize separately; according to the type of business, obtain the weights of f _t (α) _avg , f _t (β) _avg , and f _t (γ) through the analytic hierarchy process, and then weight the sum to obtain the reward value R; Q-MainNet obtains System status and reward value, through formula (9)

Calculate the reward value, where R _t+1 is the reward calculated corresponding to the state S _t+1 , and γ is the attenuation coefficient. The reward value of the agent in the current state is actually the conversion of all possible reward values in the future into this The reward value at this moment; after the action is executed, the system enters the next state S _t+1 ;

The Q-MainNet network stores the memory group (s _t , a _t , r _t , s _t+1 ), that is, the current state s _t , the current reward value _{r t} of the action space a t, and the t+1 network state into the experience pool. At each step, the Q-target network randomly samples from the experience pool, and together with the output of the Q-MainNet network, calculates the difference in the loss value relative to the parameter θ in the two networks Q, that is, (TargetQ-Q(S _t+1 , a; θ _t )) ² to execute the gradient descent algorithm; after each round of iterations, copy the parameters of the Q-MainNet network to the Q-target network; continue to cycle for training.

Description of the drawings

Figure 1 is an overall flow chart of the method of wireless network self-selection protocol based on DDQN;

Figure 2 DDQN algorithm operation chart;

Detailed ways

The specific steps of the method for wireless network self-selection protocol based on DDQN implemented according to the present invention will be described with reference to Figure 1 as follows:

Step 1: Obtain the current network environment quality parameters and node business type in real time through the environment agent module to determine the status, action and reward value;

To use the reinforcement learning algorithm, you need to define states, actions and reward values, and network quality parameters are input as state values.

State space definition: The state state space S of a terminal at time t is defined as s _mn ∈S, which represents the state when terminal m is connected to the nth network and interacts with information in the network. Its state space is:

S＝s ₁ ,s ₂ ,…,s _mn (1)

Status definition: The description of network indicators in heterogeneous networks usually uses throughput, delay, packet loss rate, network load, etc. to describe network business status, and uses network signal strength, node distance, node power consumption, cost, signal-to-noise ratio To describe user characteristics, this article will use average throughput T, delay D, signal strength P, and node distance W to describe the network status. Then the network quality Φ can be expressed as:

Φ＝T×D×P×W (2)

Action space definition: An action space needs to be set for the agent to choose. The definition of action space is:

A＝a ₁ ,a ₂ ,…, _an (3)

where a _n indicates that a node uses the nth network protocol.

Reward value definition: Each node has its own specific business characteristics when it is created, and will have its own business type. Even in the same network environment, there will be different reward values. Based on actual needs, node business types are divided into the following categories:

1. The real-time requirements are high, the delay must be as low as possible, and the transmission rate needs to be high. If the delay is too high, it will affect business realization. At the same time, a certain throughput is also required to ensure data reliability.

2. It has extremely high requirements for throughput. Compared with business 1, real-time requirements are not strong and requires a large amount of data traffic.

3. It has high latency requirements and needs to deal with network traffic in emergencies, minimize latency, and improve user experience.

4. Just ensure sufficient throughput.

Access service network parameters consist of QoS parameters. Establish a decision matrix for network QoS and solve for parameter weights:

The decision matrix is as shown in the formula, in which each element represents the importance of the QoS parameter, as defined in the table. At the same time, the decision matrix should satisfy m _ij >0; m _ji =1/m _ij ; m _ij =1.

Table 1 Relationship between attributes and parameters

2, 4, 6, and 8 that do not appear in Table 1 are used to represent the intermediate values of adjacent judgments. Since the service types are divided into four categories in the process of defining the reward value, and the three attributes of throughput, delay, and signal strength are considered, the decision matrix should be defined as a 3*3 matrix, that is, _Mi ∈R _3×3 , where i=1, 2, 3, and 4 respectively represent the four service types of Type 1, Type 2, Type 3, and Type 4. Then, a decision matrix is established for the four types of services according to the requirements of different service QoS parameters.

According to the current network service type classification standard RFC2474, the attribute values in the service level are determined through DSCP (Differentiated Services Code Point). DSCP determines the IP priority by encoding the value in the Class of Service TOS identification byte in the IP header of each data packet, using the used 6 bits and the unused 2 bits. The IP priority field can be applied to traffic classification. The larger the value, the higher the priority. The value ranges from 0 to 63, which can match 64 levels. According to the size of the level, every seven levels are divided into one category, and IP data can be sent by The DSCP field in the packet determines the relationship between service attributes and parameters.

For these four types of services, i takes the values 1, 2, 3, and 4 in sequence. Normalize the eigenvector corresponding to the largest eigenvalue, that is Each value in the normalized feature vector is the weight of the corresponding network QoS parameter. In the above four situations, different business types will have different requirements for network parameters, and these differences will have an impact on the division of reward value weights later. Considering the entire network as a whole, the ultimate goal will be to optimize the overall network quality by selecting nodes to use protocols. The reward value is a function that is strongly related to the network.

V _t =v ₁ ,v ₂ ,…,v _n (5)

t represents the state information of the network at time t, and V _t is a subset Φ of the network state space. Therefore, for a specific business B, the network space state V _t , the reward function R is expressed as, and will be solved in the next step:

R＝f _B (V _t ) (6)

The access of nodes will affect the changes of network parameters. After executing the action, the network status needs to be measured and corresponding rewards fed back. When the action performed leads to an increase in network throughput, a reduction in latency, and an increase in signal strength, it is an effective action; conversely, when the action performed results in a decrease in network throughput, latency, and signal strength, it is an invalid action. Therefore, the average throughput α _avg , the average delay β _avg , and the signal strength γ are considered when calculating the reward.

Step 2: Normalize the data based on 1), determine the node business type, and determine the reward function;

The units and values of different network parameters are usually quite different, so they need to be normalized, linearly transform all values, and map the values to [0,1].

Use min-max standardization to eliminate the impact of different data units:

Use the above equations for normalization to obtain the normalized network average throughput f _t (α) _avg , average delay f _t (β) _avg , and signal strength f _t (γ).

Combining the above formulas, we can get the reward function:

R＝ω ₁ f _t (α) _avg +ω ₂ f _t (β) _avg +ω ₃ f _t (γ) (8)

Among them, ω ₁ , ω ₂ , and ω ₃ are the weights of the average network throughput, delay, and signal strength corresponding to the eigenvectors after normalization of the decision matrix.

Step 3: Based on 2), input the data into the DDQN decision-making network for real-time training, and apply the execution results to stabilize the network state.

One of the biggest shortcomings of using DQN is that although the argmax() method can quickly bring the Q value closer to the target, it is likely to lead to overestimation. The so-called overestimation means that the algorithm model we obtain has a large deviation. In order to solve this problem, the error can be eliminated by separating the target Q value calculation and target Q value selection. Network information is in a discrete state, and DDQN can handle data in a discrete state very well.

Refer to Figure 2 to implement it using two neural networks in DQN, namely Q-MainNet and Q-target. Two networks are also used for calculations in DDQN, but the target Q value is calculated in different ways.

First initialize the state S and action space A, initialize the Q matrix to a zero matrix, initialize the Q-MainNet network and Q-target network with random parameters θ, θ is the network parameter, Q-MainNet θ is randomly set during initialization, Q-targetθ ^- =0, t represents the current time state. The agent module reads the current network state information S ₎ and inputs it into the Q-MainNet network. The Q values of different actions in the S _t state are output through the Q-MainNet network. According to the ε-greedy strategy, the Q-MainNet network randomly selects an action a _t ∈ A with probability ε, or selects an action with probability 1-ε The terminal performs corresponding actions in the heterogeneous wireless network. After obtaining the network data, the data is processed into the format required by the algorithm and handed over to the control layer for processing. Thus, the throughput α, delay β, and signal strength γ are obtained. Then normalize them separately. According to the type of business, the weights of f _t (α) _avg , f _t (β) _avg , and f _t (γ) are obtained through the analytic hierarchy process, and then the reward value R is obtained by weighted summation. Q-MainNet obtains the system status and reward value through formula (9)

Calculate the reward value, where R _t+1 is the reward calculated corresponding to the state S _t+1 , and γ is the attenuation coefficient. The reward value of the agent in the current state is actually the conversion of all possible reward values in the future into this The reward value at this moment. After the action is executed, the system enters the next state S _t+1 .

The Q-MainNet network stores the memory group (s _t , a _t , r _t , s _t+1 ), that is, the current state s _t , the current reward value _{r t} of the action space a t, and the t+1 network state into the experience pool. At each step, the Q-target network randomly samples from the experience pool, and together with the output of the Q-MainNet network, calculates the difference in the loss value relative to the parameter θ in the two networks Q, that is, (TargetQ-Q(S _t+1 , a; θ _t )) ² to perform the gradient descent algorithm. Every G steps, the parameters of the Q-MainNet network are copied to the Q-target network. Continuous training cycle.

Claims (1)

1. A method for wireless network self-selection protocol based on DDQN, which is characterized by including the following steps: Step 1: After obtaining the network environment quality parameters and the node's changing business types in real time through the environment agent module, determine the status, action and reward value; State space definition: The definition of the state space S of a terminal at time t is s _mn ∈S, which represents the state when the terminal m is connected to the nth network and interacts with information in the network; its state space is: S＝s _m1 , s _m2 ,..., s _mn #(1) State definition: Use average throughput T, delay D, signal strength P, and node distance W to describe the network state. The network quality Φ is expressed as: Φ＝T×D×P×W#(2) Action space definition: An action space needs to be set for the agent to choose. The definition of action space is: A＝a ₁ , a ₂ ,..., _an #(3) where a _n indicates that a node uses the nth network protocol; Access service network parameters consist of QoS parameters. Establish a decision matrix M for network QoS and solve for parameter weights: The decision matrix is as shown above, in which each element represents the importance of the QoS parameters. The specific definition is as follows. At the same time, the decision matrix should satisfy m _ij >0; m _ji =1/m _ij ; When i and j are equally important, m _ij is 1; When the importance of i and j is compared and i is slightly more important, m _ij is 3; When the importance of i and j is compared and i is important, m _ij is 5; When the importance of i and j is compared and i is very important, m _ij is 7; When the importance of i and j is compared and i is extremely important, m _ij is 9; The 2, 4, 6, and 8 that do not appear are used to represent the intermediate values of adjacent judgments; because in the process of defining the reward value, the business types are divided into 4 categories, and the three attributes of throughput, delay, and signal strength are considered , the decision matrix should be defined as a 3*3 matrix, that is, M _b ∈ R _3×3 , where b = 1, 2, 3, and 4 respectively represent the four business types of category 1, category 2, category 3, and category 4. Then Establish 4 decision matrices for each of the four services according to the requirements of different service QoS parameters; According to the current network service type classification standard RFC2474, DSCP is used to determine the attribute value in the service level; DSCP uses the used 6 bits and the unused 2 bits in the service category TOS identification byte in the IP header of each data packet. The IP priority is determined by encoding the value; the IP priority field can be applied to traffic classification. The larger the value, the higher the priority. The value ranges from 0 to 63, which can match 64 levels. According to the size of the level, every seven levels are divided into In the first category, the relationship between business attributes and parameters can be determined by sending the DSCP field in the IP data packet; For these four types of business, b takes the values 1, 2, 3, and 4 in sequence; the eigenvector corresponding to the largest eigenvalue is normalized, that is Each value in the normalized feature vector is the weight of the corresponding network QoS parameter; in the above four cases, there will be differences in the network parameter requirements of different business types, and these differences will be produced later in the division of reward value weights. Impact; Treat the entire network as a whole, and the ultimate goal will be to optimize the overall network quality by selecting nodes to use protocols. The reward value is a function that is strongly related to the network; V _t =v ₁ , v ₂ ,..., v _n #(5) t represents the state information of the network at time t, and V _t is a subset Φ of the network state space. Therefore, for a specific business B, the network space state V _t , the reward function R is expressed as, and will be solved in the next step: R＝f _B (V _t )#(6) The access of nodes will affect changes in network parameters. After an action is executed, the network status needs to be measured and corresponding rewards fed back; when the action performed leads to an increase in network throughput, a reduction in latency, and an increase in signal strength, it is an effective action. ; On the contrary, when the action performed results in a decrease in network throughput, delay, and signal strength, it is an invalid action; therefore, the average throughput α _avg , the average delay β _avg , and the signal strength γ are considered when calculating the reward; Step 2: Based on the first step, normalize the data, determine the node business type, and determine the reward function; Use min-max standardization to eliminate the impact of different data units: where x′ is the standardized value of x after conversion, where α, β, and γ are evaluated in sequence; Use the above equations for normalization to obtain the normalized average network throughput f _t (α) _avg at time t, the average delay f _t (β) _avg , and the signal strength f _t (γ); Combining the above formulas, we get the reward function: R＝ω ₁ f _t (α)av _g +ω ₂ f _t (β)av _g +ω ₃ f _t (γ)#(8) Among them, ω ₁ , ω ₂ , and ω ₃ are the weights of the average network throughput, delay, and signal strength corresponding to the eigenvectors after normalization of the decision matrix; Step 3: Based on the second step, input the data into the DDQN decision-making network for real-time training, and apply the execution results to stabilize the network state; First initialize the state space S and action space A, initialize the Q matrix to a zero matrix, initialize the Q-MainNet network and Q-target network with random parameters θ, θ is the network parameter, Q-MainNet θ is randomly set during initialization, and Q-target θ ^- =0, t represents the current time state. The agent module reads the current network state information S _t and inputs it into the Q-MainNet network. The Q values of different actions in the S _t state are output through the Q-MainNet network; according to ε-greedy strategy, the Q-MainNet network randomly selects an action a _t ∈A with probability ε, or selects an action with probability 1-ε The terminal performs corresponding actions in the heterogeneous wireless network. After obtaining the network data, it is processed into the format required by the algorithm and handed over to the control layer for processing; thereby obtaining throughput α, delay β, and signal strength γ; and then they are Normalize separately; according to the type of business, obtain the weights of f _t (α) _avg , f _t (β) _avg , and f _t (γ) through the analytic hierarchy process, and then weight the sum to obtain the reward value R; Q-MainNet obtains System status and reward value, through formula (9) Calculate the reward value, where R _t+1 is the reward calculated corresponding to the state S _t+1 , and γ is the attenuation coefficient. The reward value of the agent in the current state is actually the conversion of all possible reward values in the future into this The reward value at this moment; after the action is executed, the system enters the next state S _t+1 ; The Q-MainNet network stores the memory group (s _t , a _t , r _t , s _t+1 ), that is, the current state s _t , the current reward value _{r t} of the action space a t , and the t+1 network state into the experience pool. At each step, the Q-target network randomly samples from the experience pool, and together with the output of the Q-MainNet network, calculates the difference in the loss value relative to the parameter θ in the two networks Q, that is, (TargetQ-Q(S _t+1 , a, θ _t )) ² to execute the gradient descent algorithm; after each round of iterations, copy the parameters of the Q-MainNet network to the Q-target network; continue to cycle for training.