TR2022014285A2 - Digital twin-based interference reduction system and method in local autonomous networks with dense access points - Google Patents

Abstract

In order to reduce the negative impact on performance due to the interference problem in wireless networks and dense access point positioning, the invention provides the solution that reduces the impact of the most interference, by using reinforcement learning and by performing a comprehensive search process, by choosing the possibility that reduces the interference created by the access points (3) on the devices. It's about the system and method.

Description

DESCRIPTION Digital twin-based interference mitigation system and method in local autonomous networks with dense access points Technical Field Invention; In order to reduce the interference problem that occurs in wireless networks and the negative impact of this problem on performance, which increases with dense access point positioning, instead of adjusting the broadcast power of the access points with myopic solutions, by using reinforcement learning, a comprehensive search process is used to select the probability that gives the solution that reduces the effect of the most interference. It is about the system and method that reduces the interference created by the points on the devices. Known State of the Technique: TPC (Transmit Power Control), one of the CSR (Coordinated Spatial Reuse) methods in scenarios with Multiple Access Points, has been covered in the literature both within the scope of the WiFi standard and in other solutions. The solutions implemented within the scope of the standard are rule-based, and in order to achieve this control, data collected from the physical layer is collected via wireless media and may cause additional load in this area. The transmission powers of the access points used in the standard and found in other solutions are calculated on the access points with a rule-based approach. These solutions struggle with scarce resources at access points such as processor and memory. Although such algorithms can perform well under predefined conditions, they may not be able to adapt to dynamic wireless networks. Another solution is a central controller that adjusts the transmission power and channel selection of access points based on the O Learning algorithm. The state of the network is defined using two-dimensional locations collected from devices. However, this data collection process requires additional communication over the wireless medium, which may cause an overhead. Additionally, calculation from positions alone may not properly represent the enterprise. While the proposed solution produced the desired results, using an offline learning strategy may have resulted in lower interference. Access point and controller-based solutions presented in the literature do not have the infrastructure suitable for applying artificial intelligence learning and do not fully embrace real-time monitoring, bi-directional data and control flows. Some of the available academic resources are; Zhong Z, Kulkarni P, Cao F, Fan Z, Armor S. Issues and challenges in dense WiFi networks. In: 2015 International Wireless Communications and Mobile Computing Fahim M, Sharma V, Cao T-V, Canberk B, Duong TQ. Machine Learning-Based Digital Do-Duy T, Van Huynh D, Dobre OA, Canberk B, Duong TQ. Digital Twin-Aided Intelligent Offloading with Edge Selection in Mobile Edge Computing. IEEE Wireless Khorov E, Kiryanov A, Lyakhov A, Bianchi G. A Tutorial on IEEE 802.11ax High Deng C, Fang X, Han X, Wang X, Yan L, He R, et al. IEEE 802.11be Wi-Fi 7: New Challenges and Opportunities. IEEE Communications Surveys & Tutorials [Internet]. Aio K. Coordinated Spatial Reuse Performance Analysis [Internet]. 2019. Available from: performance-analysis.pptx Wang JJ-M, Ku C-T, Bajko G, Anwyl GA, Feng S, Liu J, et al. MULTI-ACCESS POINT COORDINATED SPATIAL REUSE PROTOCOL AND ALGORITHM [Internet]. European Ak E., Canberk B. FSC: Two-Scale AI-Driven Fair Sensitivity Control for 802.11ax Networks. In: GLOBECOM 2020 - 2020 IEEE Global Communications Conference He C, Hu Y, Chen Y, Fan X, Li H, Zeng B. MUcast: Linear Uncoded Multiuser Video Streaming With Channel Assignment and Power Allocation Optimization. IEEE Zhang Y, Jiang C, Han Z, Yu S, Yuan J. Interference-Aware Coordinated Power Zhao G, Li Y, Xu C, Han Z, Xing Y, Yu S. Joint Power Control and Channel Allocation for Jones W, Eddie Wilson R, Doufexi A, Sooriyabandara M. A Pragmatic Approach to Clear Channel Assessment Threshold Adaptation and Transmission Power Control for Performance Gain in CSMA/CA WLANs. IEEE Transactions on Mobile Computing Zhou C, Yang H, Duan X, Lopez D, Pastor A, Wu Q, et al. Digital Twin Network: Concepts and Reference Architecture [Internet]. IETF Datatracker. 2022 [cited 2022 Apr 24]. Available from: https://datatracker.ietf.org/doc/draft-irtf-nmrg-network-digital-twin- Wu Y, Zhang K, Zhang Y. Digital Twin Networks: A Survey. IEEE Internet of Things Barricelli BR, Casiraghi E, Fogli D. A Survey on Digital Twin: Definitions, Characteristics, Microsoft. DTDL models - Azure Digital Twins [Internet]. Microsoft Docs. 2022 [cited 2022 Apr 24]. Available from: https://docs.microsoft.com/en-us/azure/digital-twins/concepts- models ns-3 | a discrete-event network simulator for internet systems [Internet]. [cited 2022 May 6]. Available from: https://www.nsnam.org/ As a result, due to the negativities explained above and the inadequacy of existing solutions on the subject, it has become necessary to make a development in the relevant technical field. Purpose of the Invention: The invention aims to introduce a structure with different technical features that brings a new initiative in this field, unlike the structures used in the current technique. The primary purpose of the invention is; In order to reduce the interference problem that occurs in wireless networks and the negative impact of this problem on performance, which increases with dense access point positioning, the system that reduces the interference created by access points on devices by selecting the possibility that provides the solution that reduces the effect of the most interference by using reinforcement learning, by performing a comprehensive search process, and method is to reveal. The system and method subject to the invention obtain data regarding the interference by recording the packets detected by the access points, without creating additional communication load on the wireless environment, thanks to the agent program deployed at each access point in the physical layer of the proposed architecture. In the invention, a Digital Twin WiFi Network called Digital Twin WiFi Network (DTWN) is used, which provides real-time monitoring and management capabilities. In this invention, the coupling frequency with the Physical Layer is also examined. Additionally, this Digital Twin Network Layer transfers data to the Brain Layer so that it can perform calculations. Thanks to the brain layer in the cloud, Q-learning-based transmission power control, which can continuously learn by interacting with the digital network, ensures compliance with the dynamism of the physical network. By performing the calculation in the cloud instead of access points, the resource problem is also avoided. In order to fulfill the purposes described above, the invention was designed to reduce the interference problem occurring in wireless networks and the negative impact on performance due to dense access point positioning. By using reinforcement learning and performing a comprehensive search process, the possibility that provides the solution that reduces the effect of the most interference is selected and the access points created on the devices. It is a system that reduces interference and its feature is; o The physical network through which users communicate, o The station consisting of fixed or portable devices that have the ability to use certain protocols, 0 The access point, which consists of a network hardware device that enables the connection of other Wi-Fi devices to a wired network, 0 The access point records the packets detected by the access point and communicates with the controller agent application that provides communication, o Cloud consisting of a flexible online computing resource that is shared among users and can be scaled at any time, o Controller that performs all procedures and modules in the system, o Digital twin network layer that creates an interference-based representation of the physical network layer, o Digital twin with the physical network layer 0 Southern interface that provides communication between the digital twin layer, 0 Digital twin group containing digital twins, o Digital twin that creates a realistic virtual representation of the physical entity, 0 Northern interface that provides communication between the digital twin layer and the brain layer, 0 Low cost and less service impact on real networks o The brain layer where applications that can effectively run on a digital twin network platform to implement traditional or innovative network operations and make requests that must be handled by the digital twin network are deployed, o Access control that decides whether procedures need to be repeated, o Infers meaning from objects and infers the network topology topology extraction module, o Q-Learning based transmission power control, which tries to find the adjustment that reduces the interference. o Network state creation module, which creates the network state by using the requirement table, performance table and topology. 0 Reward function module, which creates rewards by looking at the difference between the network states. It contains a reinforcement learning agent that updates the Q Table and determines action according to the greed rate. The structural and characteristic features and all the advantages of the invention will be more clearly understood thanks to the figures given below and the detailed explanation written by making references to these figures, and therefore the evaluation should be made taking these figures and detailed explanation into consideration. Figures to Help Understand the Invention Figure 1 is a representation of the physical layer. Figure 2 is a representation of the digital twin network layer. Figure 3 is a representation of the brain layer. Figure 4 is a schematic representation of the method that is the subject of the invention. Figure 5 is a general representation of the system that is the subject of the invention. Drawings do not necessarily have to be scaled, and details that are not necessary to understand the present invention may be omitted. Furthermore, elements that are at least substantially identical or have at least substantially identical functions are designated by the same number. Description of Part References 1. Physical Network 2. Station 3. Access Point 4. Agent application. Cloud 6. Controller 7. Digital Twin Network Layer 8. South Interface 9. Digital Twin Aggregate. Digital Twin 11. North Interface 12. Brain Layer 13. Access Control Module 14. Topology Extraction Module. Q-Learning based Transmission Power Control Agent 17. Reward Function Module D. Digital twin data BA. Information flow GD. Return Detailed Description of the Invention In this detailed description, the preferred embodiments of the invention are explained only for a better understanding of the subject and in a way that does not create any limiting effect. Meet; In order to reduce the interference problem that occurs in wireless networks and the negative impact of this problem on performance, which increases with dense access point positioning, the system that reduces the interference created by access points on devices by selecting the possibility that provides the solution that reduces the effect of the most interference by using reinforcement learning, by performing a comprehensive search process, and It's about the method. The functions of the elements used in the system that is the subject of the invention are as follows; Physical network (1) is the physical network through which users communicate. Stations (2) are fixed or portable devices capable of using the 802.11 protocol. The access point (3) is a network hardware device that enables the connection of other Wi-Fi devices to a wired network. Agent application (4) is the application that records the packets detected by the access point (3) and ensures communication with the controller (6). Cloud (5) is a flexible online computing resource that is shared among users and can be scaled at any time. The controller (6) is the structure that carries out all procedures and modules in the system that is the subject of the invention. The digital twin network layer (7) is the interference-based representation of the physical network layer (1). The south interface (8) is the interface that provides communication between the physical network (1) layer and the digital twin layer (7). Digital twin group (9) is the unit where digital twins (10) are located. The digital twin (10) is a realistic virtual representation of the physical entity. The north interface (11) is the interface that provides communication between the digital twin (10) layer and the brain layer (12). The brain layer (12) is the layer where applications that can effectively run on a digital twin network platform to implement traditional or innovative network operations with low cost and less service impact on real networks and make requests that must be handled by the digital twin network are deployed. The access control module (13) is the module that decides whether the procedures need to be repeated. Topology extraction module (14) is the module that extracts meaning from objects and extracts the network topology. The Q-Learning based transmit power control agent (15) is the agent that tries to find the adjustment that reduces the interference. The network state creation module (16) is the module that creates the network state using the requirement table, performance table and topology. The reward function module (17) is the module that creates rewards by looking at the difference between network states. Reinforcement learning agent (18) is a reinforcement learning agent that updates the O Table and determines action according to the greed rate. The working principle of the system that is the subject of the invention is as follows. Agent applications (4) deployed on the access point (3) located in the physical network (1) record the packets detected by the access point (3) from stations (2) connected to it and not, and send them to the cloud along with the detected power value (dBm) and time stamp. (5) sends the predetermined matching frequency f to the digital twin network layer (7) inside the controller (6). The sent data also includes information about the configuration of the access point (3), the stations (2) and the traffic they create. The southern interface (8) located within the digital twin network layer (7) updates the digital twins (10) in the digital twin mass (9) according to the data received, creates new ones and disconnects the stations (2) that have been disconnected from the network. After this process, the digital twin network layer (7) transmits its topology, that is, Gtly, to the brain layer (12) through the northern interface (11). If the access control module (13) detects that a new station (2) has entered the network, the search process for an optimal setting begins in the brain layer (12). The topology extraction module (14) of the brain layer (12) extracts the topology by separating it into signal and interference type graph edges in a way that can be processed by the reinforcement learning agent (18). The network state creation module (16) located in the Q-Learning based transmission power control agent (15) creates the system state with Gt and (p) coming from the digital twin network layer (7). While creating the system state, the stations (2) (performance according to the p value The reinforcement learning agent (18) determines a value between 0-30 dBm for 9 values in action set A. After each action applied, the reward calculation is made by the reward function module (17) by looking at the difference between the system states. While the actions are applied, the topology of the network is determined. It is assumed that it does not change. Therefore, the decision is made with the logic that the change in the performance classes of the stations is related to the initiative value. The desired balance in the network is achieved by multiplying the calculated reward with the reward factor Ã, which is predetermined according to the state of the network. The Q table includes the calculated reward rt, the learning rate 0( and the discount factor It is updated using the formula containing y.Reinforcement learning algorithms work by choosing between two concepts. In the concept of discovery, a random action is selected. In the concept of exploitation, the action that promises the least interference in the table is selected. Which of the action's exploration or exploitation concepts will be used is chosen randomly according to the greed rate. The selected action is applied to the digital twin network layer (7) via the north interface (11). The digital twin network layer (7) transmits the action to the physical network (1) with the feedback flow of the south interface (8). If the action is to do nothing, the optimal solution has been reached and the process is terminated. If not, the access control module (13) continues the system cycle until the optimal solution is found. The processing steps performed with the system that is the subject of the invention are as follows; 0 The agent application (4) deployed on the access point (3) sends the data about the stations (2) it collects to the digital twin network layer (7) in the controller (6) in the cloud (5) with a predetermined matching frequency (1001), 0 North interface (11) receives the data and updates (1002) the digital twins (10) in the digital twins group (9), 0 Transmits the current status of the digital twin network layer (7) to the brain layer (12) (1003), o Adds a new station to the network ( 2) If it is detected that it has entered, the process of searching for an optimal setting begins in the brain layer (12) (1004), 0 The brain layer (7) removes the topology in a way that allows processing (1005), o Q-Learning based transmission power control agent (15). , creating the system state with the data coming from the digital twin network layer (7) thanks to the network state creation module (16) (1006) o After each action applied, the reward calculation is made by the reward function module (17) (1007), o Reinforcement learning agent ( 18) Updating the Q table with the calculated reward (1008), o Reinforcement learning agent (18) using the concept of exploration or exploitation according to the greed rate (1009), o Selecting a random action in the concept of exploration (1010), o Using the concept of exploitation in the table in the concept of exploitation selection of the action that promises less interference (1011), o application of the selected action to the digital twin network layer (7) by the north interface (11) (1012), 0 physical action of the action with the south interface (8) of the feedback flow of the digital twin network layer (7). If the action is "do nothing", it is understood that the optimal solution has been reached and the process is terminated (1014). Problem Formulation In the invention, the WiFi network is defined as an undirected weighted graph G=(V,E,w). Here V is the vertex set. In this cluster, VC are the stations and VAP are the access points (3). E in the graph is a sequence of edges corresponding to the signal reaching the station from an access point (3). In this sequence, the edges formed between VC and VAP are divided into two groups as signal (Es) and interference (Ei). If the signal type edge serves VAP Vc, it is faulty, on the other hand, if there is interference between these two points, an interference type edge occurs. Finally, w is the weight function and its weights are the strength of the signals in question. The quality of wireless communication is measured by a signal-to-interference-plus-noise ratio (SINR). Therefore, it is assumed that SINR can represent users' service quality and therefore performance. However, in this invention, instead of measuring at the station, a signal-interference indicator is defined using G graph. And it is calculated for station corners. A station forms an edge with m different access points (3) whose vertex is VC, APi E VAP. One of these edges must be of signal type and is specified as APm in the formula. (I) = wm - 10 log10 Z 10Wi/10 The w value in decibels indicates the weight of the edge e: (AP i , client). So all the interference signal type is extracted from the weight of the edge to get the ratio. When it comes to total interference, the type of weights are added after converting from dBm to mW. The total value is then converted back into dBm. If there is no interference type edge, the interference is calculated as thermal sound power, i.e. - 100dBm. Requirement Classes Variable (p values provide an understanding of the performance of the corner. It varies according to the traffic characteristics of the client. Therefore, it is necessary to determine how low a value is too low. For this reason, the requirement classes shown in the table have been created. Thanks to the analysis made in the digital twin network layer (7), stations (2) These requirements are divided into classes. Requirement Class Range B 35dB (I) 25dB The level of performance degradation is generally caused by the interference of the communication power of the access points (3) on the stations. The communication power of the access points (3) is specified as eAPi. All access points (3) ) configurations are stated as follows. Here, m value is the number of access points (3) in the network. The aim is to find the optimal (9 (t) vector with stations with a sufficient level (p value). Brain Layer 112) In this layer, the transmission of access points (3) to prevent interference. power adjustment is made.All the following parts are located within the brain layer (12). Access Control Module (13) Whenever a new station (2) enters the network, a delay depending on the mirroring frequency is detected in the brain layer (12). After detection, the search for an optimal configuration begins. In this process, Gt is converted to si and given to the reinforcement learning tool. The agent then decides on an action to be taken later. This process is repeated until the decided action is to do nothing. Topology Extraction Module (14) Edges are created by using the detected telemetries together with their values. For example, information about station Cj E and was collected by APi. The power (P) column in the incoming information is adapted as PAPi-›Cj. Thus, the edge e : (APiçcj) with value PApiâcj is placed on the graph if it is higher than a level PAPI<_CjbeIIi when it is of interference type. Q-Learning based Transmission Power Control Area (15) Network Status Creation Module (16) Network status is created using Gt and (p). The stations in the classes are expressed as C'k. Here k is the performance class. Then, how many APi connected stations will receive interference from the APj side. It is determined that it is exposed to it. This is expressed as Il! Module (17) is used for the difference between the states. The reward calculation is made using the Cd and Id matrices and the reward factor. The reward factor is the mapping of the desirability of the change in performance classes. The reward is expressed as follows. In this expression, U is the matrix of ones, the size of the matrix Uc is 3x1, the matrix Ui is Mx1 in size. Since the state of the network does not change during the calculation process, the sum of all Cd values will always be 0. Since the goal is to obtain a sufficient p value, reducing the number of stations (2) in the 3rd performance class is more important than increasing the 1st class. For this reason, the reward factor 11 = [Âl,Âz,/13] should be chosen as follows. /13 |/13 In case of performance class 2, the reward factor is taken as 0 to avoid repetition of the reward. Reinforcement learning Aiani (18) Q table is in the following format. Q: 5 x A -› R The following update formula comes into play after the next state arrives. Q(Sti at) 2 Q(Sti at) + O( [Ft + Y msx Q(St+1i a) _ Q(5ti 30] oi is the learning rate and y is the discount rate.TR TR

Claims (1)

1. CLAIMS In order to reduce the interference problem occurring in wireless networks and the negative impact on performance due to dense access point positioning, we select the possibility that provides the solution that reduces the effect of the most interference by using reinforcement learning and performing a comprehensive search process, and thus reducing the interference created by the access points (3) on the devices. It is a system that provides; The physical network through which users communicate (1), The station consisting of fixed or portable devices that have the ability to use certain protocols (2), The access point (3), which consists of a network hardware device that enables the connection of other Wi-Fi devices to a wired network, The access point ( 3) agent application (4) that records the packets it detects and communicates with the controller (6), cloud (5), which consists of a flexible online computing resource that is shared among users and can be scaled at any time, controller (6), which performs all procedures and modules in the system, physical network Digital twin network layer (7), which forms the interference-based representation of the layer (1), Digital twin group (9) containing the southern digital twins (10) that provide communication between the physical network (1) layer and the digital twin layer (7), A realistic representation of the physical entity The digital twin (10), which creates the virtual representation, and the north, which provides communication between the digital twin (10) layer and the brain layer (12), are active on a digital twin network platform to implement traditional or innovative network operations with low cost and less service impact on real networks. Brain layer (12), where applications that can work in a certain way and make requests that need to be handled by the digital twin network are deployed (12), Access control that decides whether the procedures need to be repeated or not, Topology extraction module (14), which extracts meaning from objects and extracts the network topology, Q- that tries to find the adjustment that reduces the interference. Learning-based transmission power control Network state creation module (16), which creates the network state by using the requirement table, performance table and topology, Reward function module (17), which creates rewards by looking at the difference between network states, Reinforcement learning agent (17), which updates the O Table and determines action according to the greed rate ( 18) is included. It is a method that reduces the interference caused by access points (3) on devices by selecting the solution that reduces the effect of the most interference in order to reduce the interference problem that occurs in wireless networks and the negative impact on performance due to dense access point positioning, by using reinforcement learning and performing a comprehensive search process. , feature; The agent application (4) deployed on the access point (3) sends the data regarding the stations (2) it collects to the digital twin network layer (7) in the controller (6) in the cloud (5) with a predetermined matching frequency (1001), and the North interface ( 11) receive the data and update the digital twins (10) in the digital twins group (9) (1002), transmit the current status of the digital twin network layer (7) to the brain layer (12) (1003), indicate that a new station (2) has entered the network. If detected, an optimal setting search process begins in the brain layer (12) (1004), the brain layer (7) extracts the topology in a way that allows processing (1005), the Q-Learning based transmission power control agent (15), the digital twin network layer (1004), 7) Creating the system state with the data received by the network state creation module (16) (1006). After each value applied, the reward calculation is made by the reward function module (17). (1007). The Q table in the reinforcement learning agent (18) updating with reward (1008), Reinforcement learning agent (18) using the concept of exploration or exploitation according to the greed rate (1009), Selecting a random action in the concept of exploration (1010), Selecting the action that promises the least interference in the table in the concept of exploitation (1011), Selected Application of the action to the digital twin network layer (7) by the northern interface (11) (1012), Transmission of the feedback flow of the digital twin network layer (7) to the physical layer via the southern interface (8) (1013), o Action "doing nothing" If so, it includes the process steps of understanding that the optimal solution has been reached and terminating the process (1014). TR TR