CN117121532A

CN117121532A - Wireless network control based on roaming quality assessment

Info

Publication number: CN117121532A
Application number: CN202280026575.3A
Authority: CN
Inventors: 梅·扎尔·林; 王文峰; 雅各布·托马斯
Original assignee: Juniper Networks Inc
Current assignee: Juniper Networks Inc
Priority date: 2021-09-13
Filing date: 2022-09-13
Publication date: 2023-11-24

Abstract

A technique by which a Network Management System (NMS) is configured to monitor and/or control a wireless network based on one or more roaming quality assessments is described. The NMS determines one or more roaming quality assessments for each roaming event and/or each network device in the wireless network. The roaming quality assessment includes one or more of a roaming delay score, a signal quality score, and/or a stability score. The roaming quality assessment may also include one or more of a sub-optimal roaming score, a sticky client score, and/or an inter-band score. The NMS may further detect one or more ping-pong roaming events or excessive roaming events.

Description

Wireless network control based on roaming quality assessment

Cross Reference to Related Applications

The present application claims the benefit of U.S. provisional application 63/243,414 filed on day 2021, 9 and 13, and claims the benefit of U.S. provisional application 63/268,968 filed on day 2022, 3 and 7, the entire contents of both of which are incorporated herein by reference.

Technical Field

The present disclosure relates generally to computer networks and more particularly to monitoring and control of wireless network performance.

Background

A business location, such as an office, hospital, airport, stadium, or retail store, typically installs a complex wireless network system throughout the location, including a wireless Access Point (AP) network to provide wireless network services to one or more wireless client devices (or simply "clients"). An AP is a physical electronic device that enables other devices to wirelessly connect to a wired network using various wireless network protocols and technologies, such as a wireless local area network protocol (i.e., "Wi-Fi"), bluetooth/bluetooth low energy (ble), a mesh network protocol such as ZigBee, or other wireless network technologies, that conform to one or more of the IEEE 802.11 standards. Many different types of wireless client devices, such as laptop computers, smartphones, tablet computers, wearable devices, appliances, and internet of things (IoT) devices, integrate wireless communication technology and may be configured to connect to a wireless access point in order to access a wired network when the device is within range of a compatible wireless access point. As client devices move throughout the venue, they can automatically switch or "roam" from one wireless access point to another wireless access point within range, providing seamless network connectivity to the user throughout the venue.

Disclosure of Invention

In general, this disclosure describes techniques for monitoring and/or controlling a wireless network based on one or more roaming quality assessments. A Network Management System (NMS) receives network data associated with a plurality of client devices associated with a wireless network. The network data indicates one or more aspects of wireless network performance. The NMS determines one or more roaming quality assessments for each of a plurality of roaming events and/or a plurality of client devices based on the network data. The roaming quality assessment may include a sticky client score, a sub-optimal roaming score, and/or an inter-band score. The NMS may further detect ping-pong and/or excessive roaming events for one or more client devices.

Based on the roaming quality assessment, the NMS may determine a roaming metric for the wireless network. The NMS may further identify an anomaly in the roaming performance of the wireless network, identify a root cause of one or more anomalies in the roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to resolve the identified anomaly.

The disclosed technology provides one or more technical advantages and practical applications. These techniques enable the NMS to automatically monitor and quantify roaming quality, which takes into account other factors in addition to roaming delay (e.g., the amount of time required for a client device to roam from a first AP to a second AP). For example, although roaming delay is a time-based metric, the techniques of this disclosure enable an NMS to consider RSSI measurements of wireless signal strength before and/or after a roaming event in evaluating roaming performance of a wireless network and/or configuring operation of one or more devices in the wireless network (such as one or more APs in the wireless network) to address poor roaming performance. In other examples, the techniques enable the NMS to consider the number of roaming options available during each roaming event and/or the types of roaming options available during each roaming event when assessing wireless network performance and/or configuring operation of one or more devices in the wireless network to address poor roaming performance. In such examples, this technique enables the NMS to actively monitor and analyze roaming events based on one or more factors in addition to or instead of time-based metrics (such as roaming delays). The NMS may further detect ping-pong and/or excessive roaming events for one or more client devices. Based on one or more of the roaming quality assessments, the NMS may further determine a roaming metric indicative of overall roaming performance of the wireless network, identify an anomaly in roaming performance of the wireless network, identify a root cause of one or more anomalies in roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to address the identified anomaly. In this way, from the perspective of the client device, the techniques of this disclosure may enhance wireless network performance, thereby enhancing the user experience of the wireless network.

In one example, the present disclosure relates to a system comprising a plurality of Access Point (AP) devices configured to provide a wireless network at one station; and a network management system, comprising: a memory storing network data received from a plurality of AP devices, the network data collected by a plurality of client devices or a plurality of AP devices associated with a wireless network; and one or more processors coupled to the memory and configured to determine one or more roaming quality assessments for each of a plurality of roaming events in the wireless network based on the network data, wherein the one or more roaming quality assessments include a sub-optimal roaming score for each of the plurality of roaming events based on a Received Signal Strength Indicator (RSSI) associated with the roaming event.

In another example, the present disclosure relates to a method comprising: receiving, by one or more processors of a network management system that manages a plurality of Access Point (AP) devices associated with a wireless network, network data indicative of wireless network performance, the network data collected by a plurality of client devices or a plurality of AP devices associated with the wireless network; and determining one or more roaming quality assessments for each of a plurality of roaming events in the wireless network based on the network data, wherein the one or more roaming quality assessments include a sub-optimal roaming score for each of the plurality of roaming events based on a Received Signal Strength Indicator (RSSI) associated with the roaming event.

In another example, the disclosure relates to a non-transitory computer-readable storage medium comprising instructions that, when executed, configure processing circuitry to receive network data indicative of wireless network performance, the network data collected by a plurality of client devices associated with a wireless network or a plurality of Access Point (AP) devices associated with the wireless network; and determining one or more roaming quality assessments for the wireless network based on the network data, wherein the one or more roaming quality assessments include a sub-optimal roaming score for each of the plurality of roaming events based on a Received Signal Strength Indicator (RSSI) associated with the roaming event.

The details of one or more examples of the technology of the present disclosure are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the techniques will be apparent from the description and drawings, and from the claims.

Drawings

Fig. 1 is a block diagram of an example network system in which a Network Management System (NMS) is configured to monitor and/or control one or more wireless networks based on one or more roaming quality assessments in accordance with one or more techniques of the present disclosure.

Fig. 2 is a block diagram of an example access point device in accordance with one or more techniques of this disclosure.

Fig. 3A and 3B are block diagrams of example network management systems configured to monitor and/or control one or more wireless networks based on one or more roaming quality assessments, in accordance with one or more techniques of the present disclosure.

Fig. 4 is a block diagram of an example user equipment device in accordance with one or more techniques of this disclosure.

Fig. 5 is a block diagram of an example network node (such as a router or switch) in accordance with one or more techniques of this disclosure.

Fig. 6A-6C are graphs illustrating example roaming delay scores in accordance with one or more techniques of the present disclosure.

Fig. 7A is a graph illustrating sub-optimal roaming scores for a plurality of roaming events in a wireless network in accordance with one or more techniques of this disclosure.

Fig. 7B illustrates two scatter plots of inter-band scores for multiple inter-band roaming events in accordance with one or more techniques of the present disclosure.

Fig. 8 is a flow diagram of an example process for a network management system to determine roaming metrics in accordance with one or more techniques of the present disclosure.

Fig. 9 is a flowchart of an example process for a network management system to determine sticky client scores for client devices in a wireless network in accordance with one or more techniques of this disclosure.

Fig. 10 is a flowchart of an example process for a network management system to determine whether one or more roaming quality assessments of a wireless network represent abnormal operation of the wireless network in accordance with one or more techniques of the present disclosure.

11A-11H are screen shots of example user interfaces that may be displayed in accordance with one or more techniques of the present disclosure.

Fig. 12A-12D illustrate example ping-pong roaming events that may be detected by a network management system in accordance with one or more techniques of the present disclosure.

Fig. 12E-12F illustrate examples of excessive roaming events that may be detected by a network management system in accordance with one or more techniques of the present disclosure.

Fig. 13A-13B are flowcharts of example processes by which a network management system according to one or more techniques of the present disclosure detects a ping-pong roaming event or an excessive roaming event, respectively.

Fig. 14 illustrates an example user interface including data indicating one or more ping pong roaming events or one or more excessive roaming events detected for a client device in accordance with one or more techniques of this disclosure.

Detailed Description

A business location, such as an office, hospital, airport, stadium, or retail store, typically installs throughout the location a complex wireless network system that includes a wireless Access Point (AP) network to provide wireless network services to one or more wireless client devices. The client devices may include, for example, smartphones or other mobile computing devices, internet of things (IoT) devices, and the like. Roaming occurs in a wireless network when a wireless client device disassociates itself from one AP and establishes an association with a different AP in order to provide the client device with a seamless network connection throughout the venue. While roaming to a new AP increases the wireless signal power received by the client device, roaming transitions between APs can consume time and resources and can also adversely affect communication performance. To minimize these adverse transition effects, client devices typically employ a fixed roaming threshold. For example, the client device may be configured to find potential roaming candidates only when the signal from the currently associated AP is below a fixed roaming threshold (e.g., -72 dBm). Furthermore, the RSSI of the new wireless signal must be better than the RSSI of the current wireless signal in order for roaming to occur.

The wireless network service provider may collect network data to monitor wireless network behavior and measure one or more aspects of wireless network performance at a site. Network data may be collected from, for example, one or more client devices and/or one or more APs associated with a wireless network. One or more Service Level Experience (SLE) metrics determined based on the collected network data can be used to measure various aspects of wireless network performance. SLE metrics are intended to measure and understand network performance from the perspective of the end user experience on the network. One example SLE metric is coverage metric (coverage metric) that tracks the number of user minutes for which a Received Signal Strength Indicator (RSSI) of a client device measured by an access point associated with the client is below a configurable threshold. Another example SLE metric is a roaming metric (roaming metric) that tracks the percentage of successful roaming of a client between two access points within a specified delay (e.g., time-based) threshold. Other example SLE metrics can include connection time, throughput, successful connection, capacity, AP health, and/or any other metric that can be indicative of one or more aspects of wireless network performance. SLE metrics can also include parameters such as a Received Signal Strength Indicator (RSSI) of the received wireless signal measured by the client device. The threshold may be customized and configured by the wireless network service provider to define the service level expectations at the site. The network service provider can further implement a system that automatically identifies the root cause of any SLE metrics that do not meet a threshold and/or a system that automatically implements one or more remedial actions to address the root cause, thereby automatically improving wireless network performance.

In accordance with one or more techniques of this disclosure, a Network Management System (NMS) monitors and/or controls a wireless network based on one or more roaming quality assessments. The NMS receives network data associated with a plurality of client devices associated with a wireless network. The network data indicates one or more aspects of wireless network performance. The NMS determines one or more roaming quality assessments for each roaming event based on the network data.

The roaming quality assessment may include a sub-optimal roaming score, a sticky client score, and/or an inter-band score for each roaming event and/or client device in the wireless network. The NMS may further detect ping-pong roaming (ping-pong roaming) events and/or excessive roaming (excessive roaming) events for one or more client devices.

In some examples, to determine the sub-optimal roaming score for each roaming event, the NMS determines the current RSSI of wireless signals received by the client device from the currently associated AP (i.e., the re-associated AP to which it roamed). In other examples, the NMS determines a difference (also referred to as delta) between a first RSSI of a first wireless signal received by the client device from a previously associated Access Point (AP) (i.e., a disassociated AP that roams away) and a second RSSI of a second wireless signal received by the client device from a currently associated AP (i.e., a re-associated AP that roams to). For each roaming event, the NMS assigns a sub-optimal roaming score based on the RSSI of the client device when the roaming event is completed. In some examples, the NMS assigns a sub-optimal roaming score based on the RSSI delta associated with the roaming event. To determine the sticky client score for the client device, the NMS assigns the sticky client score based on the number of roaming options available during each roaming event and/or the type of roaming options available during each roaming event. To determine the inter-band score for the client device, the NMS assigns an inter-band score based on the RSSI delta (i.e., the change or difference in RSSI) of the wireless signal received before and after the inter-band roaming event. In some examples, the NMS further detects ping-pong roaming events and/or excessive roaming events for one or more client devices, as described below.

The NMS may further determine one or more statistics based on the roaming quality assessment for the plurality of roaming events in the wireless network. For example, the NMS may determine the percentage of roaming events and/or client devices that are assigned each possible sub-optimal roaming score, each possible sticky client score, and/or each possible inter-band score for one or more defined time periods. The NMS may further combine one or more of the roaming quality evaluations (e.g., sub-optimal roaming score, sticky client score, and/or inter-band score), and/or detection of ping pong or excessive roaming behavior with one or more roaming delay scores to determine a roaming metric indicative of overall roaming performance of the wireless network.

Based on the roaming quality assessment and/or detection of ping-pong and/or excessive roaming behavior of one or more client devices, the NMS may determine a roaming metric for the wireless network. Based on the roaming quality assessment of the one or more client devices and/or detection of ping pong and/or excessive roaming events, the NMS may further identify an anomaly in the roaming performance of the wireless network, identify a root cause of one or more anomalies in the roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to address the identified anomaly.

The disclosed technology provides one or more technical advantages and practical applications. These techniques enable the NMS to automatically monitor and quantify the roaming quality, which takes into account other factors in addition to roaming delay (e.g., the amount of time required for a client device to roam from a first AP to a second AP). For example, although roaming delay is a time-based metric, the techniques of this disclosure enable an NMS to consider RSSI measurements of wireless signal strength before and after a roaming event when evaluating the roaming performance of a wireless network and/or configuring the operation of one or more devices in a wireless network (such as one or more APs in a wireless network) to address poor roaming performance. In other examples, the techniques enable the NMS to consider the number of roaming options available during each roaming event and/or the types of roaming options available during each roaming event when assessing wireless network performance and/or configuring operation of one or more devices in the wireless network to address poor roaming performance. In this way, this technique enables the NMS to actively monitor and analyze roaming events based on one or more factors in addition to or as an alternative to time-based metrics, such as roaming delays. The NMS may further detect ping-pong and/or excessive roaming events for one or more client devices. Based on one or more of the roaming quality assessments, the NMS may further determine a roaming metric indicative of overall roaming performance of the wireless network, identify an anomaly in roaming performance of the wireless network, identify a root cause of one or more anomalies in roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to address the identified anomaly. In this way, wireless network performance from the perspective of the client device may be improved, and thus the user experience of the wireless network may be improved.

Fig. 1 is a diagram of an example network system 100 in which techniques for monitoring and/or controlling a wireless network based on one or more roaming quality assessments are implemented. The example network system 100 includes a plurality of sites 102A-102N, with a network service provider managing one or more wireless networks 106A-106N at the plurality of sites 102A-102N, respectively. Although in FIG. 1, each station 102A-102N is shown to include a single wireless network 106A-106N, respectively, in some examples, each station 102A-102N may include multiple wireless networks, and the disclosure is not limited in this respect.

Each station 102A-102N includes a plurality of Access Points (APs), generally referred to as APs 142. For example, station 102A includes a plurality of APs 142A-1 through 142A-N. Similarly, station 102N includes a plurality of APs 142N-1 through 142N-N. Each AP 142 may be any type of wireless access point including, but not limited to, a business or enterprise AP, a router, or any other device capable of providing wireless network access.

Each site 102A-102N further includes a plurality of client devices (also referred to as user equipment devices (UEs), commonly referred to as UEs 148), representing the various wireless-enabled devices within each site. For example, a plurality of UEs 148A-1 through 148A-N are currently located at site 102A. Similarly, a plurality of UEs 148N-1 through 148N-N are currently located at site 102N. Each UE 148 may be any type of wireless client device including, but not limited to, a mobile device such as a smart phone, tablet or notebook computer, personal Digital Assistant (PDA), wireless terminal, smart watch, smart ring, or other wearable device. UE 148 may also include IoT client devices, such as printers, security devices, environmental sensors, or any other device, configured to communicate over one or more wireless networks.

Example network system 100 further includes various network components for providing network services within a wired network, including, for example, authentication, authorization, and accounting (AAA) server 110 for authenticating users and/or UEs 148, dynamic Host Configuration Protocol (DHCP) server 116 for dynamically assigning network addresses (e.g., IP addresses) to UEs 148 upon authentication, domain Name System (DNS) server 122 for resolving domain names to network addresses, multiple servers 128 (e.g., network servers, database servers, file servers, etc.), and Network Management System (NMS) 150. As shown in fig. 1, various devices and systems of network 100 are coupled together via one or more networks 134 (e.g., the internet and/or an intranet). Each of servers 110, 116, 122 and/or 128, AP 142, UE 148, NMS150, and any other servers or devices attached to or forming part of network system 100 may include a system log or error log module, wherein each of these devices records the status of the device, including a normal operating status and an error condition.

In the example of fig. 1, NMS150 is a cloud-based computing platform that manages wireless networks 106A-106N at one or more sites 102A-102N. As further described herein, NMS150 provides an integrated set of wireless network management tools and implements the various techniques of this disclosure.

In accordance with the techniques described herein, NMS150 accordingly monitors network data associated with wireless networks 106A-106N at each site 102A-102N and manages network resources (such as AP 142 at each site) to deliver high quality wireless network experiences to end users, ioT devices, and clients at the site. Network data 154 may be stored in a database associated with NMS150, such as database 152. In general, NMS150 may provide a cloud-based platform for network data acquisition, monitoring, activity logging, reporting, predictive analysis, network anomaly identification, and alarm generation.

For example, NMS150 may include a Virtual Network Assistant (VNA) 160 that analyzes network data received from one or more UEs 148 and/or one or more APs 142 in a wireless network, provides real-time insight and simplified troubleshooting for IT operations, and automatically takes remedial action or provides advice to proactively address wireless network issues. For example, VNA 160 may include a network data processing platform configured to process hundreds or thousands of concurrent network data flows from UE 148, sensors and/or agents associated with AP 142, and/or nodes within network 134. For example, VNA 160 of NMS150 may include a network performance engine that automatically determines one or more SLE metrics for each client device 148 in wireless network 106. The VNA 160 may also include an underlying analysis and network error recognition engine and an alarm system. The VNA 160 may further provide real-time alarms and reports to notify an administrator or IT personnel of any predicted events, anomalies, trends, and may perform root cause analysis and automatic or assisted error remediation.

In some examples, VNA160 of NMS150 may apply machine learning techniques to identify root causes of error conditions or poor wireless network performance metrics detected or predicted from event data streams. The VNA160 may generate a notification indicating the root cause and/or one or more remedies or remedial actions that may be taken to address the error condition or root cause of the poor wireless network performance metric. In some examples, if the root cause may be automatically resolved, VNA160 invokes one or more remedial or mitigating actions to resolve the error condition or the root cause of the poor wireless network performance metric, thereby automatically improving the underlying wireless network performance metric (e.g., one or more SLE metrics) and also automatically improving the user experience of the wireless network.

Example details of these and other operations implemented by VNA160 and/or NMS150 are described in the following patent applications: U.S. application Ser. No. 14/788,489 entitled "monitoring Wireless Access Point event (Monitoring Wireless Access Point Events)" filed on 6/30/2015; U.S. application Ser. No. 16/835,757 entitled "network System trouble-shooting Using machine learning model (Network System Fault Resolution Using a Machine Learning Model)" filed on 31/3/2020; U.S. application Ser. No. 16/279,243 entitled "System and method for virtual network Assistant Systems and Methods for a Virtual Network Assistant" filed on 2.19.2019; U.S. application Ser. No. 16/237,677, entitled "method and apparatus for facilitating failure detection and/or predictive failure detection (Methods and Apparatus for Facilitating Fault Detection and/or Predictive Fault Detection)" filed on 12/31/2018, and U.S. application Ser. No. 16/251,942, entitled "space-time modeling method (Method for Spatio-Temporal Modeling)" filed on 18/1/2019; U.S. application Ser. No. 16/296,902 entitled "method of transmitting AP error codes over BLE advertisement (Method for Conveying AP Error Codes Over BLE Advertisements)", filed on 3/8/2019; and U.S. application Ser. No. 17/303,222 entitled "virtual network Assistant with active analysis and correlation Engine using an unsupervised ML model" (Virtual Network Assistant Having Proactive Analytics and Correlation Engine Using Unsupervised ML Model), filed on even 24, 5, 2021, which is incorporated herein by reference in its entirety.

In operation, NMS150 observes, collects, and/or receives network data 154. The network data indicates one or more aspects of wireless network performance. For example, the network data 154 may take the form of data extracted from messages, counters, and statistics. Network data may be collected and/or measured by one or more UEs 148 and/or one or more APs 142 in wireless network 106. Some of the network data 154 may be collected and/or measured by other devices in the network system 100. According to one particular implementation, the computing device is part of a network management server 150. According to other implementations, NMS150 may include one or more computing devices, dedicated servers, virtual machines, containers, services, or other forms of environments for performing the techniques described herein. Similarly, the computing resources and components implementing VNA 160 may be part of NMS150, may execute on other servers or execution environments, or may be distributed to nodes (e.g., routers, switches, controllers, gateways, etc.) within network 134.

In accordance with one or more techniques of this disclosure, NMS150 monitors and/or controls wireless networks (such as any of wireless networks 106A-106N) based on one or more roaming quality assessments. For purposes of this disclosure, a roaming event occurs when UE 148 disassociates with a first AP in the wireless network and re-associates with a second AP. NMS150 receives network data (e.g., network data 154) associated with a plurality of client devices (e.g., UE 148) associated with a wireless network (e.g., any of wireless networks 106 a-106N). The network data indicates one or more aspects of wireless network performance.

NMS150 determines one or more roaming quality assessments for each roaming event based on the network data. The roaming quality assessment may include a sub-optimal roaming score, a sticky client score, and/or an inter-band score for each roaming event and/or client device in the wireless network. In some examples, to determine the sub-optimal roaming score for each roaming event, NMS150 determines the current RSSI of wireless signals received by the client device from the currently associated AP (i.e., the re-associated AP to which it roamed). In other examples, NMS150 determines a difference (also referred to as an increment) between a first RSSI of a first wireless signal received by a client device from a previously associated Access Point (AP) (i.e., a disassociated AP that was roamed away) and a second RSSI of a second wireless signal received by the client device from a currently associated AP (i.e., a re-associated AP that was roamed to). For each roaming event, NMS150 assigns a sub-optimal roaming score based on the current RSSI (or RSSI difference) associated with the roaming event. To determine the sticky client score for the client device, NMS150 assigns a sticky client score based on the number of roaming options available during each roaming event and/or the type of roaming options available during each roaming event. To determine the inter-band score of the client device for each roaming event, the NMS assigns an inter-band score based on the RSSI delta (i.e., the change or difference in RSSI) of the wireless signal received before and after the inter-band roaming event. The NMS may also detect ping-pong and/or excessive roaming events for one or more client devices, as described below.

NMS150 may also determine a roaming delay score for each roaming event based on the network data. For example, NMS150 may determine a roaming delay score for each roaming event based on the amount of time required for the client device to perform the roaming event; i.e., the amount of time required to disassociate with the first AP and re-associate with the second AP. In some examples, the roaming delay score may also be based on the type of roaming event. The types of roaming events may include, for example, one or more of standard roaming events, opportunity Key Cache (OKC) roaming events, and/or IEEE fast transition (802.11 r) roaming events.

NMS150 may further determine one or more statistics based on the roaming quality assessment for the plurality of roaming events in the wireless network. For example, NMS150 may determine the percentage of roaming events assigned each possible sub-optimal roaming score and/or the percentage of client devices assigned each possible sticky client score over one or more defined time periods. NMS150 may further combine one or more of the roaming quality assessments (e.g., the sub-optimal roaming score, the sticky client score, and/or the inter-band score) with one or more roaming delay scores to determine a roaming metric indicative of the overall roaming performance of the wireless network. NMS150 may further determine one or more statistics based on detection of ping-pong and/or excessive roaming events for one or more client devices. Based on the roaming quality assessment and/or detection of ping-pong or excessive roaming events, NMS150 may further identify anomalies in the roaming performance of the wireless network, identify root causes of one or more anomalies in the roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to address the identified anomalies.

In general, the roaming quality assessment score may be quantified on a digital scale such that the range of possible scores is meaningful and/or statistically significant. For example, the roaming delay score, sub-optimal roaming score, sticky client score, and/or inter-band score for each client device may be assigned based on a binary scale (such as "good" or "poor"). In other examples, the roaming delay score, sub-optimal roaming score, sticky client score, and/or inter-band score may be quantified on a digital scale, such as:

1: excellent and excellent properties

2: good quality

3: in general

4: poor quality

5: worst case of

Thus, the roaming quality assessment score may be scored, quantified, or categorized in many different ways, and the disclosure is not limited in this respect.

The disclosed technology provides one or more technical advantages and practical applications. These techniques enable NMS150 to automatically monitor and quantify roaming quality, which takes into account other factors in addition to roaming delay (e.g., the amount of time required for a client device to roam from a first AP to a second AP). For example, although roaming delay is a time-based metric, the techniques of this disclosure enable NMS150 to consider RSSI measurements of wireless signal strength before and after a roaming event in evaluating the roaming performance of a wireless network and/or configuring the operation of one or more devices in a wireless network, such as one or more APs in a wireless network, to address poor roaming performance. In other examples, these techniques enable NMS150 to consider the number of roaming options available during each roaming event and/or the types of roaming options available during each roaming event when assessing wireless network performance and/or configuring operation of one or more devices in the wireless network to address poor roaming performance. In this way, this technique enables NMS150 to actively monitor and analyze roaming events based on one or more factors in addition to or instead of time-based metrics (such as roaming delays). NMS150 may further detect ping-pong and/or excessive roaming events for one or more client devices. Based on one or more of the roaming quality assessments, NMS150 may further determine a roaming metric indicative of overall roaming performance of the wireless network, identify anomalies in roaming performance of the wireless network, identify root causes of one or more anomalies in roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to resolve the identified anomalies. The remedial action may include, for example, automatically reconfiguring operation of at least one of the plurality of AP devices 142 in the wireless network. In this way, wireless network performance from the perspective of the client device may be improved, and thus the user experience of the wireless network may be improved.

Although the techniques of this disclosure are described in this example as being performed by NMS150, it should be understood that the techniques described herein may be performed by any other computing device, system, and/or server, and that the disclosure is not limited in this respect. For example, one or more computing devices configured to perform the functions of the techniques of this disclosure may reside in a dedicated server, or may be included in any other server (such as any of servers 128A-128N) other than NMS150, or may be distributed throughout network 100, and may or may not form part of NMS 150.

Additionally or alternatively, in some examples, a network node (e.g., a router or switch in network 134) and/or access point 142 may be configured to locally determine one or more roaming quality assessments in wireless network 106 at station 102, or perform any of the functions of the techniques described in this disclosure.

Fig. 2 is a block diagram of an example Access Point (AP) device 200 configured in accordance with one or more techniques of this disclosure. The example access point 200 shown in fig. 2 may be used to implement any AP 142 shown and described herein with respect to fig. 1. The access point 200 may comprise, for example, a Wi-Fi, bluetooth, and/or Bluetooth Low Energy (BLE) base station or any other type of wireless access point.

In the example of fig. 2, access point 200 includes a wired interface 230, wireless interfaces 220A-220B, one or more processors 206, memory 212, and user interface 210 coupled together via bus 214, various elements may exchange data and information via bus 214. The wired interface 230 represents a physical network interface and includes a receiver 232 and a transmitter 234 for sending and receiving network communications (e.g., packets). The wired interface 230 directly or indirectly couples the access point 200 to the network 134 of fig. 1. First wireless interface 220A and second wireless interface 220B represent wireless network interfaces and include receivers 222A and 222B, respectively, each of which includes a receive antenna via which access point 200 may receive wireless signals from a wireless communication device, such as UE 148 of fig. 1. The first wireless interface 220A and the second wireless interface 220B further include a receiver 224A and a receiver 224B, respectively, each including a transmit antenna via which the access point 200 may transmit wireless signals to a wireless communication device, such as the UE 148 of fig. 1. In some examples, the first wireless interface 220A may include a Wi-fi802.11 interface (e.g., 2.4GHz and/or 5 GHz) and the second wireless interface 220B may include a bluetooth interface and/or a Bluetooth Low Energy (BLE) interface. However, these are given for illustrative purposes only and the disclosure is not limited in this respect.

The processor 206 is a programmable hardware-based processor configured to execute software instructions (such as software instructions for defining software or a computer program) stored in a computer-readable storage medium (such as memory 212), such as a non-transitory computer-readable medium that includes storage instructions to cause the one or more processors 206 to execute one or more techniques described herein (such as a magnetic disk drive or optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory.

Memory 212 includes one or more devices configured to store programming modules and/or data associated with the operation of access point 200. For example, the memory 212 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including a storage device (e.g., a magnetic or optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause the one or more processors 206 to perform one or more of the techniques described herein.

In this example, memory 212 stores executable software including an Application Programming Interface (API) 240, a communication manager 242, configuration settings 250, a device status log 252, and a data store 254. The device status log 252 includes network data, such as a list of network parameters and/or network events, specific to the AP 200 and/or client devices currently or previously associated with the AP 200. The network data may include, for example, any network parameters and/or network data indicative of one or more aspects of the performance of the wireless network or the AP 200 itself. In some examples, the network data may include a plurality of states that are periodically measured as time series data. Network data may be measured by UE device 148 and transmitted to AP 200, may be measured by AP 200 itself or any other device associated with the wireless network, and transmitted to AP 200.

The network data stored in the data store 254 may include, for example, AP events and/or UE events. In some examples, the network events are classified as positive network events, neutral network events, and/or negative network events. The network events may include, for example, memory state, restart event, crash event, ethernet port state, upgrade failure event, firmware upgrade event, configuration change, authentication event, DNS event, DHCP event, one or more types of roaming event, etc., as well as a time and date stamp for each event. The logging controller 255 determines the logging level of the device based on instructions from the NMS 150. Data 254 may store any data used and/or generated by access point 200, including data collected from UE 148 (such as data used to calculate one or more SLE metrics) transmitted by access point 200 for cloud-based management of wireless network 106A by NMS 150. For example, UE 148 executing a roaming event may measure or collect data regarding the number of available roaming options and/or the type of roaming option at the time of the roaming event, a first RSSI of a first wireless signal received from a first AP associated with the roaming event (e.g., a current AP or a disassociated AP), and a second RSSI of a second wireless signal received from a second AP associated with the roaming event (e.g., a re-associated target AP or AP).

Based on the roaming event data in the device status log 252 received from the AP 200, a network management system (such as NMS150 of fig. 1) may identify each roaming event in the wireless network and may further determine when each roaming event occurs, the type of each roaming event, the roaming delay associated with each roaming event, and one or more roaming quality assessments for each roaming event. The roaming quality assessment may include a current RSSI at the completion of the roaming event, an RSSI delta associated with each roaming event, a number of available roaming options per roaming event, and/or a type of available roaming options per roaming event. NMS150 may further determine a sub-optimal roaming score, a sticky client score, and/or an inter-band score. NMS150 may further detect ping-pong and/or excessive roaming events for one or more client devices.

Communication manager 242 includes program code that, when executed by processor 206, allows access point 200 to communicate with UE 148 and/or network 134 via any of interfaces 230 and/or 220A-220B. Configuration settings 250 include any device settings of access point 200, such as radio settings of each of wireless interfaces 220A-220B. These settings may be manually configured or may be monitored remotely and/or managed or configured automatically by NMS150 to periodically (e.g., hourly or daily) optimize wireless network performance.

Input/output (I/O) 210 represents physical hardware components capable of interacting with a user, such as buttons, touch screens, displays, and the like. Although not shown, memory 212 typically stores executable software for controlling a user interface with respect to inputs received via I/O210.

In some examples, rather than NMS150 determining one or more roaming quality assessments, AP 200 may be configured to determine a roaming quality assessment and/or automatically perform other functions based on the roaming quality assessment.

Fig. 3A and 3B are block diagrams of an example Network Management System (NMS) 300 configured to monitor and/or control one or more wireless networks based on one or more roaming quality assessments, in accordance with one or more techniques of the present disclosure. NMS 300 may be used to implement NMS150 in fig. 1, for example. In such examples, the NMS 300 is responsible for monitoring and managing one or more wireless networks 106A-106N at the sites 102A-102N, respectively. In some examples, NMS 300 receives network data measured by network devices such as AP 200 and/or collected by AP 200 from UE 148, such as network data for determining one or more roaming quality assessments, and analyzes this data for cloud-based management of wireless networks 106A-106N. In some examples, NMS 300 may be part of another server shown in fig. 1, or part of any other server.

NMS 300 includes a communication interface 330, one or more processors 306, a user interface 310, memory 320, and database 312. Various elements are coupled together via bus 314, which may exchange data and information via bus 314.

The processor 306 executes software instructions (such as software instructions for defining software or a computer program) stored in a computer-readable storage medium (such as memory 320), such as a non-transitory computer-readable medium including storage devices (e.g., a magnetic disk drive or optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause the one or more processors 306 to perform the techniques described herein.

The communication interface 330 may comprise, for example, an ethernet interface. Communication interface 330 couples NMS 300 to a network and/or the internet, such as any of networks 134 and/or any of the local area networks shown in fig. 1. Communication interface 330 includes a receiver 332 and a receiver 334 by which nms 300 receives/transmits data and information to/from any AP 142, servers 110, 116, 122, 128, and/or any other device or system forming part of network 100 as shown in fig. 1. The data and information received by the NMS 300 may include, for example, network data and/or event log data received from the access point 200, used by the NMS 300 to remotely monitor and/or control the performance of the wireless networks 106A-106N. The NMS may further transmit data to any network device (such as AP 142) at any network site 102A-102N via communication interface 330 to remotely manage wireless networks 106A-106N.

Memory 320 includes one or more devices configured to store programming modules and/or data associated with the operation of NMS 300. For example, memory 320 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including a storage device (e.g., a magnetic or optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause one or more processors 306 to perform the techniques described herein.

In this example, memory 320 includes an API 322, a Virtual Network Assistant (VNA)/AI engine 350, a roaming manager 356, a Radio Resource Management (RRM) engine 358, and a location engine 360. The VNA engine 350 includes a SLE metrics engine 352 and a root cause analysis engine 354.NMS 300 may also include any other programming modules, software engines, and/or interfaces configured for remote monitoring and management of wireless networks 106A-106N, including remote monitoring and management of any AP 142/200.

NMS 300 further includes RRM engine 360 that monitors one or more metrics for each site 106A-106N in order to learn and optimize the RF environment for each site. For example, RRM engine 360 can monitor coverage and capacity SLE metrics of wireless network 106 at sites 102 to identify potential problems with coverage and/or capacity in wireless network 106 and adjust radio settings of access points at each site to address the identified problems. RRM engine 360 may determine the channel and transmit power distribution across all APs 142 in each network 106A-106N. RRM engine 360 may monitor events, power, channels, bandwidth, and the number of clients connected to each AP. RRM engine 360 can further automatically change or update the configuration of one or more APs 142 at site 106 in order to improve coverage and/or capacity SLE metrics and thus provide an improved wireless experience for the user.

The VNA/AI engine 350 analyzes network data received from the AP 142/200 and from other network devices as well as its own data to monitor performance of the wireless networks 106A-106N. For example, the VNA engine 350 may identify when an abnormal or abnormal state is encountered in one of the wireless networks 106A-106N. The VNA/AI engine 350 may use the root cause analysis module 354 to identify the root cause of any abnormal or abnormal state. In some examples, the root cause analysis module 354 utilizes artificial intelligence based techniques to help identify the root cause of any bad SLE metrics at one or more of the wireless networks 106A-106N. Further, the VNA/AI engine 350 can automatically invoke one or more remedial actions aimed at resolving the identified root cause of the one or more poor SLE metrics. Examples of remedial actions that may be automatically invoked by the VNA/AI engine 350 may include, but are not limited to, invoking RRM 360 to restart one or more APs and/or adjust/modify transmit power of a particular radio in a particular AP, adding an SSID configuration to a particular AP, changing channels on an AP or a group of APs, and the like. The remedial action may also include restarting the switch and/or router, invoking a download of new software to the AP, switch or router, etc. These remedial actions are given for example purposes only, and the disclosure is not limited in this regard. If automatic remedial action is not available or sufficient to address the root cause, the VNA/AI engine 350 can proactively and automatically provide notifications that include recommended remedial actions to be taken by IT personnel to address abnormal or anomalous wireless network operation.

SLE metrics engine 352 enables the establishment and tracking of thresholds for one or more SLE metrics for each of wireless networks 106A-106N. SLE metrics engine 352 further analyzes network data (e.g., stored as network data 318) collected by APs and/or UEs associated with wireless networks 106A-106N, such as any AP 142 from UE 148 in each wireless network 106A-106N. For example, APs 142A-1 through 142A-N collect network data from UEs 148A-1 through 148A-N currently associated with wireless network 106A. In addition to any network data collected by one or more APs 142A-1 through 142A-N in wireless network 106A, this data is transmitted to NMS 300 and stored as, for example, network data 318.

NMS 300 executes SLE metrics engine 352 to determine one or more SLE metrics for each UE 148 associated with wireless network 106. One or more SLE metrics can be further aggregated to each AP at a site to learn each AP's contribution to wireless network performance at the site. SLE metrics track whether the service level of each particular SLE metric meets a configured threshold. In some examples, each SLE metric can further include one or more classifiers. If a certain metric does not meet the SLE threshold configured for that site, the failure can be attributed to one of the classifiers to further understand how and/or why the failure occurred.

Example SLE metrics and their classifiers that can be determined by NMS 300 are shown in table 1.

TABLE 1

/>

A more detailed view of SLE metrics engine 352 is shown in fig. 3B. SLE metrics engine 352 includes one or more modules for determining SLE and/or other network performance metrics, including roaming metrics module 370, coverage metrics module 376, throughput metrics module 378, capacity metrics module 380, connection time module 382, successful connection module 384, and AP health module 386.SLE metrics engine 352 can include some or all of these modules and can also include other modules for determining other wireless network performance metrics.

In accordance with one or more techniques of this disclosure, roaming metric module 370 includes a roaming delay module 372, a roaming quality module 374, and a roaming stability module 376. The roaming delay module 372, when executed by one or more processors (such as the processor 306 of fig. 3A), determines and scores each roaming event based on the type of roaming (e.g., standard roaming, OKC roaming, 802.11r roaming, etc.) and the amount of time required for the client to successfully complete the roaming event.

The roaming quality module 374, when executed by one or more processors (such as the processor 306 of fig. 3A), determines and scores one or more roaming quality assessments for each roaming event and/or client device. The roaming quality assessment may include, for example, a sub-optimal roaming score, a sticky client score, and/or an inter-band score. In some examples, to determine a sub-optimal roaming score for each roaming event, the sub-optimal roaming score is based on a current RSSI of wireless signals received by the client device from the currently associated AP (i.e., the re-associated AP to which it roamed). In other examples, the sub-optimal roaming score is based on an RSSI delta defined as the difference between a first RSSI of a first wireless signal received by the client device from a first Access Point (AP) (i.e., a disassociated AP that roams away) and a second RSSI of a second wireless signal received by the client device from a second AP (i.e., a re-associated AP that roams to). In some examples, for each roaming event, the roaming quality module 374 assigns a suboptimal roaming score based on the current RSSI of the wireless signal received by the client device at the completion of the roaming event. In some examples, the roaming quality module 374 assigns a suboptimal roaming score based on the RSSI delta of the current RSSI after the roaming event is completed as compared to the previous RSSI before the roaming event occurred. Further, the roaming quality module 374 assigns a sticky client score to the client device based on the number of roaming options available during each roaming event and/or the type of roaming options available. Further, the roaming quality module 374 assigns an inter-band score to the client device for each roaming event based on the RSSI delta (i.e., the change or difference in RSSI) of the wireless signal received before and after the inter-band roaming event. The roaming quality module 374 may further detect ping pong and/or excessive roaming events for one or more client devices.

The roaming stability module 376, when executed by one or more processors (such as the processor 306 of fig. 3A), determines and scores one or more roaming stability assessments for each roaming event and/or client device. The roaming stability assessment may include, for example, a failure score for fast roaming. Generally, the roaming stability assessment accounts for undesirable roaming events, such as failure of fast roaming.

In some examples, the delay score for each roaming event, the suboptimal roaming score for each roaming event, the sticky client score for each client device, and/or the inter-band score for each roaming event may be assigned based on empirical observations or measurements of wireless network behavior. In other examples, roaming delay module 372 and/or roaming quality module 274 may automatically generate and train one or more supervised and/or unsupervised Machine Learning (ML) models to assign delay scores, sub-optimal roaming scores, sticky client scores, inter-band scores, roaming quality scores, and/or roaming stability scores to each roaming event and/or client device based on network data extracted from historical observation data and/or statistics of the wireless network.

Fig. 4 illustrates an example User Equipment (UE) device 400. The example UE device 400 shown in fig. 4 may be used to implement any UE 148 shown and described herein with respect to fig. 1. UE device 400 may include any type of wireless client device and the disclosure is not limited in this respect. For example, UE device 400 may include a mobile device such as a smart phone, tablet or laptop computer, personal Digital Assistant (PDA), wireless terminal, smart watch, smart ring, or any other type of mobile or wearable device. The UE 400 may also include any type of IoT client device, such as a printer, security sensor or device, environmental sensor, or any other connected device configured to communicate over one or more wireless networks.

In accordance with one or more techniques of this disclosure, network data indicative of one or more aspects of wireless network performance (that is, data used by NMS150 to calculate one or more SLE metrics) is stored as network data 454 in UE memory 412 and transmitted to NMS140/300 via one or more APs 142 in the wireless network. For example, NMS150 receives network data from UEs 148 in networks 106A-106N of FIG. 1. In some examples, NMS150 receives relevant network data from UEs 148 on a continuous basis (e.g., every 2 seconds or other suitable period of time), and the NMS may calculate one or more SLE metrics for each UE on a periodic basis (e.g., every 10 minutes or other predetermined period of time) defined by the first predetermined period of time.

The network data 454 may include, for example, RSSI measurements of one or more wireless signals received by the UE 400 from one or more AP devices as measured by the UE 400. The network data can also include a log of one or more UE association events or states, such as one or more roaming association events, re-association events, roaming away events, disassociation events, association failure events, and any other data or events associated with determining one or more roaming quality assessments and/or other SLE metrics. Some specific messages associated with roaming are 802.11FT grant request, 802.11FT grant response, re-association request response, action frame FT request, FT frame response, etc.

UE device 400 includes wired interface 430, wireless interfaces 420A-420C, one or more processors 406, memory 412, and user interface 410. Various elements are coupled together via a bus 414, which may exchange data and information via the bus 414. The wired interface 430 includes a receiver 432 and a transmitter 434. The UE 400 may be coupled to the network 134 of fig. 1 using a wired interface 430, if desired. The first, second, and third wireless interfaces 420A, 420B, and 420C include receivers 422A, 422B, and 422C, respectively, and each include a receive antenna via which the UE 400 may receive wireless signals from wireless communication devices, such as the AP 142 of fig. 1, the AP 200 of fig. 2, other UEs 148, or other devices configured for wireless communication. The first, second, and third wireless interfaces 420A, 420B, and 420C further include receivers 424A, 424B, and 424C, respectively, and each include a transmit antenna via which the UE 400 may transmit wireless signals to wireless communication devices, such as the AP 142 of fig. 1, the AP 200 of fig. 2, other UEs 138, and/or other devices configured for wireless communication. In some examples, the first wireless interface 420A may include a Wi-fi802.11 interface (e.g., 2.4GHz and/or 5 GHz) and the second wireless interface 420B may include a bluetooth interface and/or a bluetooth low energy interface. The third wireless interface 420C may comprise, for example, a cellular interface through which the UE device 400 may connect to a cellular network.

The processor 406 executes software instructions (such as software instructions for defining software or a computer program) stored in a computer-readable storage medium (such as memory 412), such as a non-transitory computer-readable medium that includes storage instructions to cause the one or more processors 406 to perform the techniques described herein (e.g., a magnetic disk drive or optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory.

Memory 412 includes one or more devices configured to store programming modules and/or data associated with the operation of UE 400. For example, the memory 412 may include a computer-readable storage medium (such as a non-transitory computer-readable medium) including a storage device (e.g., a magnetic disk drive or an optical disk drive) or memory (such as flash memory or RAM) or any other type of volatile or non-volatile memory that stores instructions to cause the one or more processors 406 to perform the techniques described herein.

In this example, memory 412 includes an operating system 440, applications 442, communication modules 444, configuration settings 450, and data store 454. The data store 454 may include, for example, a status/error log including network data specific to the UE 400. As described above, the data 454 may include any network data, events, and/or states that may be relevant to the determination of one or more roaming quality assessments. The network data may include event data such as a log of normal events and error events according to a log level based on instructions from a network management system (e.g., NMS 150/300). The data store 454 may store any data used and/or generated by the UE 400, such as network data for computing one or more SLE metrics, which is collected by the UE 400 and transmitted to any AP 138 in the wireless network 106 for further transmission to the NMS150.

The communication module 444 includes program code that, when executed by the processor 406, enables the UE 400 to communicate using any of the wired interface 430, the wireless interfaces 420A-420B, and/or the cellular interface 450C. Configuration settings 450 include any device settings for UE 400 settings for each of wireless interfaces 420A-420B and/or cellular interface 420C. The roaming module 446 includes program code that, when executed by the processor 406, enables the UE 400 to perform one or more roaming operations in the wireless network 106. For example, the roaming module 446 includes program code that, when executed by the processor 406, enables the UE 400 to find potential roaming candidates among other APs 148 in the wireless network 106 in response to determining that measured roaming parameters (such as RSSI of wireless signals received from the current AP, number of failed connection attempts, etc.) are below a configured roaming threshold. As another example, the roaming module 446 includes program code that, when executed by the processor 406, enables the UE 400 to find potential roaming candidates among other APs 148 in the wireless network 106 in response to receiving a disassociation message from the current AP.

Fig. 5 is a block diagram illustrating an example network node 500 configured in accordance with the techniques described herein. In one or more examples, network node 500 implements a device or server (such as a router, switch, AAA server, DHCP server, DNS server, VNA, network server, etc.) attached to network 134 of fig. 1, or a network device (such as, for example, a router, switch, etc.). In some implementations, the network node 400 of fig. 4 is a router/switch of the servers 110, 116, 122, 128 of fig. 1 or the network 134 of fig. 1.

In this example, network node 500 includes a communication interface 502 (e.g., an ethernet interface), a processor 506, an input/output 508 (e.g., a display, buttons, keyboard, keypad, touch screen, mouse, etc.), a memory 512, and a set of components 516 (e.g., a set of hardware modules, e.g., a set of circuits) coupled together via a bus 509, which various elements may exchange data and information via the bus 509. Communication interface 502 couples network node 500 to a network, such as an enterprise network.

Although only one interface is shown as an example, those skilled in the art will recognize that a network node may, and typically does, have multiple communication interfaces. The communication interface 502 includes a receiver 520 via which the network node 500 (e.g., a server) may receive data and information, including, for example, operation related information such as registration requests, AAA services, DHCP requests, simple Notification Service (SNS) lookup, and web page requests. Communication interface 502 includes a transmitter 522 via which network node 500 (e.g., a server) may transmit data and information, including, for example, configuration information, authentication information, web page data, etc.

Memory 512 stores executable software applications 532, operating system 540, and data/information 530. Data 530 includes a system log and/or error log that stores network data and/or SLE metrics for node 500 and/or other devices, such as wireless access points, based on log levels according to instructions from a network management system. In some examples, network node 500 may forward network data and/or SLE metrics to a network management system (e.g., NMS150 of fig. 1) for analysis as described herein.

Referring again to FIG. 1, a UE (e.g., UE 148A-1) moves around site 102A and thus moves around with respect to APs 142A-1 through 142A-N of wireless network 106A. To perform a roaming event, assume that UE 148A-1 is associated with first AP 142A-1 and that the fixed roaming threshold for UE 148A-1 is-65 dBm. When UE 148A-1 detects that the RSSI of the wireless signal received from first AP 142A-1 reaches-65 dBm, UE 148A-1 initiates a roaming operation by issuing a probe packet to identify potential roaming candidate APs in wireless network 106A. Upon discovering one or more potential roaming candidate APs, the UE 30A disassociates with the first AP 142A-1 and re-associates with one of the potential roaming candidate APs (e.g., the second AP 142A-N) based on one or more defined criteria.

Fig. 6A-6B are graphs illustrating example roaming delay scores for different types of roaming events in accordance with one or more techniques of this disclosure. To assign a roaming delay score to each roaming event, the roaming events are first categorized according to the type of roaming event. Types of roaming events may include, for example, standard (slow) roaming events, OKC (fast) roaming events, and 802.11r (fast) roaming events. In some examples, the OKC roaming event and the 802.11r roaming event are categorized together into a single category as a fast roaming event.

Fig. 6A is a graph illustrating example roaming delay scores for a plurality of slow standard roaming events in accordance with one or more techniques of this disclosure. The figure shows the roaming delay (in seconds) versus the event count, which corresponds to the number of slow standard roaming events that occur within a predetermined time frame.

In the example of fig. 6A, roaming delay scores of 1 to 5 are assigned to the roaming delay of each slow standard roaming event, with roaming delay score "1" being considered the highest or "excellent" roaming delay score and roaming delay score "5" being considered the lowest or "worst" roaming delay score. A roaming delay score of 1 (e.g., "excellent") is assigned to a roaming event of less than 1 second, a roaming delay score of 2 (e.g., "good") is assigned to a roaming event of between 1 and 2 seconds, a roaming delay score of 3 (e.g., "general") is assigned to a roaming event of between 2 and 3 seconds, a roaming delay score of 4 (e.g., "poor") is assigned to a roaming event of between 3 and 4 seconds, and a roaming delay score of 5 (e.g., "worst") is assigned to a roaming event of greater than 4 seconds. Roaming events with delay scores of 3, 4 or 5 correspond correspondingly to relatively long roaming delays, being more likely to be perceived by the user, thereby having an increasingly severe impact on the user experience of the wireless network.

In general, the roaming delay scores of slow standard roaming events are assigned based on their expected relative impact on the user experience of the wireless network. The threshold for the score is determined based on what we consider the system should be able to perform when using a particular roaming method. The number of roaming delay scores for slow standard roaming events may be determined to best represent the roaming behavior of client devices in the wireless network. The threshold for assigning roaming delay scores for slow standard roaming events may be determined based on empirical observations or measurements of roaming behavior or client devices in the wireless network. Further, the threshold may be dynamically updated based on analysis of roaming behavior in the wireless network over time and/or based on applications running on the UE while the roaming process is ongoing. In other examples, NMS150/300 may automatically generate and train one or more supervised and/or unsupervised Machine Learning (ML) models to assign a delay score to each slow standard roaming event based on historical network data and/or statistics of the wireless network. Accordingly, the number of delay scores, the threshold for assigning delay scores to slow standard roaming events, and/or the frequency of score updates are not limited to the specific examples described herein, and the disclosure is not limited in this respect.

Fig. 6B is a graph illustrating example roaming delay scores for a plurality of "fast" roaming events (in this example, 802.11r and OKC roaming events) in accordance with one or more techniques of this disclosure. The figure shows the roaming delay (in seconds) versus event count corresponding to the number of fast roaming events that occur within a predetermined time frame.

In the example of fig. 6B, only roaming delay scores of 1 (e.g., "excellent"), 3 (e.g., "general"), 4 (e.g., "poor"), or 5 (e.g., "worst") are assigned to fast roaming events. A roaming delay score of 1 (e.g., "excellent") is assigned to roaming events less than about 300 milliseconds, a roaming delay score of 3 (e.g., "general") is assigned to roaming events between about 300 and 400 milliseconds, a roaming delay score of 4 (e.g., "worse") is assigned to roaming events between about 400 milliseconds and 1 second, and a roaming delay score of 5 (e.g., "worst") is assigned to roaming events greater than 1 second. Roaming events with delay scores of 3, 4 or 5 correspond correspondingly to relatively long roaming delays, being more likely to be perceived by the user, thereby having an increasingly severe impact on the user experience of the wireless network. In the example of FIG. 6B, there is no general 802.11r/OKC roaming, so this is not shown in the figure.

A roaming delay score is assigned to each fast roaming event. The roaming delay score for a fast roaming event is distributed based on the relative expected distribution time of roaming using a roaming method. The number of roaming delay scores for fast roaming events may be determined to best represent the roaming behavior of client devices in the wireless network. The threshold for assigning a roaming delay score for a fast roaming event may be determined based on empirical observations or measurements of roaming behavior or client devices in the wireless network. Furthermore, the threshold may be dynamically updated based on analysis of roaming behavior in the wireless network over time. In other examples, NMS150/300 may automatically generate and train one or more supervised and/or unsupervised Machine Learning (ML) models to assign a delay score to each fast roaming event based on historical network data and/or statistics of the wireless network. Accordingly, the number of delay scores, the threshold for assigning each delay score to a fast roaming event, and/or the frequency of score updates are not limited to the specific examples described herein, and the disclosure is not limited in this respect.

Fig. 6C is a graph illustrating example roaming delay scores for a plurality of slow standard roaming events in accordance with one or more techniques of this disclosure. The figure shows the roaming delay (in seconds) versus the event count, which corresponds to the number of slow standard roaming events that occur within a predetermined time frame.

In the case of a fast roaming failure event, as shown in the example of fig. 6C, the fact that the fast roaming failure event occurs is generally an unsatisfactory result. That is, the wireless client device may perform fast roaming (e.g., enable fast roaming for the client device), but instead perform slow roaming for some reason. In other words, in these examples, there is no failure event of "excellent" or "good" fast roaming. Thus, a roaming delay score of 3-5 is assigned for each fast roaming failure event, with a roaming delay score of 3 (e.g., "general") assigned to roaming events between 400 milliseconds and 1 second, a roaming delay score of 4 (e.g., "worse") assigned to roaming events between 1 and 2 seconds, and a roaming delay score of 5 (e.g., "worst") assigned to roaming events greater than 2 seconds.

A fast roaming failure event delay score is assigned to each fast roaming failure event. In general, the roaming delay scores for failure events for fast roaming are distributed based on the relative expected distribution time of roaming using the roaming method. The number of roaming delay scores for failure events of fast roaming may be determined to best represent the roaming behavior of client devices in the wireless network. The threshold for assigning a roaming delay score for a failure event of fast roaming may be determined based on empirical observations or measurements of roaming behavior or client devices in the wireless network. Furthermore, the threshold may be dynamically updated based on analysis of roaming behavior in the wireless network over time. In other examples, NMS150/300 may automatically generate and train one or more supervised and/or unsupervised Machine Learning (ML) models to assign a delay score to each fast roaming failure event based on historical network data and/or statistics of the wireless network. Accordingly, the number of delay scores, the threshold for assigning each delay score to a failure event of fast roaming, and/or the frequency of score updates are not limited to the specific examples described herein, and the disclosure is not limited in this respect.

Fig. 7A is a graph illustrating sub-optimal roaming scores for a plurality of roaming events in a wireless network in accordance with one or more techniques of this disclosure. The x-axis represents a previous RSSI (measured by the client device in decibel milliwatts (dBm)) of the first wireless signal received from the first AP associated with the client device at the beginning of the roaming operation (e.g., before the roaming operation occurs). The y-axis represents the RSSI (measured by the client device in decibel milliwatts (dBm)) of the second wireless signal received from the second AP associated with the client device after the roaming operation is completed.

In the example of fig. 7A, roaming events may be scored as shown in the following table.

Current RSSI	Suboptimal roaming score
		>60dBm	3-well
-72dBm to-60 dBm	4-difference
		<-72	5-worst case

Any roaming event that results in an RSSI above the predetermined threshold (in this example, -60 dBm) is considered acceptable and is therefore assigned a sub-optimal roaming score of 3 (e.g., "general"). In other words, in these examples, there are no "excellent" or "good" sub-optimal roaming events. Thus, a sub-optimal roaming score of 3-5 is assigned to each roaming event, where a score of 3 (e.g., "general") is assigned to sub-optimal roaming events resulting in RSSI above a threshold of about-60 dBm, a score of 4 (e.g., "poor") is assigned to roaming events resulting in RSSI between about-72 and-60 dBm, and a score of 5 (e.g., "worst") is assigned to roaming events resulting in RSSI less than about-72 dBm. Roaming events with sub-optimal roaming scores of 3, 4 and 5 correspondingly result in relatively poor RSSI values and have increasingly severe impact on the user experience of the wireless network.

A sub-optimal roaming score is calculated for each roaming event. In general, sub-optimal roaming scores are assigned based on their relative impact on the user experience of the wireless network. The number of sub-optimal roaming scores may be determined to best represent the roaming behavior of the client device in the wireless network. The threshold for assigning a sub-optimal roaming score to each roaming event may be determined based on empirical observations or measurements of roaming behavior or client devices in the wireless network. In the example of fig. 7A, the threshold between sub-optimal roaming scores is constant. However, in other examples, the boundary need not be constant, and the disclosure is not limited in this respect. Furthermore, the threshold may be dynamically updated based on analysis of roaming behavior in the wireless network over time. In other examples, NMS150/300 may automatically generate and train one or more supervised and/or unsupervised Machine Learning (ML) models to assign a sub-optimal roaming score for each roaming event based on historical network data and/or statistics of the wireless network. Thus, the number of sub-optimal roaming scores, the threshold for assigning sub-optimal roaming scores to roaming events, and/or the frequency of sub-optimal roaming score updates are not limited to the specific examples described herein, and the disclosure is not limited in this respect.

In accordance with one or more techniques of this disclosure, one or more roaming evaluations of a client device determined by NMS150/300 may include a sticky client score. For purposes of this disclosure, a sticky client is defined as a client device having a current RSSI value that is less than a fixed roaming threshold of the client device and having one or more roaming options that are at least 7dBm better than the current RSSI value. NMS150/300 may assign a sticky client score to a client device based on the number of roaming options available to the client device and/or the type of roaming options available. For example, NMS150/300 may assign a sticky client score to a client device according to:

in general, the sticky client score is assigned based on the relative "stickiness" of the client devices and indicates the relative severity of the impact of roaming failure on the user experience of the wireless network. For a sticky client classifier, there is no good or excellent score (e.g., score 1 or 2) because a sticky client is undesirable. For example, a sticky client score of "3" is assigned to a client device that meets the sticky client criteria "different APs same/different frequency bands". A sticky client score of "4" (multiple roaming options, different APs the same/different frequency bands) is considered to have a relatively more severe impact on the user experience than a sticky client score of "3". A sticky client score of "5" (a different frequency band for the same AP) is considered to have a relatively more severe impact on the user experience than a sticky client score of "4". A situation where the client device has only one roaming option for the same AP on different frequency bands (e.g., roaming from 2.4GHz to 5GHz in the same AP, and vice versa) may indicate that there is a coverage hole in the wireless network at that location within the site. This case is given a sticky client score of "5" because the presence of coverage holes is highly likely to result in a poor user experience for the wireless network. Remedial action to address the potential coverage hole includes adding one or more additional APs to the wireless network and/or adjusting transmit power balance between one or more APs in the wireless network.

In accordance with one or more techniques of this disclosure, one or more roaming assessments of the wireless client device as determined by NMS150/300 may include an inter-band score. An inter-band score for each of a plurality of client devices is determined based on an RSSI delta (e.g., a change or difference in RSSI) associated with each of the plurality of inter-band roaming events for the client device.

For purposes of this disclosure, inter-band roaming is defined as a wireless client device roaming from a first AP in the 2.4GHz band to a second AP in the 5GHz (or 6 GHz) band, or vice versa from a first AP in the 5GHz (or 6 GHz) band to a second AP in the 2.4GHz band. Inter-band scores are assigned to both types of inter-band roaming events.

In some examples, the threshold for determining the inter-band score for 2.4 to 5/6GHz roaming events is different from the threshold for determining the inter-band score for 5/6 to 2.4GHz roaming events. In examples where the 5/6GHz band is generally more preferred than the 2.4GHz band (due to faster speeds), roaming from 2.4 to 5/6GHz is generally more preferred than roaming from 5/6GHz to 2.4 GHz. In general, for each roaming event, a positive change in RSSI (also referred to as an increment or delta) indicates an improvement in the RSSI of the client device, while a negative change or increment in RSSI indicates a degradation in the RSSI of the client device. The threshold indicates an RSSI delta at which the inter-band score is assigned to one of 1, 2-excellent/good, 3-general, 4-poor, or 5-worst. Although specific example thresholds are described herein, other thresholds may be used and the present disclosure is not limited in this respect. Further, while the thresholds for the 5GHz and 6GHz bands are the same in these examples, in other examples the threshold for roaming to/from the 5GHz band may be different than the threshold for roaming to/from the 6GHz band, and the disclosure is not limited in this respect.

For example, NMS150/300 may assign an inter-band score to a client device according to:

inter-band roaming for different APs 5/6- > 2.4GHz

Score of	Description of the invention
		5	Worst-when the client RSSI is not improved or reduced (RSSI delta)<＝0)
4	Poor-when the client RSSI improves but is below 5dB
		3	Typically-when the client RSSI improves by more than 5dB and less than 10dB
1,2	Good-when the client RSSI improves by more than 10dB

Inter-band roaming for different APs 2.4- & gt 5/6GHz

In some examples, when an inter-band roaming event is between bands of the same AP, the inter-band score/threshold is moved downward by a score value as follows:

inter-band roaming for the same AP 5/6 (2.4 GHz)

Inter-band roaming for the same AP 2.4 (5/6 GHz)

Score of	Description of the invention
		5	Worst-when the client RSSI decreases by more than 15dB
4	Poor-when the client RSSI decreases by more than 10dB and less than 15dB

In these examples, inter-band roaming of the same AP does not have a general or good/excellent score, because in these examples inter-band roaming of the same AP is generally undesirable.

Although specific examples of inter-band roaming events, thresholds, and scores are described herein, different or alternative thresholds and/or scores may be assigned to one or more types of inter-band roaming events, and the disclosure is not limited in this respect. For example, while the 5GHz and 6GHz bands are shown as having the same threshold/score, roaming events from the 2.4GHz band to the 6GHz band may be assigned different thresholds and/or scores than roaming events from the 5GHz band to the 6GHz band, and the disclosure is not limited in this respect.

FIG. 7B illustrates a scatter plot of example inter-band scores for multiple inter-band roaming events. The left side of fig. 7B is a scatter plot of an example inter-band scatter of multiple inter-band roaming events between two different APs from the 5/6GHz band to the 2.4GHz band. The right side of fig. 7B is a scatter plot of an example inter-band roaming score for a plurality of inter-band roaming events between two different APs from the 2.4GHz band to the 5/6GHz band. The threshold between scores is a threshold from the upper table. However, it should be understood that different thresholds may be used, and that the disclosure is not limited in this respect.

A comparison of the two scatter plots shows that, in general, in this example, more inter-band roaming events between two different APs from the 2.4GHz band to the 5/6GHz band are classified as "good" than inter-band roaming events between two different APs from the 5/6GHz band to the 2.4GHz band (left side of fig. 7B).

Possible root causes of general, poor, or worst inter-band scores include, but are not limited to, too few APs at the site, coverage holes at the site, or power imbalance between APs and/or bands at the site. Remedial action to address the potential coverage hole includes adding one or more additional APs to the wireless network and/or adjusting transmit power balance between one or more APs in the wireless network. The remedial action may also include generating a notification that includes adding one or more additional APs to the wireless network and/or a suggestion to adjust a transmit power balance between one or more APs in the wireless network. The remedial action may also include automatically adjusting the transmit power of one or more frequency bands of one or more APs in the wireless network.

Fig. 8 is a flow diagram of an example process (800) for a network management system to determine roaming metrics for wireless client devices in a wireless network in accordance with one or more techniques of this disclosure. The NMS determines the type and delay (e.g., duration) of the roaming event (802). The roaming event types may include, for example, standard (slow) roaming events, OKC (fast) roaming events, and/or 802.11r (fast) roaming events. The NMS assigns a delay score to the roaming event based on the delay of the roaming event (804). The NMS determines the RSSI that initiated the roaming event (806). For example, the RSSI initiating a roaming event may include a first (or "previous" as shown in fig. 6C) RSSI of a first wireless signal received by a client device from a first Access Point (AP) associated with the client at the beginning of the roaming event (i.e., a disassociated AP for roaming away) and a second (or "current" as shown in fig. 6B) RSSI of a second wireless signal received by the client device from a second AP associated with the client device (i.e., a re-associated or target AP) after the roaming event is completed. The NMS assigns a suboptimal roaming score to the roaming event based on the RSSI corresponding to the roaming event that originated the roaming event (808). In some examples, the NMS may further determine a sticky client score based on the number of available roaming options available to the client device and/or the type of roaming option (810). In some examples, the NMS may determine the inter-band score further based on RSSI delta experienced by the wireless client device for the roaming event (e.g., a change in RSSI of the wireless signal received before and after the inter-band roaming event) (811). In some examples, the NMS may further determine a roaming stability score based on the failure of fast roaming (812). In some examples, the NMS may further determine a roaming metric based on one or more of the delay score, the sub-optimal roaming score, the sticky client score, and/or the inter-band score (814). The NMS may repeat the process continuously for the wireless network (800) and update the delay score, the sub-optimal roaming score, the sticky client score, the inter-band score, the roaming stability score, and the roaming metric on a continuous basis.

Fig. 9 is a flowchart of an example process (900) of processing circuitry of a network management system, such as NMS150 or NMS 300, determining sticky client scores for client devices in a wireless network in accordance with one or more techniques of the present disclosure. For purposes of this disclosure, a sticky client is defined as a client device having a current RSSI value that is less than a fixed roaming threshold of the client device and having one or more roaming options that are at least 7dBm better than the current RSSI value. The NMS determines the current RSSI of the client device (902). The current RSSI of the client device is the RSSI of the wireless signal received from the AP with which the client device is currently associated. The current RSSI is determined from the perspective of the client device.

The NMS determines whether the client device satisfies the definition of the sticky client. In this example, the NMS determines if the current RSSI is less than a fixed roaming threshold for the client device and if so, determines if the client device has not performed a roaming operation (904). If the client device does not meet the definition of a sticky client ("no" branch of 904), the NMS assigns a sticky client score indicating that the client device is a non-sticky client (906).

If the client device satisfies the definition of a sticky client ("yes" branch of 904), the NMS determines the number of roaming options available to the client device and the type of roaming options available to the client device (908). The NMS assigns a sticky client score to the client device based on the number of available roaming options and/or the type of available roaming options for the client device (910). In some examples, the NMS may further determine a roaming metric indicating roaming performance of the wireless network based on the sticky client scores of the plurality of client devices (912). The NMS may continually repeat the process (900) for each client associated with the wireless network and update the sticky client score on a continuous basis.

Fig. 10 is a flowchart of an example process (1020) by which a computing device, such as a network management system, such as NMS150 and/or 300, determines whether one or more roaming quality assessments represent abnormal operation of a wireless network in accordance with one or more techniques of the present disclosure. The NMS receives network data indicating wireless network performance (1022). The network data may be obtained or determined by a plurality of client devices in the wireless network, a plurality of APs associated with the wireless network, and/or any other device associated with the wireless network. The NMS determines one or more roaming quality assessments for each roaming event and/or client device based on the network data (1024). For example, the roaming quality assessment may include one or more of a delay score associated with each roaming event, a sub-optimal roaming score associated with each roaming event, a sticky client score associated with each roaming event, and an inter-band score associated with each roaming event. In some examples, the roaming quality assessment may also include detection of ping pong and/or excessive roaming events for one or more client devices, which in some examples are considered abnormal roaming behavior.

The NMS determines if one or more of the roaming quality assessments is indicative of an abnormal operation of the wireless network (1026). For example, the NMS may apply a trained machine learning model to classify the operation of the wireless network based on historical network data. The trained machine learning model may classify the operation of the wireless network as normal or expected operation, or one or more abnormal operation classifications, depending on, for example, which roaming quality assessment is indicative of abnormal operation and/or the relative severity of abnormal operation.

If the NMS determines that the one or more roaming quality assessments do not indicate an abnormal operation of the wireless network (e.g., the operation is normal or expected) (no branch of 1028), the NMS continues to monitor the network data for abnormal operation during operation of the wireless network (1022, 1024, 1026). In some examples, if the NMS determines that the one or more roaming quality assessments indicate abnormal operation of the wireless network ("yes" branch of 1028), the NMS may further determine one or more root causes of the abnormal operation (1030). In some examples, to determine the root cause of the abnormal operation (1030), the processor may refer to root cause association data stored in a database, such as database 312 shown in fig. 3A. The root cause association data 60 may include, for example, one or more database structures that associate classifications of abnormal operation with one or more potential root causes. In other examples, to determine the root cause of the abnormal operation (1030), the NMS may invoke a Virtual Network Assistant (VNA), such as VNA engine 350 shown in fig. 3A, to invoke more complex and/or computationally costly root cause analysis.

The NMS may further determine one or more remedial actions to address the root cause of the abnormal operation (1032). In some examples, to determine the remedial action (1032) to address the root cause of the abnormal operation, the NMS may refer to the root cause association data stored in a database, such as database 312 shown in fig. 3A. The root cause association data may include, for example, one or more database structures that associate classifications of abnormal operation with one or more potential root causes, and further associate each root cause with one or more remedial actions that may be suggested and/or automatically invoked to address the corresponding root cause. In other examples, to determine a remedial action to address the root cause of the abnormal operation (532), the processor may invoke a Virtual Network Assistant (VNA), such as VNA 350 shown in fig. 3A, to invoke root cause/remedial action analysis that is more complex and/or computationally costly.

The NMS may further generate a notification comprising an indication of one or more roaming quality assessments, an indication of any abnormal operation and/or abnormal operation classification, one or more root causes of abnormal operation, and/or one or more remedial actions to address the root causes of abnormal operation (1034). In some examples, the NMS may further automatically invoke one or more of the determined remedial actions to address the root cause of the determined wireless network abnormal operation (1034). The NMS may continually repeat the process (1020) for the wireless network and update the roaming quality assessment, monitor for abnormal network behavior, determine the root cause of the abnormal behavior and automatically perform remedial actions to address the abnormal behavior on a continuous basis.

Example techniques for performing root cause analysis and/or automatically invoking remedial actions by a network management system are described in the following patent applications: U.S. application Ser. No. 14/788,489 entitled "monitoring Wireless Access Point event (Monitoring Wireless Access Point Events)" filed on 6/30/2015; U.S. application Ser. No. 16/835,757 entitled "network System trouble-shooting Using machine learning model (Network System Fault Resolution Using a Machine Learning Model)" filed on 31/3/2020; U.S. application Ser. No. 16/279,243 entitled "System and method for virtual network Assistant Systems and Methods for a Virtual Network Assistant" filed on 2.19.2019; U.S. application Ser. No. 16/237,677, entitled "method and apparatus for facilitating failure detection and/or predictive failure detection (Methods and Apparatus for Facilitating Fault Detection and/or Predictive Fault Detection)" filed on 12/31/2018, and U.S. application Ser. No. 16/251,942, entitled "space-time modeling method (Method for Spatio-Temporal Modeling)" filed on 18/1/2019; U.S. application Ser. No. 16/296,902 entitled "method of transmitting AP error codes over BLE advertisements" (Method for Conveying AP Error Codes Over BLE Advertisements) filed on 3/8/2019, which is incorporated herein by reference in its entirety.

Fig. 11A-11F are screen shots of example user interfaces generated by one or more processors, such as one or more processors of NMS150/300, including user interface elements representing one or more roaming quality assessments in accordance with one or more techniques of the present disclosure. The one or more roaming quality assessments may include, for example, a roaming SLE metric, a roaming delay classifier, a roaming signal quality classifier, and/or a roaming stability classifier. The roaming quality assessment may also include one or more roaming delay sub-classifiers, such as a sub-optimal 802.11r roaming sub-classifier, a sub-optimal OKC roaming sub-classifier, and/or a slow roaming sub-classifier; one or more signal quality sub-classifiers, such as a sub-optimal roaming sub-classifier, a sticky client sub-classifier, and/or an inter-band sub-classifier; and/or one or more stability subcategories, such as a fast roaming failure subcategories. Some or all of these example roaming metrics, classifiers, and/or sub-classifiers may be determined by one or more processors of NMS150/300 executing roaming metrics module 370 of SLE metrics module 352 as shown in fig. 3B.

Fig. 11A illustrates an example user interface 1100 that includes user interface elements that represent a roaming SLE metrics architecture 1100. In some examples, NMS150/300 generates a data representation comprising selectable user interface elements for one or more of roaming SLE metrics 1110, one or more roaming metric classifiers 1112 (e.g., delay, signal quality, and stability), and one or more roaming degree quantum classifiers 1114 (e.g., failure of sub-optimal 802.11r roaming, sub-optimal OKC roaming, and slow roaming, sticky clients, inter-band, and fast roaming). The roaming delay classifier scores each roaming event by time and thus includes time domain based sub-classifiers such as sub-optimal 11r roaming, sub-optimal OKC roaming, and slow roaming scores. The roaming signal quality classifier classifies each roaming event according to signal strength (e.g., RSSI), and thus includes sub-classifiers based on RSSI and/or roaming type, such as sub-optimal roaming, sticky client, and inter-band scores. The stability classifier attempts to score each roaming event to detect certain undesirable roaming patterns, such as failure events for fast roaming.

Fig. 11B-11F illustrate an example user interface including the example user interface element 1100 of fig. 11A. For example, in fig. 11B, the NMS may generate data representing a user interface 1120, the user interface 1120 including one or more selectable user interface elements and/or one or more sub-windows including roaming quality assessment information. In this example, the user interface 1120 includes a selectable user interface element 1122 through which a user can select to view data associated with the roaming quality assessment of the present disclosure. In this example, the user has selected user interface element 1122 corresponding to the roaming SLE metric, which causes NMS150/300 to display selectable user interface elements corresponding to the roaming assessment classifier delay, stability and signal quality. The user interface 1120 further includes a sub-window 1126 having one or more user selectable tabs through which the user can view selected roaming assessment information. For example, selecting the "statistics" tab 1124 as shown in fig. 11B shows that the roaming success rate for the currently selected time period (selected "yesterday" in the upper right hand corner of user interface 1120) is 95%, meaning that 95% of the roaming events that occur within the selected time period meet the threshold for a "good" or "excellent" roaming SLE metric. In this example, the average roaming metric score is 1.2 (i.e., between the "excellent" score of 1 and the "good" score of 2).

As another example, as shown in the example of fig. 11C, selection of the "distribute" tab 1134 causes the NMS150/300 to display roaming quality assessment information for multiple device types. The user may further select any of the selectable user interface elements "device OS", "access point", "WLAN", or "radio band" to display roaming quality assessment data (e.g., percentage of service level failure) corresponding to each of these attributes.

As another example, as shown in the example of fig. 11D, selection of the "timeline" tab 1144 causes the NMS150/300 to show a "timeline" chart in the sub-window 1146. In addition, fig. 11D further illustrates that the user may hover over the selectable user interface element 1142, causing the NMS150/300 to display a pop-up box 1143 describing further details corresponding to the signal quality classifier.

As shown in the example of fig. 11E, selection of the selectable user interface element 1142 of the signal quality classifier causes NMS150/300 to generate a user interface 1150 that includes the associated sub-classifier sticky client, inter-band and sub-optimal roaming selectable user interface element 1152.

As shown in the example of fig. 11F, the user may hover over the selectable user interface element 1167 causing the NMS150/300 to display a pop-up box 1168 describing further details of the sub-optimal roaming sub-classifier corresponding to the signal quality classifier. In general, each selectable user interface element (e.g., user interface element 1122), one of the roaming quality classifiers (e.g., user interface element 1142), and/or one of the roaming quality sub-classifiers (e.g., user interface element 1152) corresponding to a roaming SLE metric includes a percentage value (in this example, 1161, 1165, and 1167, respectively). In some examples, NMS150/300 executes VNA/AI engine 350 and root cause analysis module 354 to determine how to attribute roaming event failures between the roaming assessment classifier and the sub-classifier. For example, NMS150/300 may perform one or more steps of the example process (1020) as shown in fig. 10 to determine the root cause of a roaming event failure and/or an attribute roaming event failure between a roaming assessment classifier and a sub-classifier.

In the example of fig. 11F, for roaming SLE metric 1122, user interface element 1161 indicates that a roaming event of 95% of the selected time period satisfies the roaming threshold set for that station. The user interface elements of roaming classifier delay, stability and signal quality include percentages of 1%, 4% and 95%, respectively. These values mean that, among the 5% of the roaming events that do not meet the roaming threshold, 1% is attributable to the delay problem, 4% is attributable to the stability problem, and 95% is attributable to the signal quality problem. Further, in this example, the 100% signal quality problem may be due to sub-optimal roaming, as indicated by reference numeral 1167.

As shown in the example of fig. 11G, the user interface 1170 includes selectable user interface elements 1172 corresponding to the interband sub-classifier. The user may hover over the user interface element 1172 causing the NMS150/300 to display a pop-up 1178 describing further details of the interband sub-classifier corresponding to the signal quality classifier. In the example of fig. 11G, 25% of the signal quality problem is attributable to the sticky client terminal classifier and 75% is attributable to the inter-band sub-classifier.

The sub-window 1176 displays data and statistics corresponding to the "affected item" label of the inter-band sub-classifier. In this example, the "access point" has been selected such that sub-window 1176 displays a list of access points that fail to meet the service level objective (e.g., general, worse, or worst inter-band scores). Details regarding each affected AP are also listed, including name, overall impact, failure rate, MAC address and IP address to provide guidance to IT personnel in diagnosing and resolving failures.

In the example of fig. 11H, the "timeline" tab has been selected such that the sub-window 1182 displays a timeline of attempted inter-band roaming. The user may hover the mouse over a particular time or select a particular time (in this example, selected 2 months 22 days Tuesday 5:40 A.M. to 5:50 A.M.) to view more specific information for the selected time or time period.

In some examples, NMS140/300 may detect ping-pong roaming events for one or more client devices associated with a wireless network in accordance with one or more techniques of this disclosure. For purposes of this disclosure, a "ping-pong roaming event" describes a sequence of two or more consecutive roaming events that meet one or more predetermined criteria, as described below.

Fig. 12A illustrates an example ping-pong roaming scenario for a wireless client device that may be detected by an NMS, such as NMS150 and/or 300, in accordance with one or more techniques of this disclosure. In some examples, ping-pong roaming events are detected when a client device performs two or more consecutive roaming events between a set of APs, wherein a time difference between each consecutive roaming event occurs within a predetermined period of time. The set of APs may include, for example, a set of two APs, a set of three APs, or a set of more than three APs. In other examples, a ping-pong roaming event is detected when the client device performs more than two consecutive roaming events between two or more frequency bands of a single AP within a predetermined period of time.

Fig. 12A illustrates an example ping-pong roaming scenario 1230 during which a client device (not shown) performs a sequence of consecutive roaming events among four APs (i.e., APX, AP1, AP2, and APY). In fig. 12A, a "ping-pong roaming event" consists of two roaming events 1232 and 1233 between a set of two APs (i.e., AP1 and AP 2).

Assume that a client device (not shown) is associated with a first AP (i.e., APX). The client device performs a first roaming event (as indicated by arrow 1231) during which the client device roams from APX to AP1. Thus, APX is the "previous" AP of roaming event 1231 and AP1 is the "target" AP of roaming event 1231. At a subsequent time, the client device performs a second continuous roaming event, indicated by arrow 1232, during which the client device roams from AP1 to AP2 (AP 1 being the previous AP and AP2 being the target AP for roaming event 1232). At a subsequent time, as indicated by arrow 1233, the client device performs a third continuous roaming event during which the client device roams from AP2 back to AP1 (AP 2 is the previous AP and AP1 is the target AP for roaming event 1233). The client device then performs a fourth continuous roaming event 1234 during which the client device roams from AP1 to APY (AP 1 being the previous AP and APY being the target AP for roaming event 1234).

Fig. 12B shows a timing diagram of the roaming event of fig. 12A. At time T1, the client device performs a first roaming event 1231 from APX to AP 1. At time T2, the client device performs a second continuous roaming event 1232 from AP1 to AP 2. At time T3, the client device performs a third continuous roaming event 1233 from AP2 back to AP 1. At time T4, the client device performs a fourth continuous roaming event 1234 from AP1 to APY.

The computing device determines a time difference between execution of each successive roaming event. For example, as shown in FIG. 12B, the computing device determines the time difference between execution of roaming events 1231 and 1232 (T2-T1), the time between execution of roaming events 1232 and 1233 (T3-T2), and the time between execution of roaming events 1233 and 1234 (T4-T3). The time differences (T2-T1), (T3-T2), and (T4-T3) may thus be considered the effective "residence time" for the client device associated with AP1 (after completion of the first roaming event 1231), AP2, and AP1 (after completion of the roaming event 1234), respectively.

To detect a ping-pong roaming event such as shown in the examples of fig. 12A-12B, a computing device (such as NMS 150/300) determines whether a time difference between two consecutive roaming events performed by a client device satisfies a predetermined period of time. In particular, the computing device determines whether the time difference between two consecutive roaming events (in this example, roaming events 1232 and 1233) performed by the client device satisfies a predetermined period of time. In addition, the computing device determines whether a previous AP (AP 1) of a first one (1232) of the two consecutive roaming events matches a target AP (AP 1) of a second one (1233) of the two consecutive roaming events. If both criteria are met, the computing device determines that the combination of roaming events 1232 and 1233 qualify as a ping-pong roaming event.

The predetermined period of time may be configured based on one or more factors including, but not limited to, the type of ping-pong roaming event (e.g., the total number of individual roaming events and/or APs in the ping-pong event), the type of roaming (e.g., slow roaming, fast roaming (802.11 r), opportunistic Key Cache (OKC) roaming, inter-band roaming, etc.), the site at which the AP is deployed to provide the wireless network, the location of the client device, the roaming behavior of the client device, the type of client device, or based on any other factors, and the disclosure is not limited in this respect. In some examples, the predetermined period of time may be in the range of 0 to 60 seconds. However, other ranges of predetermined time periods may be used, and the disclosure is not limited in this respect.

Fig. 12C-12D illustrate another example ping-pong roaming scenario 1240 during which a client device (not shown) performs a sequence of consecutive roaming events between five APs (i.e., APX, AP1, AP2, AP3, and APY). For example, assume that a client device is associated with APX. At a first time T1, the client device performs a first roaming event (as indicated by arrow 1241) during which the client device roams from APX to AP1. At a subsequent time T2, the client device performs a second continuous roaming event (as indicated by arrow 1242), during which the client device roams from AP1 to AP2. At a subsequent time T3, the client device performs a third continuous roaming event (as indicated by arrow 1243), during which the client device roams from AP2 to AP3. At a subsequent time T4, the client device performs a fourth continuous roaming event (as indicated by arrow 1244), during which the client device roams from AP3 back to AP1. At a subsequent time T5, the client device performs a fifth continuous roaming event 1245 during which the client device roams from AP1 to APY.

The computing device determines a time difference between execution of each successive roaming event. For example, as shown in FIG. 12D, the computing device determines the time difference (T2-T1) between execution of roaming events 1241 and 1242, the time (T3-T2) between execution of roaming events 1242 and 1243, the time (T4-T3) between execution of roaming events 1243 and 1244, and the time (T-T4) between execution of roaming events 1244 and 1245. The time differences (T2-T1), (T3-T2), (T4-T3), and (T5-T4) may thus be considered as the effective "dwell times" that the client device is associated with AP1 (after completion of roaming event 1241), AP2, AP3, and AP1 (after completion of roaming event 1245), respectively.

To detect ping-pong roaming events such as shown in the examples of fig. 12C-12D, a computing device (such as NMS 150/300) determines whether a time difference between each of at least three consecutive roaming events performed by a client device satisfies a predetermined period of time. In particular, the computing device determines whether a time difference between each of three consecutive roaming events (in this example, (T3-T2) and (T4-T3)) performed by the client device satisfies a predetermined period of time. In addition, the computing device further determines whether a previous AP of a first one of the three consecutive roaming events matches a target AP of a third one of the three consecutive roaming events. In other words, the client device completes a loop between three or more APs. In the example of fig. 12C-12D, the computing device determines whether the client device performed three consecutive roaming events (e.g., 1242, 1243, and 1244), wherein the current AP (AP 1) of the first of the three consecutive roaming events (1242) matches the target AP (also AP 1) of the third of the three consecutive roaming events (1244). Although fig. 12C and 12D illustrate ping-pong roaming events between three APs, three or more APs may be used to define other ping-pong roaming events, and the disclosure is not limited in this respect.

In addition to detecting whether there is a previous/target AP match for the first and third roaming events of the sequence of roaming events (or in general, the first and nth roaming events of the n consecutive roaming events), the computing device may further apply additional criteria as described above with respect to fig. 12C to the sequence of three consecutive roaming events to detect ping-pong roaming events. For example, the computing device may determine whether each of the consecutive roaming events is performed within a predetermined period of time of an immediately preceding roaming event in the sequence of roaming events. Thus, in this example, the time periods (T3-T2), (T4-T3), and (T5-T4) must each meet a predetermined time period in order for the combination of roaming events 1242, 1243, and 1244 to qualify as ping-pong roaming events. As discussed above, the predetermined period of time may be configured based on one or more factors. In some examples, the predetermined period of time may be in the range of 0 to 60 seconds. However, other ranges of predetermined time periods may be used, and the disclosure is not limited in this respect.

Different criteria may be defined for different types of roaming events. For example, the number m of APs may be defined, where each integer value of m corresponds to a different type of ping-pong roaming event. Different ping-pong roaming criteria may be defined for different m integer values. For example, different predetermined time periods may be applied to different values of m. In addition, the number of roaming events that qualify as a requirement for a ping roaming event is also configurable. For example, in fig. 12A, a ping-pong roaming event may be detected when the time difference between two consecutive roaming events 1232 and 1233 satisfies a predetermined period. In other examples, three or more consecutive roaming events between AP1 and AP2 may be required (i.e., the client device may continue to roam back and forth between AP1 and AP2 multiple times), each of which meets a predetermined period of time relative to the immediately preceding roaming event in order to qualify as a ping-pong roaming event.

The examples of fig. 12A-12D illustrate ping-pong roaming events between two or more different APs. In other examples, the computing device may also detect ping-pong roaming events that occur between two different frequency bands of a single AP. For example, rather than detecting a ping-pong roaming event between two different APs (as shown in fig. 12A-12B), a similar timing standard as described above may be used to detect a ping-pong roaming event between two different frequency bands of a single AP (e.g., roaming back and forth between the 2.4GHz and 5GHz frequency bands of a single AP). As another example, in addition to detecting ping-pong roaming events between three different APs (as shown in fig. 12C-12D), ping-pong roaming events between two different frequency bands of a first AP and one frequency band of a second AP, and/or ping-pong roaming events between three different frequency bands of a single AP (e.g., roaming between 2.4GHz, 5GHz, and 6GHz frequency bands of a single AP) may be detected. Thus, although fig. 12A-12D illustrate examples of ping-pong roaming events between different APs, any of the ping-pong roaming events described herein may also include at least one separate roaming event between two different frequency bands of a single AP, and the disclosure is not limited in this respect.

In some examples, NMS150/300 detects one or more types of excessive roaming events for one or more client devices associated with a wireless network in accordance with one or more techniques of this disclosure. For purposes of this disclosure, the term "excessive roaming event" describes a combination of two or more roaming events that meet one or more predetermined criteria, as described below. For example, the computing device may detect an excessive roaming event when the client device executes a sequence of three or more consecutive roaming events, each event being within a predetermined period of time. For excessive roaming events, consecutive roaming events are not performed within the same set of APs as ping-pong roaming events.

Fig. 12E-12F illustrate an example excessive roaming scenario 1250 during which a client device (not shown) performs a sequence of consecutive roaming events between a set of four APs (i.e., AP1, AP2, AP3, and AP 4), each AP being within a predetermined period of time. In fig. 12E, the "excessive roaming event" includes three roaming events 1251, 1252, and 1253.

Assume that a client device (not shown) is associated with a first AP (i.e., AP 1). At a first time T1, the client device performs a first roaming event (as indicated by arrow 1251) during which the client device roams from AP1 to AP2. At a subsequent time T2, the client device performs a second continuous roaming event (as indicated by arrow 1252), during which the client device roams from AP2 to AP3. At a subsequent time T3, the client device performs a third continuous roaming event (as indicated by arrow 1253), during which the client device roams from AP3 to AP4.

The computing device determines a time difference between execution of each successive roaming event. For example, as shown in FIG. 12F, the computing device determines the time difference (T2-T1) between the execution of roaming events 1251 and 1252 and the time (T3-T2) between the execution of roaming events 1252 and 1253. The time differences (T2-T1) and (T3-T2) may thus be considered as the effective "dwell times" that the client device associates with AP2 and AP3, respectively.

To detect an excessive roaming event, a computing device (such as NMS 150/300) determines whether a time difference between two consecutive roaming events in a sequence of at least n roaming events performed by a client device satisfies a predetermined period of time. In particular, to detect an excessive roaming event with respect to the examples of fig. 12E-12F, the computing device determines whether a time difference between two consecutive roaming events in a sequence of at least three roaming events (in this example, (T2-T1) and (T3-T2)) performed by the client device satisfies a predetermined period of time.

The predetermined period of time for detecting an excessive roaming event may be configured based on one or more factors as discussed above with respect to the ping pong roaming event. In some examples, the predetermined period of time may be in the range of 0 to 60 seconds. However, other ranges of predetermined time periods for detecting excessive roaming events may also be used, and the disclosure is not limited in this respect.

Fig. 13A is a flowchart of an example process (1300) of a computing device, such as a network management system (e.g., NMS 150/300), detecting a ping-pong roaming event in accordance with one or more techniques of the present disclosure. The computing device determines if a time difference between two consecutive roaming events performed by the client device satisfies a predetermined time period (1302). For example, the computing device determines whether a time difference between the client device executing a first roaming event and the client device executing an immediately following second roaming event satisfies a predetermined time period (1302). If the predetermined period of time is not met ("NO" branch of 1304), the computing device determines that the sequence of two consecutive roaming events is not eligible as a ping-pong roaming event (1306).

In some examples, the computing device detects a ping-pong roaming event upon determining that a time difference between two consecutive roaming events performed by the client device satisfies a predetermined period of time (1302). In such an example, the combination of roaming events 1232 and 1233 in fig. 12A would qualify as a ping-pong roaming event. In other examples, a ping-pong roaming event is defined as a sequence of n consecutive roaming events between the same two APs, where each of the roaming events meets a predetermined period of time relative to an immediately preceding roaming event. For example, referring again to fig. 12A, rather than detecting a ping-pong roaming event upon determining that the time difference between two consecutive roaming events (1232 and 1233) satisfies a predetermined time period, the computing device may require the client device to execute a sequence of n consecutive roaming events between AP1 and AP2, each of the consecutive roaming events satisfying a predetermined time period relative to the immediately preceding roaming event such that the sequence of n roaming events qualify as a ping-pong roaming event. The number n is an integer and may be configured based on various factors including, but not limited to, client roaming behavior, type of client device (e.g., mobile computing device such as smart phone and robotic automation for e-commerce applications, etc.), type of roaming (e.g., slow roaming, fast roaming, OKC roaming, inter-band roaming, etc.), site, or based on any other basis, and the disclosure is not limited in this respect.

If the predetermined time period is met (and/or a sequence of n consecutive roaming events has been detected that meets the predetermined time period) (yes branch of 1304), the computing device determines whether the set of m APs corresponding to two (or more) consecutive roaming events meet one or more ping pong roaming event criteria (1308). For example, if the sequence of n consecutive roaming events includes two roaming events, such as roaming events 1232 and 1233 of fig. 12A, the set of m APs includes two APs, AP1 and AP2. In such examples, the roaming criteria may include determining whether a previous AP of a first one of the two consecutive roaming events (e.g., AP1 of roaming event 1232) matches a target AP of a second, immediately following one of the two consecutive roaming events (e.g., AP1 of roaming event 1233). In some examples, the roaming criteria may also include determining whether the target AP of the first roaming event (e.g., AP2 of roaming event 1232) matches the previous AP of the immediately following second roaming event (e.g., AP2 of roaming event 1233). As another example, if the sequence of consecutive roaming events includes three roaming events, such as roaming events 1242, 1243 and 1244 of fig. 12C, the set of m APs includes three APs, namely AP1, AP2 and AP3. The computing device determines whether a previous AP of a first one of the three consecutive roaming events matches a target AP of a third one of the three consecutive roaming events. Ping-pong roaming events may also be defined for three or more APs, including different roaming modes between three or more APs (e.g., a loop including roaming events 1242, 1243, and 1244 is shown in fig. 12C, however, three APs, i.e., three loops of continuous roaming between AP1, AP2, and AP3, are not necessarily required), and the disclosure is not limited in this respect.

If the set of APs meets the ping-pong roaming event criteria ("yes" branch of 1310), the computing device determines that the two or more consecutive roaming events qualify as ping-pong roaming events (1314). If the set of APs does not meet the ping-pong roaming event criteria ("no" branch of 1310), the computing device determines that the two or more consecutive roaming events are not eligible as ping-pong roaming events (1312).

In some examples, the NMS may apply a trained machine learning model to classify a sequence of consecutive roaming events as ping-pong roaming events. One or more characteristics of the sequence of roaming events (e.g., the number of individual roaming events in the sequence, the amount of time associated with each roaming event, the type of roaming event (e.g., fast roaming, slow roaming, OKC roaming, inter-band roaming, the number of APs, the type of client device, etc.)) may be applied as an input to a machine learning model to determine whether to classify the sequence of consecutive roaming events as ping-pong roaming events.

The NMS may further determine one or more root causes of the one or more ping pong roaming events (1316). In some examples, to determine the root cause of the ping pong roaming event (1316), the processor may reference root cause association data stored in a database, such as database 312 shown in fig. 3A. The root cause association data may include, for example, one or more database structures that associate classifications of ping-pong roaming events with one or more potential root causes. In other examples, to determine the root cause of a ping-pong roaming event, the NMS may invoke a Virtual Network Assistant (VNA), such as VNA engine 350 shown in fig. 3A, to invoke more complex and/or computationally costly root cause analysis.

The NMS may further determine one or more remedial actions to address the root cause of the ping-pong roaming event (1318). In some examples, to determine the remedial action (1318), the NMS may refer to root cause association data stored in a database, such as database 312 as shown in fig. 3A. The root cause association data may include, for example, one or more database structures that associate ping-pong roaming events with one or more potential root causes, and further associate each root cause with one or more remedial actions that may be suggested and/or automatically invoked to address the respective root cause. In other examples, to determine a remedial action to address the root cause of the ping pong roaming event (1318), the processor may invoke a Virtual Network Assistant (VNA), such as VNA350 as shown in fig. 3A, to invoke root cause/remedial action analysis that is more complex and/or computationally costly.

The NMS may further generate a notification including an indication of any detected ping-pong roaming event, an indication of the client device and/or AP involved in any detected ping-pong roaming event, one or more root causes of the detected ping-pong roaming event, and/or one or more suggested remedial actions that may be used to address the root cause (1318). In some examples, the NMS may further automatically invoke one or more of the determined remedial actions to address the determined root cause of the wireless network abnormal operation (1324). The NMS may continually repeat the process (1300) for the wireless network to detect ping-pong roaming events, determine their root cause, generate notifications, and/or automatically perform remedial actions on a continuous, periodic, or scheduled basis.

Fig. 13B is a flowchart of an example process (1350) of detecting an excessive roaming event by a computing device, such as a network management system (e.g., NMS 150/300), in accordance with one or more techniques of this disclosure. The computing device determines if a time difference between two consecutive roaming events performed by the client device satisfies a predetermined time period (1352). For example, the computing device determines whether a time difference between the client device performing a first roaming event and the client device performing an immediately subsequent second roaming event satisfies a predetermined time period (1352). If the predetermined period of time is not met (NO branch of 1354), the computing device determines that the sequence of two consecutive roaming events is not eligible as an excessive roaming event (1356).

In some examples, an excessive roaming event is defined as a sequence of n consecutive roaming events, where each of the roaming events satisfies a predetermined period of time relative to an immediately preceding roaming event in the sequence of roaming events. For example, fig. 12E shows a sequence of n=3 roaming events, including roaming events 1251, 1252, and 1253. The number n is an integer and may be configured based on various factors including, but not limited to, client roaming behavior, type of client device (e.g., mobile computing device such as smart phone and robotic automation for e-commerce applications, etc.), type of roaming (e.g., slow roaming, fast roaming, OKC roaming, inter-band roaming, etc.), site, or based on any other basis, and the disclosure is not limited in this respect.

If the predetermined time period is met (yes branch of 1354), the computing device increments the number of detected consecutive roaming events that meet the predetermined time period (1358). The computing device determines whether the number of consecutive roaming events that meet the predetermined period of time meets one or more excessive roaming criteria (1360). For example, the excessive roaming criterion may define a number n of consecutive roaming events that satisfy a predetermined period of time that is defined as being required for an excessive roaming event.

If the excessive roaming criteria is met ("yes" branch of 1360), the computing device determines that n consecutive roaming events qualify as excessive roaming events (1364). If the excessive roaming criteria is not met (the "NO" branch of 1360), the computing device determines that the sequence of consecutive roaming events detected so far is not eligible as an excessive roaming event (1362). In such examples, the computing device repeats steps (1352, 1354, 1358, 1360) until two consecutive roaming events are detected that do not satisfy the predetermined period of time (the "no" branch of 1354), or n consecutive roaming events are detected that satisfy the predetermined period of time (the "yes" branch of 1360).

In some examples, the NMS may apply a trained machine learning model to classify a sequence of continuous roaming events as excessive roaming events. One or more characteristics of the sequence of roaming events (e.g., the number of individual roaming events in the sequence, the amount of time associated with each roaming event, the type of roaming event (e.g., fast roaming, slow roaming, OKC roaming, inter-band roaming, etc.), the set of APs, the number of APs, the type of client device, etc.) may be used as inputs to a machine learning model to determine whether to classify the sequence of consecutive roaming events as excessive roaming events.

The NMS may further determine one or more root causes of the one or more excessive roaming events (1366). In some examples, to determine the root cause of the excessive roaming event (1366), the processor may reference root cause association data stored in a database, such as database 312 as shown in fig. 3A. The root cause association data may include, for example, one or more database structures that associate classifications of excessive roaming events with one or more potential root causes. In other examples, to determine the root cause of an excessive roaming event, the NMS may invoke a Virtual Network Assistant (VNA), such as VNA engine 350 shown in fig. 3A, to invoke more complex and/or computationally costly root cause analysis.

The NMS may further determine one or more remedial actions to address the root cause of the excessive roaming event (1368). In some examples, to determine the remedial action (1368), the NMS may refer to root cause association data stored in a database (such as database 312 shown in fig. 3A). The root cause association data may include, for example, one or more database structures that associate an excessive roaming event with one or more potential root causes, and further associate each root cause with one or more remedial actions that may be suggested and/or automatically invoked to address the corresponding root cause. In other examples, to determine a remedial action to address the root cause of the excessive roaming event (1368), the processor may invoke a Virtual Network Assistant (VNA), such as VNA350 shown in fig. 3A, to invoke more complex and/or computationally costly root cause/remedial action analysis.

The NMS may further generate a notification comprising an indication of any detected excessive roaming event, an indication of the client device and/or AP involved in any detected excessive roaming event, one or more root causes of the detected excessive roaming event, and/or one or more suggested remedial actions that may be used to address the root cause (1368). In some examples, the NMS may further automatically invoke one or more of the determined remedial actions to address the root cause of the determined wireless network abnormal operation (1368). The NMS may continually repeat the process 1350 for the wireless network to detect excessive roaming events, determine its root cause, generate notifications, and/or automatically perform remedial actions on a continual, periodic, or scheduled basis.

Fig. 14 illustrates an example user interface 1420 that may be generated by NMS150/300 that includes data indicating one or more ping pong roaming events or one or more excessive roaming events detected for a client device in accordance with one or more techniques of this disclosure. In this example, the user interface 1420 includes a user selectable interface element 1424 through which a user can select to view data associated with a particular client device. The user interface element 1424 includes a drop down menu through which a user can select one of a plurality of client devices associated with a wired or wireless network. In this example, the user has selected the client device "android-3 af2 …". The user also selects the time range "today" in user interface element 1426 and "insight" tab 1422, which causes NMS150/300 to display a sub-window 1452 comprising a list of events for the selected client device corresponding to the selected time range. Each row of the event list includes one or more details corresponding to the detected event, in this example, including a MAC address of the selected client device, a device type of the selected client device, a date/time stamp of the detected event, an event type (in this example, a ping-pong event or an over-roaming event), and a list of APs involved in the corresponding ping-pong event or over-roaming event. In this example, the event list 1452 includes a list of two ping-pong events and one excessive roaming event for the selected client device.

Although a particular user interface is shown showing ping pong and/or excessive roaming events, other types of user interfaces or devices showing data representing or associated with one or more ping pong and/or excessive roaming events may also be generated, and the disclosure is not limited in this respect.

The disclosed technology provides one or more technical advantages and practical applications. These techniques enable NMS150/300 to automatically monitor and quantify roaming quality that accounts for other factors in addition to roaming delay (e.g., the amount of time required for a client device to roam from a first AP to a second AP). For example, although roaming delay is a time-based metric, the techniques of this disclosure enable NMS150/300 to consider RSSI measurements of wireless signal strength before and after a roaming event when evaluating the roaming performance of a wireless network and/or configuring the operation of one or more devices in a wireless network (such as one or more APs in a wireless network) to address poor roaming performance. In other examples, these techniques enable NMS150/300 to consider the number of roaming options available during each roaming event and/or the types of roaming options available during each roaming event in evaluating wireless network performance and/or configuring operation of one or more devices in the wireless network to address poor roaming performance. In other examples, these techniques enable NMS150/300 to identify undesirable roaming behavior, such as failure to fast roam when evaluating wireless network performance and/or configuring operation of one or more devices in a wireless network to address poor roaming performance. In other examples, NMS150/300 may further detect ping-pong and/or excessive roaming events for one or more client devices. In this way, this technique enables NMS150/300 to actively monitor and analyze roaming events based on one or more factors in addition to or instead of time-based metrics (such as roaming delays). Based on one or more of the roaming quality assessment and/or the roaming stability assessment, the NMS150/300 may further determine a roaming metric indicative of overall roaming performance of the wireless network, identify anomalies in roaming performance of the wireless network, identify root causes of one or more anomalies in roaming performance of the wireless network, and/or automatically invoke one or more remedial actions to address the identified anomalies. The remedial action may include, for example, automatically reconfiguring operation of at least one of the plurality of AP devices 142 in the wireless network. In this way, wireless network performance from the perspective of the client device may be improved, and thus the user experience of the wireless network may be improved.

The techniques described herein may be implemented using software, hardware, and/or a combination of software and hardware. Various examples relate to devices such as mobile nodes, mobile wireless terminals, base stations (e.g., access points), communication systems. Various examples also relate to methods, such as methods of controlling and/or operating a communication device (e.g., a wireless terminal (UE), a base station, a control node, an access point, and/or a communication system). Various examples also relate to non-transitory machines (e.g., computers), readable media (e.g., ROM, RAM, CD, hard disks, etc.), including machine-readable instructions for controlling one or more steps of a machine-implemented method.

It should be understood that the specific order or hierarchy of steps in the processes disclosed are examples of example approaches. Based on design preferences, it is understood that the specific order or hierarchy of steps in the processes may be rearranged while remaining within the scope of the present disclosure. The accompanying method claims present elements of the various steps in a sample order, and are not meant to be limited to the specific order or hierarchy presented.

In various examples, the devices and nodes described herein are implemented using one or more modules to perform steps corresponding to one or more methods, such as signal generation, transmission, processing, and/or reception steps. Thus, in some examples, the various features are implemented using modules. Such modules may be implemented using software, hardware, or a combination of software and hardware. In some examples, each module is implemented as a separate circuit, where a device or system includes separate circuits for implementing the functions corresponding to each described module. Many of the above-described methods or method steps may be implemented using machine-executable instructions, such as software, included in a machine-readable medium, such as a memory device (e.g., RAM, floppy disk, etc.) to control a machine, such as a general purpose computer with or without additional hardware, to implement all or portions of the above-described methods, such as in one or more nodes. Thus, various examples relate to, among other things, machine-readable media, such as non-transitory computer-readable media, including machine-executable instructions for causing a machine (e.g., processor and associated hardware) to perform one or more steps of the methods described above. Some examples relate to a device including a processor configured to implement one, more, or all steps of one or more methods of one example aspect.

In some examples, one or more devices (e.g., communication devices such as wireless terminals (UEs)) and/or one or more processors (e.g., CPUs) of an access node are configured to perform steps of a method described as being performed by the device. The configuration of the processor may be achieved by controlling the processor configuration using one or more modules (e.g. software modules) and/or by including hardware (e.g. hardware modules) in the processor to perform this step and/or controlling the processor configuration. Thus, some, but not all, examples relate to a communication device (e.g., user equipment) having a processor that includes modules corresponding to each of the various described steps of the method performed by the device including the processor. In some, but not all examples, a communication device includes modules corresponding to each of the various described method steps performed by the device including a processor. These modules may be implemented purely in hardware, for example as a circuit, or may be implemented using software and/or hardware or a combination of software and hardware.

Some examples relate to computer program products that include a computer-readable medium that includes code for causing a computer or computers to perform various functions, steps, actions, and/or operations (e.g., one or more of the steps described above). In some examples, the computer program product may, and sometimes does, include different code for each step to be performed. Thus, a computer program product may, and sometimes does, comprise code for each individual step of a method, such as a method of operating a communication device (e.g. a wireless terminal or node). The code may be in the form of machine (e.g., a computer), executable instructions stored on a computer readable medium such as RAM (random access memory), ROM (read only memory), or other type of storage device. In addition to involving computer program products, some examples involve processors configured to implement one or more of the various functions, steps, actions and/or operations of one or more methods described above. Thus, some examples relate to a processor, such as a CPU, a Graphics Processing Unit (GPU), a Digital Signal Processing (DSP) unit, or the like, configured to implement some or all of the steps of the methods described herein. The processor may be used, for example, in a communication device or other device described in the present application.

Many additional variations of the methods and apparatus of the various examples described above will be apparent to those skilled in the art in view of the above description. Such variations are considered to be within the scope of the disclosure. These methods and apparatus may, and in various examples are, used with BLE, LTE, CDMA, orthogonal Frequency Division Multiplexing (OFDM), and/or various other types of communications techniques which may be used to provide wireless communications links between access nodes and mobile nodes. In some examples, the access node is implemented as a base station that establishes a communication link with a user equipment device (e.g., a mobile node) using OFDM and/or CDMA. In various examples, the mobile node is implemented as a notebook computer, personal Data Assistant (PDA), or other portable device that includes receiver/transmitter circuitry and logic and/or routines for implementing the method.

In the specific embodiments, numerous specific details are set forth in order to provide a thorough understanding of some examples. However, it will be understood by those of ordinary skill in the art that some examples may be practiced without these specific details. In other instances, well-known methods, procedures, components, units and/or circuits have not been described in detail so as not to obscure the discussion.

Some examples may be used in conjunction with various devices and systems (e.g., user Equipment (UE), mobile Device (MD), wireless Station (STA), wireless Terminal (WT), personal Computer (PC), desktop computer, mobile computer, laptop computer, notebook computer, tablet computer, server computer, handheld device, personal Digital Assistant (PDA) device, handheld PDA device, in-vehicle device, off-vehicle device, hybrid device, in-vehicle device, off-vehicle device, mobile or portable device, consumer device, non-mobile or non-portable device, wireless communication station, wireless communication device, wireless Access Point (AP), wired or wireless router, wired or wireless modem, video device, audio-video (a/V) device, wired or wireless network, wireless area network, wireless Video Area Network (WVAN), local Area Network (LAN), wireless LAN (WLAN), personal Area Network (PAN), wireless PAN (etc.).

Some examples may be used in conjunction with and/or in conjunction with devices and/or networks operating in accordance with existing Wireless Gigabit Alliance (WGA) specifications (wireless gigabit alliance corporation, wiGig MAC and PHY specifications version 1.1, 2011 month 4, end specifications), devices and/or networks operating in accordance with existing IEEE 802.11 standards (IEEE 802.11-2012, IEEE information technology standards—inter-system telecommunication and information exchange local area networks and metropolitan area networks—specifically requiring part 11: wireless LAN Medium Access Control (MAC) and physical layer (PHY) specifications, 3 months of 2012, 29 days; IEEE 802.11ac-2013 ("IEEE p 802.11ac-2013, IEEE information technology standards-inter-system telecommunication and information exchange-local and metropolitan area network-specific requirements-11 part: wireless LAN Medium Access Control (MAC) and physical layer (PHY) specifications-4: enhancing the throughput of frequency bands below 6 GHz", 2013 month 12 months; IEEE 802.11ad ("IEEE 802.11 ad" and "radio 802.11ac-2013, 2012, radio 802 ac-2013, IEEE information exchange-2013, IEEE information technology standards-system telecommunication and/or" radio access sector-4 "; radio LAN medium access control (IEEE 802.11-2012) and/or" radio access medium access of the very high frequency band "of the radio access network" ("IEEE 802.11/2012), draft of information technology standard of 6 months 2014-specific requirements of telecommunication and information exchange local and metropolitan area networks between systems; part 11: wireless LAN Medium Access Control (MAC) and physical layer (PHY) specifications); IEEE 802.11-ay (P802.11 ay information technology standard-telecommunication and information exchange between system local area network and metropolitan area network-specific requirements part 11 wireless LAN Medium Access Control (MAC) and physical layer (PHY) specifications-revisions to enhance throughput operating in unlicensed bands above 45GHz, IEEE 802.11-2016 and/or future versions and/or derivative versions thereof, devices and/or network (WFA) point-to-point (P2P) specifications operating in accordance with the existing wireless fidelity (Wi-Fi) alliance (Wi-FiP 2P specifications, release 1.5, month 2014) and/or future versions and/or derivative versions thereof, devices and/or networks and/or protocols operating in accordance with existing cellular specifications, such as third generation partnership project (3 GPP), 3GPP Long Term Evolution (LTE) and/or future versions and/or derivative versions thereof, units and/or devices that are part of the above network, or use of one or more of any of the above, etc.

Some examples may be associated with unidirectional and/or bidirectional radio communication systems, cellular radiotelephone communication systems, mobile telephones, cellular telephones, wireless telephones, personal Communication Systems (PCS) devices, PDA devices which incorporate wireless communication devices, mobile or portable Global Positioning System (GPS) devices, devices which incorporate GPS receivers or transceivers or chips, devices which incorporate RFID elements or chips, multiple Input Multiple Output (MIMO) transceivers or devices, single Input Multiple Output (SIMO) transceivers or devices, multiple Input Single Output (MISO) transceivers or devices, devices with one or more internal and/or external antennas, digital Video Broadcasting (DVB) devices or systems, multi-standard radio devices or systems, wired or wireless handheld devices such as smartphones, wireless Application Protocol (WAP) devices, and the like.

Some examples may be associated with one or more types of wireless communication signals and/or systems (e.g., radio Frequency (RF), infrared (IR), frequency Division Multiplexing (FDM), orthogonal FDM (OFDM), orthogonal Frequency Division Multiple Access (OFDMA), FDM timeDivision Multiplexing (TDM), time Division Multiple Access (TDMA), multi-user MIMO (MU-MIMO), space Division Multiple Access (SDMA), extended TDMA (E-TDMA), general Packet Radio Service (GPRS), extended GPRS, code Division Multiple Access (CDMA), wideband CDMA (WCDMA), CDMA 2000, single carrier CDMA, multi-carrier modulation (MDM), discrete Multitone (DMT), bluetooth, global Positioning System (GPS), wi-Fi, wi-Max, zigBee ^TM Ultra Wideband (UWB), global system for mobile communications (GSM), 2G, 2.5G, 3G, 3.5G, 4G, fifth generation (5G) or sixth generation (6G) mobile networks, 3GPP, long Term Evolution (LTE), LTE advanced, enhanced data rates for GSM evolution (EDGE), etc.). Other examples may be used in various other devices, systems, and/or networks.

Some illustrative examples may be used in connection with a WLAN (wireless local area network), such as a Wi-Fi network. Other examples may be used in conjunction with any other suitable wireless communication network, such as a wireless area network, "piconet", WPAN, WVAN, etc.

Some examples may be used in connection with wireless communication networks that communicate over frequency bands of 2.4GHz, 5GHz, and/or 60 GHz. However, other examples may be implemented with any other suitable wireless communication frequency band, such as an Extremely High Frequency (EHF) band (millimeter wave) band, e.g., a band within a band between 20GhH and 300GHz, a WLAN band, a WPAN band, a band according to the WGA specification, and the like.

While only a few simple examples of various device configurations are provided above, it should be understood that many variations and permutations are possible. Furthermore, the technique is not limited to any particular channel, but is generally applicable to any frequency range/channel. Further, as discussed, the techniques discussed may be useful in unlicensed spectrum.

Although examples are not limited in this respect, discussions utilizing terms such as, for example, "processing," "computing," "calculating," "determining," "establishing", "analyzing", "checking", or the like, may refer to the operation(s) and/or process (es) of a computer, computing platform, computing system, communications system, or subsystem, or other electronic computing device, that manipulate and/or transform data represented as physical (e.g., electronic) quantities within the computer's registers and/or memories into other data similarly represented as physical quantities within the computer's registers and/or memories or other information storage medium that may store instructions to perform operations and/or processes.

Although examples are not limited thereto, the terms "multiple (multiple)" and "multiple (a multiple)" as used herein may include, for example, "multiple" or "two or more". The term "plurality" or "a plurality" may be used throughout the specification to describe two or more components, devices, elements, units, parameters, circuits, etc. For example, "multiple stations" may include two or more stations.

It may be advantageous to set forth definitions of certain words and phrases used throughout this document: the terms "include" and "comprise," as well as derivatives thereof, mean inclusion without limitation; the term "or" is inclusive, meaning and/or; the phrases "associated with" and derivatives thereof may mean inclusion, interconnection, interconnected, received, contained within, connected to or with a @ connection, coupling to or with a @ coupling, communicable, mated, interleaved, juxtaposed, proximate, tethered or otherwise, having a characteristic, or the like; the term "controller" refers to any device, system, or portion thereof that controls at least one operation, such device may be implemented in hardware, circuitry, firmware, or software, or some combination of the at least two. It should be noted that the functionality associated with any particular controller may be centralized or distributed, whether locally or remotely. Definitions for certain words and phrases are provided throughout this document, and those of ordinary skill in the art should understand that in many, if not most instances, such definitions apply to prior, as well as future uses of such defined words and phrases.

These examples have been described with respect to communication systems and protocols, techniques, means, and methods for performing communications, such as in a wireless network, or in general in any communication network operating with any communication protocol. Examples of such are home or access networks, wireless home networks, wireless corporate networks, etc. However, it should be understood that the systems, methods, and techniques disclosed herein will be equally applicable to other types of communication environments, networks, and/or protocols in general.

For purposes of explanation, numerous details are set forth in order to provide a thorough understanding of the present technology. However, it is understood that the present disclosure may be practiced in a variety of ways beyond the specific details set forth herein. Furthermore, while the examples shown herein illustrate various components of the system in juxtaposition, it should be understood that the various components of the system may be located within remote portions of a distributed network (such as a communication network, nodes, at domain hosts and/or the internet), or within a dedicated secure, unsecure and/or encryption system, and/or within a network operation or management device located within or outside of the network. As an example, a domain master may also be used to refer to any device, system, or module that manages and/or configures any one or more aspects of or communicates with the network or communication environment and/or transceivers and/or stations and/or access points described herein.

Thus, it should be understood that components of the system may be combined into one or more devices or separated between devices, such as transceivers, access points, stations, domain hosts, network operations or management devices, nodes, or configured on particular nodes of a distributed network (such as a communication network). As will be appreciated from the description below, and for reasons of computational efficiency, the components of the system may be arranged at any location within the distributed network without affecting the operation thereof. For example, the various components may reside in a domain master, a node, a domain management device such as a MIB, a network operation or management device, a transceiver, a station, an access point, or some combination thereof. Similarly, one or more functional portions of the system may be distributed between the transceiver and the associated computing device/system.

Furthermore, it should be understood that the various links of any communication channel/element/line including connecting elements may be wired or wireless links, or any combination thereof, or any other known or later developed element capable of supplying and/or communicating data to and from the connected elements. The term module, as used herein, may refer to any known or later developed hardware, circuitry, software, firmware, or combination thereof that is capable of performing the functionality associated with that element. The terms "determine," "calculate," and "compute," and variations thereof, as used herein, are used interchangeably and include any type of method, process, technique, mathematical operation, or protocol.

Furthermore, while some examples described herein relate to receiver portions of transceivers that perform some functions, or receiver portions of transceivers that perform some functions, the present disclosure is intended to include corresponding and complementary receiver-side or receiver-side functions, respectively, in both the same transceiver and/or other transceivers, and vice versa.

These examples are described with respect to enhanced communications. However, it should be understood that in general, the systems and methods herein will be equally applicable to any type of communication system in any environment utilizing any one or more protocols, including wired communication, wireless communication, power line communication, coaxial cable communication, fiber optic communication, and the like.

With respect to IEEE 802.11 and/orAnd/or +.>The low power transceiver and associated communication hardware, software, and communication channels describe example systems and methods. However, well-known structures and devices, which may be shown in block diagram form or otherwise summarized in order to avoid unnecessarily obscuring the present disclosure, are omitted from the following description.

Although the above-described flow diagrams have been discussed with respect to a particular sequence of events, it will be appreciated that changes to this sequence can occur without materially affecting the operation of the example. In addition, the example techniques illustrated herein are not limited to the examples specifically illustrated, but may be used with other examples as well and each described feature is claimed separately and separately.

The above-described system may be implemented on a wireless telecommunication device/system (e.g., IEEE 802.11 transceiver, etc.). Examples of wireless protocols that may be used with this technology include IEEE 802.11a, IEEE 802.11b, IEEE 802.11G, IEEE 802.11n, IEEE 802.11ac, IEEE 802.11ad, IEEE 802.11af, IEEE 802.11ah, IEEE 802.11ai, IEEE 802.11aj, IEEE 802.11aq, IEEE 802.11ax, wi-Fi, LTE, 4G,wireless HD, wiGig, wiGi, 3GPP, wireless LAN, wiMAX, densifsig, unifi SIG, 3GPP LAA (licensed assisted access), etc.

In addition, the systems, methods, and protocols may be implemented to modify one or more of a special purpose computer, a programmed microprocessor or microcontroller and peripheral integrated circuit elements, an ASIC or other integrated circuit, a digital signal processor, a hardwired electronic or logic circuit such as a discrete element circuit, a programmable logic device such as PLD, PLA, FPGA, PAL, a modem, a transmitter/receiver, any similar device, or the like. In general, any device capable of implementing a state machine, which in turn is capable of implementing the methods shown herein, may benefit from various communication methods, protocols, and techniques according to the disclosure provided herein.

Examples of processors as described herein may include, but are not limited to, at least one of the following:800 and 801, 4GLTE integration and 64 bit computation->610 and 615, having a 64-bit architecture>A7 processor, +.>M7 motion coprocessor->Series of,Cool Rui ^TM Serial processor, < > and->To the intensity ^TM Serial processor, < > and->Slush movement ^TM A series of processors,Anteng ^TM Serial processor, < > and->Cool Rui ^TM i5-4670K and i7-4770K22 nanometer HaswellCooli 5-3570K22nm Ivy Bridge, < >>FX ^TM Serial processor, < > and->FX ^TM 4300, FX-6300 and FX-835032nm Vishera, < >>Kaveri processor->Jacinto C6000 ^TM Automobile information entertainment processor>OMAP ^TM Automobile-level mobile processor->Cortex-M processor,Cortex-A and->926EJ-S processor, < > and->The AirForce BCM4704/BCM4703 wireless network processor, AR7100 wireless network processing unit, other industry equivalent processors, and may use any known or future developed standard, instruction set, library, and/or architecture to perform computing functions.

Furthermore, the disclosed methods can be readily implemented in software using an object or object-oriented software development environment that provides portable source code that can be used on a variety of computer or workstation platforms. Alternatively, the disclosed system may be implemented partially or entirely in hardware using standard logic circuits or VLSI designs. Whether software or hardware is used to implement a system according to the examples depends upon the speed and/or efficiency requirements of the system, the particular functionality, and the particular software or hardware system or microprocessor or microcomputer system used. Based on the functional description provided herein and the general basic knowledge in the computer and telecommunications arts, one of ordinary skill in the application arts can readily implement the communication systems, methods and protocols described herein in hardware and/or software using any known or later developed systems or structures, devices and/or software.

Furthermore, the disclosed techniques may be readily implemented in software and/or firmware that may be stored on a storage medium to improve the performance of a programmed general-purpose computer in conjunction with a controller and memory, special-purpose computer, microprocessor, or the like. In these cases, the systems and methods may be implemented as programs embedded on personal computers, such as applets, java. Or CGI scripts, resources residing on a server or computer workstation, routines embedded in a dedicated communication system or system component, etc. The system may also be implemented by physically incorporating the system and/or method into software and/or hardware systems, such as hardware and software systems of a communications transceiver.

It is therefore apparent that there has been provided, at least, a system and method for enhancing and improving a conversational user interface. Many alternatives, modifications, and variations will be apparent to those skilled in the art. Accordingly, the present disclosure is intended to embrace all such alternatives, modifications, equivalents and variations as fall within the spirit and scope of the present disclosure.

Claims

1. A system, comprising:

a plurality of access point, AP, devices configured to provide a wireless network at a station; and

A network management system, comprising:

a memory storing network data received from a plurality of AP devices, the network data collected by a plurality of client devices or the plurality of AP devices associated with the wireless network; and

one or more processors coupled to the memory and configured to determine one or more roaming quality assessments for each of a plurality of roaming events in the wireless network based on the network data, wherein one or more of the roaming quality assessments includes a sub-optimal roaming score for each of the plurality of roaming events based on a received signal strength indicator, RSSI, associated with the roaming event.

2. The system of claim 1, wherein one or more of the roaming quality assessments further comprises a sticky client score for each of the plurality of client devices based on a number of roaming options available to the client device and a type of roaming options available to the client device, wherein the sticky client score for each of the plurality of client devices is further based on whether a current RSSI of the client device is less than a fixed roaming threshold associated with the client device.

3. The system of claim 1, wherein the RSSI associated with the roaming event is a first RSSI of a first wireless signal received from a first AP associated with the client device prior to the roaming event.

4. The system of claim 1, wherein one or more of the processors are further configured to determine a roaming delay score for each of the plurality of roaming events.

5. The system of claim 4, wherein the processor is further configured to determine a roaming metric indicative of roaming performance of the wireless network based on the roaming quality assessment and the roaming delay score for each of the plurality of roaming events.

6. The system of claim 1, wherein the processor is further configured to identify an abnormal operation of the wireless network based on one or more of the roaming quality assessments and generate a notification indicating the abnormal operation of the wireless network and at least one remedial action to resolve the abnormal operation of the wireless network.

7. The system of claim 6, wherein the processor is further configured to automatically invoke at least one of the remedial actions to address the abnormal operation of the wireless network, wherein the remedial actions include reconfiguring operation of at least one of the plurality of APs in the wireless network.

8. The system of claim 1, wherein the processor is further configured to classify operation of the wireless network as one of normal operation and abnormal operation based on a trained machine learning model and one or more of the roaming quality assessments.

9. The system of claim 1, wherein one or more of the roaming quality assessments further includes an inter-band score for each of the plurality of client devices based on an RSSI delta associated with each of a plurality of inter-band roaming events.

10. The system of claim 1, further comprising: for at least one of the plurality of client devices, detecting a ping-pong roaming event performed by the client device comprising at least two consecutive roaming events, wherein a time difference between the performance of a first one of the at least two consecutive roaming events and the performance of a second one of the at least two consecutive roaming events satisfies a predetermined period of time.

11. The system of claim 1, further comprising: detecting, for at least one of the plurality of client devices, an excessive roaming event comprising at least three consecutive roaming events performed by the client device, wherein a time difference between the performance of a first one of the at least three consecutive roaming events by the client device and the performance of a second one of the at least two consecutive roaming events by the client device satisfies a predetermined time period, and wherein a time difference between the performance of the second one of the at least three consecutive roaming events by the client device and the performance of the third one of the at least three consecutive roaming events by the client device satisfies the predetermined time period.

12. A method, comprising:

receiving, by one or more processors of a network management system that manages a plurality of Access Point (AP) devices associated with a wireless network, network data indicating performance of the wireless network, the network data collected by a plurality of client devices or a plurality of AP devices associated with the wireless network; and is also provided with

Based on the network data, one or more roaming quality assessments for each of a plurality of roaming events in the wireless network are determined, wherein one or more of the roaming quality assessments includes a sub-optimal roaming score for each of the plurality of roaming events based on a received signal strength indicator, RSSI, associated with the roaming event.

13. The method of claim 12, wherein one or more of the roaming quality assessments further comprises a sticky client score for each of the plurality of client devices based on a number of roaming options available to the client device and a type of roaming options available to the client device, wherein the sticky client score for each of the plurality of client devices is further based on whether a current RSSI of the client device is less than a fixed roaming threshold associated with the client device.

14. The method of claim 13, wherein the processor is further configured to determine a roaming metric indicative of roaming performance of the wireless network based on the roaming quality assessment and the sticky client score for the each of the plurality of roaming events.

15. The method of claim 12, wherein the processor is further configured to identify abnormal operation of the wireless network based on one or more of the roaming quality assessments.

16. The method of claim 12, further comprising at least one of:

generating a notification indicating an abnormal operation of the wireless network and at least one remedial action to resolve the abnormal operation of the wireless network; or alternatively

Automatically invoking at least one remedial action to address abnormal operation of the wireless network, wherein the remedial action includes reconfiguring operation of at least one of the plurality of APs in the wireless network.

17. The method of claim 12, wherein determining one or more of the roaming quality assessments further comprises determining an inter-band score for each of the plurality of client devices based on an RSSI delta associated with each of a plurality of inter-band roaming events.

18. The method of claim 12, further comprising: a ping-pong roaming event comprising at least two consecutive roaming events is detected for at least one of the plurality of client devices, wherein a time difference between the execution of a first one of the at least two consecutive roaming events by the client device and the execution of a second one of the at least two consecutive roaming events by the client device satisfies a predetermined time period.

19. A system, comprising:

a network management system, comprising:

one or more processors; and

at least one memory including instructions that, when executed by one or more of the processors, configure the one or more processors to:

determining, for at least one client device associated with a wireless network comprising a plurality of access point devices, APs, whether a time difference between a first roaming event and a second roaming event in a sequence of two consecutive roaming events performed by the client device satisfies a predetermined time period; and is also provided with

And detecting a ping-pong roaming event based on determining that a current AP of the first roaming event in the plurality of APs matches a target AP of the second roaming event in the plurality of APs.

20. The system of claim 19, wherein the at least one memory includes instructions that, when executed by one or more of the processors, configure one or more of the processors to:

automatically invoking at least one remedial action to address a root cause of the ping-pong roaming event, wherein the remedial action includes reconfiguring operation of at least one of the plurality of APs in the wireless network.