WO2023047334A1 - Decision testing in automated networks - Google Patents

Abstract

Apparatuses, methods, and systems are disclosed for decision testing in automated networks. An apparatus (700) includes a transceiver (725) and a processor (705) coupled to the transceiver (725). The processor (705) is configured to cause the apparatus (700) to receive a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; select a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, create a network isolation within the network or network testing environment; and select a correct decision from the subset of the plurality of decisions according to the created network isolation.

Description

DECISION TESTING IN AUTOMATED NETWORKS

CROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of United States Provisional Patent Application Number 63/247,211, entitled “METHOD AND APPARATUS FOR DECISION TESTING IN AUTOMATED NETWORKS” and filed on September 22, 2022, for Ishan Vaishnavi, which is incorporated herein by reference.

FIELD

[0002] The subject matter disclosed herein relates generally to wireless communications and more particularly relates to decision testing in automated networks.

BACKGROUND

[0003] A decision stage in a closed loop is responsible for issuing decisions based on some conditions detected or notifications received from the network. There are cases in which the same conditions or notification could trigger completely different decisions. There is currently no way of simultaneously testing multiple decisions in the network to see which decision performs best for the same set of conditions.

BRIEF SUMMARY

[0004] Disclosed are solutions for decision testing in automated networks. The solutions may be implemented by apparatus, systems, methods, or computer program products.

[0005] In one embodiment, a first apparatus includes a transceiver and a processor coupled to the transceiver. In one embodiment, the processor is configured to cause the apparatus to receive a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; select a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, create a network isolation within the network or network testing environment; and select a correct decision from the subset of the plurality of decisions according to the created network isolation.

[0006] In one embodiment, a first method receives a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; selects a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, creates a network isolation within the network or network testing environment; and selects a correct decision from the subset of the plurality of decisions according to the created network isolation.

BRIEF DESCRIPTION OF THE DRAWINGS

[0007] A more particular description of the embodiments briefly described above will be rendered by reference to specific embodiments that are illustrated in the appended drawings. Understanding that these drawings depict only some embodiments and are not therefore to be considered to be limiting of scope, the embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings, in which:

[0008] Figure 1 is a schematic block diagram illustrating one embodiment of a wireless communication system for decision testing in automated networks;

[0009] Figure 2 depicts an example of open and closed control loops;

[0010] Figure 3 depicts a high level solution of the subject matter described herein;

[0011] Figure 4 depicts a solution example with policies;

[0012] Figure 5 is a diagram illustrating one embodiment of a new radio (“NR”) protocol stack;

[0013] Figure 6 is a block diagram illustrating one embodiment of a user equipment apparatus that may be used for decision testing in automated networks;

[0014] Figure 7 is a block diagram illustrating one embodiment of a network apparatus that may be used for decision testing in automated networks; and

[0015] Figure 8 is a flowchart diagram illustrating one embodiment of a method for decision testing in automated networks.

DETAILED DESCRIPTION

[0016] As will be appreciated by one skilled in the art, aspects of the embodiments may be embodied as a system, apparatus, method, or program product. Accordingly, embodiments may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects.

[0017] For example, the disclosed embodiments may be implemented as a hardware circuit comprising custom very-large-scale integration (“VLSI”) circuits or gate arrays, off-the-shelf semiconductors such as logic chips, transistors, or other discrete components. The disclosed embodiments may also be implemented in programmable hardware devices such as field programmable gate arrays, programmable array logic, programmable logic devices, or the like. As another example, the disclosed embodiments may include one or more physical or logical blocks of executable code which may, for instance, be organized as an object, procedure, or function.

[0018] Furthermore, embodiments may take the form of a program product embodied in one or more computer readable storage devices storing machine readable code, computer readable code, and/or program code, referred hereafter as code. The storage devices may be tangible, non- transitory, and/or non-transmission. The storage devices may not embody signals. In a certain embodiment, the storage devices only employ signals for accessing code.

[0019] Any combination of one or more computer readable medium may be utilized. The computer readable medium may be a computer readable storage medium. The computer readable storage medium may be a storage device storing the code. The storage device may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, holographic, micromechanical, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing.

[0020] More specific examples (a non-exhaustive list) of the storage device would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random-access memory (“RAM”), a read-only memory (“ROM”), an erasable programmable read-only memory (“EPROM” or Flash memory), a portable compact disc readonly memory (“CD-ROM”), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0021] Code for carrying out operations for embodiments may be any number of lines and may be written in any combination of one or more programming languages including an object- oriented programming language such as Python, Ruby, Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the “C” programming language, or the like, and/or machine languages such as assembly languages. The code may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (“LAN”), wireless LAN (“WLAN”), or a wide area network (“WAN”), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider (“ISP”)).

[0022] Furthermore, the described features, structures, or characteristics of the embodiments may be combined in any suitable manner. In the following description, numerous specific details are provided, such as examples of programming, software modules, user selections, network transactions, database queries, database structures, hardware modules, hardware circuits, hardware chips, etc., to provide a thorough understanding of embodiments. One skilled in the relevant art will recognize, however, that embodiments may be practiced without one or more of the specific details, or with other methods, components, materials, and so forth. In other instances, well-known structures, materials, or operations are not shown or described in detail to avoid obscuring aspects of an embodiment.

[0023] Reference throughout this specification to “one embodiment,” “an embodiment,” or similar language means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment. Thus, appearances of the phrases “in one embodiment,” “in an embodiment,” and similar language throughout this specification may, but do not necessarily, all refer to the same embodiment, but mean “one or more but not all embodiments” unless expressly specified otherwise. The terms “including,” “comprising,” “having,” and variations thereof mean “including but not limited to,” unless expressly specified otherwise. An enumerated listing of items does not imply that any or all of the items are mutually exclusive, unless expressly specified otherwise. The terms “a,” “an,” and “the” also refer to “one or more” unless expressly specified otherwise.

[0024] As used herein, a list with a conjunction of “and/or” includes any single item in the list or a combination of items in the list. For example, a list of A, B and/or C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C or a combination of A, B and C. As used herein, a list using the terminology “one or more of’ includes any single item in the list or a combination of items in the list. For example, one or more of A, B and C includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C or a combination of A, B and C. As used herein, a list using the terminology “one of’ includes one and only one of any single item in the list. For example, “one of A, B and C” includes only A, only B or only C and excludes combinations of A, B and C. As used herein, “a member selected from the group consisting of A, B, and C,” includes one and only one of A, B, or C, and excludes combinations of A, B, and C.” As used herein, “a member selected from the group consisting of A, B, and C and combinations thereof’ includes only A, only B, only C, a combination of A and B, a combination of B and C, a combination of A and C or a combination of A, B and C.

[0025] Aspects of the embodiments are described below with reference to schematic flowchart diagrams and/or schematic block diagrams of methods, apparatuses, systems, and program products according to embodiments. It will be understood that each block of the schematic flowchart diagrams and/or schematic block diagrams, and combinations of blocks in the schematic flowchart diagrams and/or schematic block diagrams, can be implemented by code. This code may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart diagrams and/or block diagrams.

[0026] The code may also be stored in a storage device that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the storage device produce an article of manufacture including instructions which implement the function/act specified in the flowchart diagrams and/or block diagrams.

[0027] The code may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other devices to produce a computer implemented process such that the code which execute on the computer or other programmable apparatus provide processes for implementing the functions/acts specified in the flowchart diagrams and/or block diagrams.

[0028] The flowchart diagrams and/or block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods, and program products according to various embodiments. In this regard, each block in the flowchart diagrams and/or block diagrams may represent a module, segment, or portion of code, which includes one or more executable instructions of the code for implementing the specified logical function(s).

[0029] It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Other steps and methods may be conceived that are equivalent in function, logic, or effect to one or more blocks, or portions thereof, of the illustrated Figures.

[0030] Although various arrow types and line types may be employed in the flowchart and/or block diagrams, they are understood not to limit the scope of the corresponding embodiments. Indeed, some arrows or other connectors may be used to indicate only the logical flow of the depicted embodiment. For instance, an arrow may indicate a waiting or monitoring period of unspecified duration between enumerated steps of the depicted embodiment. It will also be noted that each block of the block diagrams and/or flowchart diagrams, and combinations of blocks in the block diagrams and/or flowchart diagrams, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and code.

[0031] The description of elements in each figure may refer to elements of proceeding figures. Like numbers refer to like elements in all figures, including alternate embodiments of like elements.

[0032] Generally, the present disclosure describes systems, methods, and apparatuses for decision testing in automated networks. In certain embodiments, the methods may be performed using computer code embedded on a computer-readable medium. In certain embodiments, an apparatus or system may include a computer-readable medium containing computer-readable code which, when executed by a processor, causes the apparatus or system to perform at least a portion of the below described solutions.

[0033] A decision stage in a closed loop is responsible for issuing decisions based on some conditions detected or notifications received from the network. There are cases in which the same conditions or notification could trigger completely different decisions. There is currently no way of simultaneously testing multiple decisions in the network to see which decision performs best for the same set of conditions.

[0034] In one embodiment, a scenario addressed by the subject matter described herein is related to decision management in control loops, and, in particular, when decisions are taken based on the existing context and a specific condition detected in or notification received from the network. In one embodiment, for the same combination of context and condition, multiple decisions may be possible. For example, when a virtual network function (“VNF”) reaches 80% usage and it is predicted that more demand is expected to follow, the management system needs to increase resources to the network. This can be done by increasing the number of VNF instances or by assigning more resources to the existing VNF instances or any combination of those two approaches. However, which one of these decisions is the best performing one? Currently there is no way of testing multiple decision strategies in the network.

[0035] The telecom network is increasingly supporting automation in closed loops (“CLs”). CLs could take decisions based on existing policies. For example, a policy could be when an NF reaches 80% resource utilization, scale the NF up to have 40% more resources. However, another policy for the same event could have an action to create more instances of the NF instead. Which action is better for the network performance? Currently there is no way to test that on an operational network. [0036] The subject matter herein describes solutions that allow autonomic decisions to be made by the network that can now be tested and the best set of performing decisions can now be selected. This enables automation of autonomic decision making. For example, different decision strategies can be tested simultaneously for multiple closed loops. Another example is that multiple network policies that control the autonomous behavior of the network can be tested. Automated testing of multiple substitutable policies or decision-making strategies in the network assists in enhancing the autonomous operation of the network, thereby reducing operating expense costs for the network operator.

[0037] Figure 1 depicts a wireless communication system 100 supporting decision testing in automated networks, according to embodiments of the disclosure. In one embodiment, the wireless communication system 100 includes at least one remote unit 105, a radio access network (“RAN”) 120, and a mobile core network 130. The RAN 120 and the mobile core network 130 form a mobile communication network. The RAN 120 may be composed of a base unit 121 with which the remote unit 105 communicates using wireless communication links 115. Even though a specific number of remote units 105, base units 121, wireless communication links 115, RANs 120, and mobile core networks 130 are depicted in Figure 1, one of skill in the art will recognize that any number of remote units 105, base units 121, wireless communication links 115, RANs 120, and mobile core networks 130 may be included in the wireless communication system 100.

[0038] In one implementation, the RAN 120 is compliant with the 5G system specified in the Third Generation Partnership Project (“3GPP”) specifications. For example, the RAN 120 may be a New Generation Radio Access Network (“NG-RAN”), implementing NR RAT and/or 3GPP Long-Term Evolution (“LTE”) RAT. In another example, the RAN 120 may include non- 3GPP RAT (e.g., Wi-Fi® or Institute of Electrical and Electronics Engineers (“IEEE”) 802.11- family compliant WLAN). In another implementation, the RAN 120 is compliant with the LTE system specified in the 3GPP specifications. More generally, however, the wireless communication system 100 may implement some other open or proprietary communication network, for example Worldwide Interoperability for Microwave Access (“WiMAX”) or IEEE 802.16-family standards, among other networks. The present disclosure is not intended to be limited to the implementation of any particular wireless communication system architecture or protocol.

[0039] In one embodiment, the remote units 105 may include computing devices, such as desktop computers, laptop computers, personal digital assistants (“PDAs”), tablet computers, smart phones, smart televisions (e.g., televisions connected to the Internet), smart appliances (e.g., appliances connected to the Internet), set-top boxes, game consoles, security systems (including security cameras), vehicle on-board computers, network devices (e.g., routers, switches, modems), or the like. In some embodiments, the remote units 105 include wearable devices, such as smart watches, fitness bands, optical head-mounted displays, or the like. Moreover, the remote units 105 may be referred to as the UEs, subscriber units, mobiles, mobile stations, users, terminals, mobile terminals, fixed terminals, subscriber stations, user terminals, wireless transmit/receive unit (“WTRU”), a device, or by other terminology used in the art. In various embodiments, the remote unit 105 includes a subscriber identity and/or identification module (“SIM”) and the mobile equipment (“ME”) providing mobile termination functions (e.g., radio transmission, handover, speech encoding and decoding, error detection and correction, signaling and access to the SIM). In certain embodiments, the remote unit 105 may include a terminal equipment (“TE”) and/or be embedded in an appliance or device (e.g., a computing device, as described above).

[0040] The remote units 105 may communicate directly with one or more of the base units 121 in the RAN 120 via uplink (“UL”) and downlink (“DL”) communication signals. Furthermore, the UL and DL communication signals may be carried over the wireless communication links 123. Here, the RAN 120 is an intermediate network that provides the remote units 105 with access to the mobile core network 130.

[0041] In some embodiments, the remote units 105 communicate with an application server via a network connection with the mobile core network 130. For example, an application 107 (e.g., web browser, media client, telephone and/or Voice-over-Intemet-Protocol (“VoIP”) application) in a remote unit 105 may trigger the remote unit 105 to establish a protocol data unit (“PDU”) session (or other data connection) with the mobile core network 130 via the RAN 120. The mobile core network 130 then relays traffic between the remote unit 105 and the application server (e.g., the content server 151 in the packet data network 150) using the PDU session. The PDU session represents a logical connection between the remote unit 105 and the User Plane Function (“UPF”) 131.

[0042] In order to establish the PDU session (or PDN connection), the remote unit 105 must be registered with the mobile core network 130 (also referred to as ‘“attached to the mobile core network” in the context of a Fourth Generation (“4G”) system). Note that the remote unit 105 may establish one or more PDU sessions (or other data connections) with the mobile core network 130. As such, the remote unit 105 may have at least one PDU session for communicating with the packet data network 150, e.g., representative of the Internet. The remote unit 105 may establish additional PDU sessions for communicating with other data networks and/or other communication peers. [0043] In the context of a 5G system (“5GS”), the term “PDU Session” a data connection that provides end-to-end (“E2E”) user plane (“UP”) connectivity between the remote unit 105 and a specific Data Network (“DN”) through the UPF 131. A PDU Session supports one or more Quality of Service (“QoS”) Flows. In certain embodiments, there may be a one-to-one mapping between a QoS Flow and a QoS profile, such that all packets belonging to a specific QoS Flow have the same 5G QoS Identifier (“5QI”).

[0044] In the context of a 4G/ETE system, such as the Evolved Packet System (“EPS”), a Packet Data Network (“PDN”) connection (also referred to as EPS session) provides E2E UP connectivity between the remote unit and a PDN. The PDN connectivity procedure establishes an EPS Bearer, i.e., a tunnel between the remote unit 105 and a Packet Gateway (“PGW”, not shown) in the mobile core network 130. In certain embodiments, there is a one-to-one mapping between an EPS Bearer and a QoS profile, such that all packets belonging to a specific EPS Bearer have the same QoS Class Identifier (“QCI”).

[0045] The base units 121 may be distributed over a geographic region. In certain embodiments, a base unit 121 may also be referred to as an access terminal, an access point, a base, a base station, a Node-B (“NB”), an Evolved Node B (abbreviated as eNodeB or “eNB,” also known as Evolved Universal Terrestrial Radio Access Network (“E-UTRAN”) Node B), a 5G/NR Node B (“gNB”), a Home Node-B, a relay node, a RAN node, or by any other terminology used in the art. The base units 121 are generally part of a RAN, such as the RAN 120, that may include one or more controllers communi cably coupled to one or more corresponding base units 121. These and other elements of radio access network are not illustrated but are well known generally by those having ordinary skill in the art. The base units 121 connect to the mobile core network 130 via the RAN 120.

[0046] The base units 121 may serve a number of remote units 105 within a serving area, for example, a cell or a cell sector, via a wireless communication link 123. The base units 121 may communicate directly with one or more of the remote units 105 via communication signals. Generally, the base units 121 transmit DE communication signals to serve the remote units 105 in the time, frequency, and/or spatial domain. Furthermore, the DL communication signals may be carried over the wireless communication links 123. The wireless communication links 123 may be any suitable carrier in licensed or unlicensed radio spectrum. The wireless communication links 123 facilitate communication between one or more of the remote units 105 and/or one or more of the base units 121. Note that during NR-U operation, the base unit 121 and the remote unit 105 communicate over unlicensed radio spectrum. [0047] In one embodiment, the mobile core network 130 is a 5GC or an Evolved Packet Core (“EPC”), which may be coupled to a packet data network 150, like the Internet and private data networks, among other data networks. A remote unit 105 may have a subscription or other account with the mobile core network 130. Each mobile core network 130 belongs to a single public land mobile network (“PLMN”). The present disclosure is not intended to be limited to the implementation of any particular wireless communication system architecture or protocol.

[0048] The mobile core network 130 includes several network functions (“NFs”). As depicted, the mobile core network 130 includes at least one UPF 131. The mobile core network 130 also includes multiple control plane (“CP”) functions including, but not limited to, an Access and Mobility Management Function (“AMF”) 133 that serves the RAN 120, a Session Management Function (“SMF”) 135, a Network Exposure Function (“NEF”), a Policy Control Function (“PCF”) 137, a Unified Data Management function (“UDM”) and a User Data Repository (“UDR”).

[0049] The UPF(s) 131 is responsible for packet routing and forwarding, packet inspection, QoS handling, and external PDU session for interconnecting Data Network (“DN”), in the 5G architecture. The AMF 133 is responsible for termination of NAS signaling, NAS ciphering & integrity protection, registration management, connection management, mobility management, access authentication and authorization, security context management. The SMF 135 is responsible for session management (i.e., session establishment, modification, release), remote unit (i.e., UE) IP address allocation & management, DL data notification, and traffic steering configuration for UPF for proper traffic routing.

[0050] The NEF is responsible for making network data and resources easily accessible to customers and network partners. Service providers may activate new capabilities and expose them through APIs. These APIs allow third-party authorized applications to monitor and configure the network’s behavior for a number of different subscribers (i.e., connected devices with different applications). The PCF 137 is responsible for unified policy framework, providing policy rules to CP functions, access subscription information for policy decisions in UDR.

[0051] The UDM is responsible for generation of Authentication and Key Agreement (“AKA”) credentials, user identification handling, access authorization, subscription management. The UDR is a repository of subscriber information and can be used to service a number of network functions. For example, the UDR may store subscription data, policy-related data, subscriber- related data that is permitted to be exposed to third party applications, and the like. In some embodiments, the UDM is co-located with the UDR, depicted as combined entity “UDM/UDR” [0052] In various embodiments, the mobile core network 130 may also include an Authentication Server Function (“AUSF”) (which acts as an authentication server), a Network Repository Function (“NRF”) (which provides NF service registration and discovery, enabling NFs to identify appropriate services in one another and communicate with each other over Application Programming Interfaces (“APIs”)), or other NFs defined for the 5GC. In certain embodiments, the mobile core network 130 may include an authentication, authorization, and accounting (“AAA”) server.

[0053] In various embodiments, the mobile core network 130 supports different types of mobile data connections and different types of network slices, wherein each mobile data connection utilizes a specific network slice. Here, a “network slice” refers to a portion of the mobile core network 130 optimized for a certain traffic type or communication service . A network instance may be identified by a single-network slice selection assistance information (“S-NSSAI,”) while a set of network slices for which the remote unit 105 is authorized to use is identified by network slice selection assistance information (“NSSAI”).

[0054] Here, “NSSAI” refers to a vector value including one or more S-NSSAI values. In certain embodiments, the various network slices may include separate instances of network functions, such as the SMF 135 and UPF 131. In some embodiments, the different network slices may share some common network functions, such as the AMF 133. The different network slices are not shown in Figure 1 for ease of illustration, but their support is assumed. Where different network slices are deployed, the mobile core network 130 may include a Network Slice Selection Function (“NSSF”) which is responsible for selecting of the Network Slice instances to serve the remote unit 105, determining the allowed NSSAI, determining the AMF set to be used to serve the remote unit 105.

[0055] Although specific numbers and types of network functions are depicted in Figure 1 , one of skill in the art will recognize that any number and type of network functions may be included in the mobile core network 130. Moreover, in an LTE variant where the mobile core network 130 comprises an EPC, the depicted network functions may be replaced with appropriate EPC entities, such as a Mobility Management Entity (“MME”), a Serving Gateway (“SGW”), a PGW, a Home Subscriber Server (“HSS”), and the like. For example, the AMF 133 may be mapped to an MME, the SMF 135 may be mapped to a control plane portion of a PGW and/or to an MME, the UPF 131 may be mapped to an SGW and a user plane portion of the PGW, the UDM/UDR 139 may be mapped to an HSS, etc.

[0056] While Figure 1 depicts components of a 5G RAN and a 5G core network, the described embodiments apply to other types of communication networks and RATs, including IEEE 802.11 variants, Global System for Mobile Communications (“GSM”, i.e., a 2G digital cellular network), General Packet Radio Service (“GPRS”), UMTS, LTE variants, CDMA 2000, Bluetooth, ZigBee, Sigfox, and the like.

[0057] In the following descriptions, the term “gNB” is used for the base station but it is replaceable by any other radio access node, e.g., RAN node, eNB, Base Station (“BS”), Access Point (“AP”), NR, etc. Further the operations are described mainly in the context of 5G NR. However, the proposed solutions/methods are also equally applicable to other mobile communication systems supporting decision testing in automated networks.

[0058] As background, there are two extremes of open and closed control loops - open control loops and closed control loops as shown in Figure 2. Open loops involve the operator to be a part of at least one of the stages in the loop, while in the closed loop stage the operator only defines a goal for the closed control loop and the loop once configured runs automatically. Both control loops attempt in controlling the status of a managed object trying to keep it to as close as possible to an operator specified goal.

[0059] Network policies, in one embodiment, are a set of rules that provide a mechanism for default decisions to be taken without human intervention in the network. Typically, they are organized as event-action(s) or event-condition-action(s) tuples, where in the event is an occurrence in the network, a condition is a constraint to be met and action(s) are a list of configurations or changes to be carried out when the event happens, and the conditions are met. Policies may include a set of conditions and corresponding actions with (optional) event triggers.

[0060] Policies are frequently used in many places in network management and operations to take automatic decisions. A policy-based decision stage consists of an event, condition, and action tuple. When an event is detected in the network and a set of specific conditions is met the actions specified by the ECA policy are carried out by the policy enforcer. The actions essentially are a “decision” to be executed in the network. Currently there is no way to simultaneously compare one action with another in an operation network. Note that one of either the condition or the event may be missing in the policy description.

[0061] The high-level solution, shown in Figure 3, is to test multiple set of possible decisions simultaneously. This disclosure refers to a case where the possible set of decisions is bounded (as is the case for any practical network management application).

[0062] At step 1 305, a subset (size N) of the possible decisions is chosen for testing in the operational environment and configured in the Decision Testing Service (“DTS”), with individual probability of selection for the decisions in the set, given of course the same context and the set of events that can trigger the decision. As such, the decisions in the set are considered as alternatives to each other. Each decision may be triggered only when a specified condition or an event in the network is met. In this case, when the decision is associated together with the event or a condition to form an event-condition-action policy for the network.

[0063] At step 2 310, for each possible decision a network isolation is created. As used herein, network isolation refers to a separation of the network environment for the purposes of executing the decision, such that the decisions do not influence each other and can be separately evaluated to the extent possible. An example of such isolation is creation of N identical network slice instances each following one of the N different decisions. Another example is the separation of users on a telecom network of the network into N different groups each uses different decisions.

[0064] Note: In one embodiment, the separation is required to ensure that: i. The performance of the different networks that use the different decisions can be separately measured; and ii. the decisions do not influence or conflict with each other.

[0065] If these two considerations are not applicable, the selection and deployment of separate network isolations is not necessary.

[0066] At step 3 315, the network is configured to select the correct decision in the network according to the created network isolation.

[0067] At step 4 320, the performance of each of the network isolations is recorded and sent for review based on a given known set of key performance indicators (“KPIs”) to evaluate which decision(s) were the most suitable.

[0068] At step 5 325, based on the results, a new set of decisions is provided for testing and unsuitable or error causing decisions are dropped and a new set of decisions to be tested are provided.

[0069] In one embodiment, steps 2 to 4 are repeated till a desirable subset of decisions that achieve a desirable performance over the network is achieved.

[0070] In one embodiment, a different decision implies that the strategy to solve the same or a related problem is different. Therefore, different decisions may or may not be triggered by identical conditions and events. Taking the example of NF load at 80% and expected increase in traffic, another strategy could be to drop randomly selected new NF requests in place of increasing resources. As it relates to the subject matter described herein, the strategy the network employs to solve a problem is different, which is equivalent to saying that the decision taken by the system is different. In other words, the decision that needs to be tested is anywhere along the policy continuum from the imperative to declarative to intense. [0071] Figure 4 depicts a procedure flow 400 for network slice management. It is assumed that the decision is expressed in policies that effect the performance of an entire network slice instance. Hence in this example the PMSP acts as the DTS.

[0072] At step 1 (see messaging 405), the policy management service producer (“PMSP”) 404 receives a request to test N decisions expressed as policies. In one embodiment, the policies are alternatives to each other. The request may include: i . A policy selection probability distribution or the cases in which to use which policy, e.g., use policy i for user group i using a set of network entities. Policies could, for example, express how a UE’s session requests in a network slice instance are handled. ii. The network entity the policy is applicable to. iii. The final number of policies to be selected. iv. The KPI (or collection of KPIs) to be optimized.

[0073] Note, the policy may be a high level declarative policy that may further be translated into multiple imperative policies in the network.

[0074] For steps 2-5 below, in one embodiment, loop/repeat steps 2-5 for all instances i.

[0075] At step 2 (see messaging 410), the PMSP 404 requests the provisioning service producer 406 to instantiate N network slice instances (corresponding to the N policies) each with the probability or conditions for selection provided in step 1. An example of policies is 20% of all users are randomly assigned to policy 1 and the remaining 80% to policy 2. An example of conditional separation could be that all users in geography 1 are assigned to network slice instance (“NSI”) 1 and all users in geography 2 are assigned to NSI 2.

[0076] At step 3 (see block 415), provisioning procedure is performed, e.g., as described in item xx of TS 28.530, which is incorporated herein by reference.

[0077] At step 4 (see block 420), the probability of selection is configured in the appropriate selection entity. A selection entity, as used herein, is the software or hardware entity responsible for selecting the appropriate instance of the network entity or the appropriate policy corresponding to the network entity.

[0078] At step 5 (see messaging 425), confirmation that the entity i with the appropriate selection probability or conditions is allocated.

[0079] At step 6 (see block 430), the normal monitoring procedures of all network entities is performed (e.g., for NSI, monitoring in accordance with TS 28.540, which is incorporated herein by reference).

[0080] For steps 7-10, loop/repeat steps 7-10 until appropriate policies are selected. [0081] At step 7 (see messaging 435), based on the performance of different policies, the consumer 402 may select: i. a subset of policies to continue with ii. different distribution of probabilities, or users or conditions. iii. Different network entities the policies may be applicable to iv. or different KPIs to report on regarding the performance.

[0082] This step could also explicitly request removing policies.

[0083] At step 8 (see messaging 440), the selected changes are communicated to the PMSP 404.

[0084] At step 9 (see messaging 445), the updated changes are configured in the entity i (for NSI, it is using the modified NSI request).

[0085] At step 10 (see block 450), the probability is reconfigured in the selection entities to reflect the changes requested in step 8 for entity i.

[0086] At step 11 (see messaging 455), confirmation is received that the entity i has been correctly configured and instantiated.

[0087] At step 12 (see messaging 460), entities are removed that are no longer used (this step could be merged with step 9).

[0088] At step 13 (see block 465), the deletion procedure is performed, e.g., according to TS 28.530.

[0089] At step 14 (see block 470), the normal monitoring procedures for the network entities are used to collect performance data (e.g., for NSI, monitoring in accordance with TS28.540).

[0090] Figure 5 depicts a NR protocol stack 500, according to embodiments of the disclosure. While Figure 5 shows the remote unit 105, the base unit 121 and the mobile core network 130, these are representative of a set of UEs interacting with a RAN node and a NF (e.g., AMF) in a core network. As depicted, the protocol stack 500 comprises a User Plane protocol stack 501 and a Control Plane protocol stack 503. The User Plane protocol stack 501 includes a physical (“PHY”) layer 505, a Medium Access Control (“MAC”) sublayer 510, a Radio Uink Control (“RLC”) sublayer 515, a Packet Data Convergence Protocol (“PDCP”) sublayer 520, and Service Data Adaptation Protocol (“SDAP”) sublayer 525. The Control Plane protocol stack 503 also includes a physical layer 505, a MAC sublayer 510, a RLC sublayer 515, and a PDCP sublayer 520. The Control Place protocol stack 503 also includes a Radio Resource Control (“RRC”) sublayer 530 and a Non-Access Stratum (“NAS”) layer 545. [0091] The AS protocol stack for the Control Plane protocol stack 503 consists of at least RRC, PDCP, RLC and MAC sublayers, and the physical layer. The AS protocol stack for the User Plane protocol stack 501 consists of at least SDAP, PDCP, RLC and MAC sublayers, and the physical layer. The Layer-2 (“L2”) is split into the SDAP, PDCP, RLC and MAC sublayers. The Layer-3 (“L3”) includes the RRC sublayer 530 and the NAS layer 535 for the control plane and includes, e.g., an Internet Protocol (“IP”) layer or PDU Layer (note depicted) for the user plane. LI and L2 are referred to as “lower layers” such as PUCCH/PUSCH or MAC CE, while L3 and above (e.g., transport layer, application layer) are referred to as “higher layers” or “upper layers” such as RRC.

[0092] The physical layer 505 offers transport channels to the MAC sublayer 510. The MAC sublayer 510 offers logical channels to the RLC sublayer 515. The RLC sublayer 515 offers RLC channels to the PDCP sublayer 520. The PDCP sublayer 520 offers radio bearers to the SDAP sublayer 525 and/or RRC sublayer 530. The SDAP sublayer 525 offers QoS flows to the mobile core network 130 (e.g., 5GC). The RRC sublayer 530 provides for the addition, modification, and release of Carrier Aggregation and/or Dual Connectivity. The RRC sublayer 530 also manages the establishment, configuration, maintenance, and release of Signaling Radio Bearers (“SRBs”) and Data Radio Bearers (“DRBs”). In certain embodiments, a RRC entity functions for detection of and recovery from radio link failure.

[0093] Figure 6 depicts a user equipment apparatus 600 that may be used for decision testing in automated networks, according to embodiments of the disclosure. In various embodiments, the user equipment apparatus 600 is used to implement one or more of the solutions described above. The user equipment apparatus 600 may be one embodiment of a UE, such as the remote unit 105 and/or the UE 205, as described above. Furthermore, the user equipment apparatus 600 may include a processor 605, a memory 610, an input device 615, an output device 620, and a transceiver 625. In some embodiments, the input device 615 and the output device 620 are combined into a single device, such as a touchscreen. In certain embodiments, the user equipment apparatus 600 may not include any input device 615 and/or output device 620. In various embodiments, the user equipment apparatus 600 may include one or more of: the processor 605, the memory 610, and the transceiver 625, and may not include the input device 615 and/or the output device 620.

[0094] As depicted, the transceiver 625 includes at least one transmitter 630 and at least one receiver 635. Here, the transceiver 625 communicates with one or more base units 121. Additionally, the transceiver 625 may support at least one network interface 640 and/or application interface 645. The application interface(s) 645 may support one or more APIs. The network interface(s) 640 may support 3GPP reference points, such as Uu and PC5. Other network interfaces 640 may be supported, as understood by one of ordinary skill in the art.

[0095] The processor 605, in one embodiment, may include any known controller capable of executing computer-readable instructions and/or capable of performing logical operations. For example, the processor 605 may be a microcontroller, a microprocessor, a central processing unit (“CPU”), a graphics processing unit (“GPU”), an auxiliary processing unit, a field programmable gate array (“FPGA”), a digital signal processor (“DSP”), a co-processor, an application-specific processor, or similar programmable controller. In some embodiments, the processor 605 executes instructions stored in the memory 610 to perform the methods and routines described herein. The processor 605 is communicatively coupled to the memory 610, the input device 615, the output device 620, and the transceiver 625. In certain embodiments, the processor 605 may include an application processor (also known as “main processor”) which manages application-domain and operating system (“OS”) functions and a baseband processor (also known as “baseband radio processor”) which manages radio functions.

[0096] The memory 610, in one embodiment, is a computer readable storage medium. In some embodiments, the memory 610 includes volatile computer storage media. For example, the memory 610 may include a RAM, including dynamic RAM (“DRAM”), synchronous dynamic RAM (“SDRAM”), and/or static RAM (“SRAM”). In some embodiments, the memory 610 includes non-volatile computer storage media. For example, the memory 610 may include a hard disk drive, a flash memory, or any other suitable non-volatile computer storage device. In some embodiments, the memory 610 includes both volatile and non-volatile computer storage media.

[0097] In some embodiments, the memory 610 stores data related to decision testing in automated networks. For example, the memory 610 may store parameters, configurations, resource assignments, policies, and the like as described above. In certain embodiments, the memory 610 also stores program code and related data, such as an operating system or other controller algorithms operating on the user equipment apparatus 600, and one or more software applications.

[0098] The input device 615, in one embodiment, may include any known computer input device including a touch panel, a button, a keyboard, a stylus, a microphone, or the like. In some embodiments, the input device 615 may be integrated with the output device 620, for example, as a touchscreen or similar touch -sensitive display. In some embodiments, the input device 615 includes a touchscreen such that text may be input using a virtual keyboard displayed on the touchscreen and/or by handwriting on the touchscreen. In some embodiments, the input device 615 includes two or more different devices, such as a keyboard and a touch panel. [0099] The output device 620, in one embodiment, is designed to output visual, audible, and/or haptic signals. In some embodiments, the output device 620 includes an electronically controllable display or display device capable of outputting visual data to a user. For example, the output device 620 may include, but is not limited to, an LCD display, an LED display, an OLED display, a projector, or similar display device capable of outputting images, text, or the like to a user. As another, non-limiting, example, the output device 620 may include a wearable display separate from, but communicatively coupled to, the rest of the user equipment apparatus 600, such as a smart watch, smart glasses, a heads-up display, or the like. Further, the output device 620 may be a component of a smart phone, a personal digital assistant, a television, a table computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, or the like.

[00100] In certain embodiments, the output device 620 includes one or more speakers for producing sound. For example, the output device 620 may produce an audible alert or notification (e.g., a beep or chime). In some embodiments, the output device 620 includes one or more haptic devices for producing vibrations, motion, or other haptic feedback. In some embodiments, all or portions of the output device 620 may be integrated with the input device 615. For example, the input device 615 and output device 620 may form a touchscreen or similar touch- sensitive display. In other embodiments, the output device 620 may be located near the input device 615.

[00101] The transceiver 625 includes at least transmitter 630 and at least one receiver 635. The transceiver 625 may be used to provide UL communication signals to a base unit 121 and to receive DL communication signals from the base unit 121, as described herein. Similarly, the transceiver 625 may be used to transmit and receive SL signals (e.g., V2X communication), as described herein. Although only one transmitter 630 and one receiver 635 are illustrated, the user equipment apparatus 600 may have any suitable number of transmitters 630 and receivers 635. Further, the transmitter(s) 630 and the receiver(s) 635 may be any suitable type of transmitters and receivers. In one embodiment, the transceiver 625 includes a first transmitter/receiver pair used to communicate with a mobile communication network over licensed radio spectrum and a second transmitter/receiver pair used to communicate with a mobile communication network over unlicensed radio spectrum.

[00102] In certain embodiments, the first transmitter/receiver pair used to communicate with a mobile communication network over licensed radio spectrum and the second transmitter/receiver pair used to communicate with a mobile communication network over unlicensed radio spectrum may be combined into a single transceiver unit, for example a single chip performing functions for use with both licensed and unlicensed radio spectrum. In some embodiments, the first transmitter/receiver pair and the second transmitter/receiver pair may share one or more hardware components. For example, certain transceivers 625, transmitters 630, and receivers 635 may be implemented as physically separate components that access a shared hardware resource and/or software resource, such as for example, the network interface 640.

[00103] In various embodiments, one or more transmitters 630 and/or one or more receivers 635 may be implemented and/or integrated into a single hardware component, such as a multi-transceiver chip, a system-on-a-chip, an ASIC, or other type of hardware component. In certain embodiments, one or more transmitters 630 and/or one or more receivers 635 may be implemented and/or integrated into a multi-chip module. In some embodiments, other components such as the network interface 640 or other hardware components/circuits may be integrated with any number of transmitters 630 and/or receivers 635 into a single chip. In such embodiment, the transmitters 630 and receivers 635 may be logically configured as a transceiver 625 that uses one more common control signals or as modular transmitters 630 and receivers 635 implemented in the same hardware chip or in a multi-chip module.

[00104] Figure 7 depicts one embodiment of a network apparatus 700 that may be used for decision testing in automated networks, according to embodiments of the disclosure. In some embodiments, the network apparatus 700 may be one embodiment of a RAN node and its supporting hardware, such as the base unit 121 and/or gNB, described above. Furthermore, network apparatus 700 may include a processor 705, a memory 710, an input device 715, an output device 720, and a transceiver 725. In certain embodiments, the network apparatus 700 does not include any input device 715 and/or output device 720.

[00105] As depicted, the transceiver 725 includes at least one transmitter 730 and at least one receiver 735. Here, the transceiver 725 communicates with one or more remote units 105. Additionally, the transceiver 725 may support at least one network interface 740 and/or application interface 745. The application interface(s) 745 may support one or more APIs. The network interface(s) 740 may support 3GPP reference points, such as Uu, Nl, N2, N3, N5, N6 and/or N7 interfaces. Other network interfaces 740 may be supported, as understood by one of ordinary skill in the art.

[00106] When implementing an NEF, the network interface(s) 740 may include an interface for communicating with an application function (i.e., N5) and with at least one network function (e.g., UDR, SFC function, UPF) in a mobile communication network, such as the mobile core network 130.

[00107] The processor 705, in one embodiment, may include any known controller capable of executing computer-readable instructions and/or capable of performing logical operations. For example, the processor 705 may be a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, an FPGA, a DSP, a co-processor, an application-specific processor, or similar programmable controller. In some embodiments, the processor 705 executes instructions stored in the memory 710 to perform the methods and routines described herein. The processor 705 is communicatively coupled to the memory 710, the input device 715, the output device 720, and the transceiver 725. In certain embodiments, the processor 705 may include an application processor (also known as “main processor”) which manages application-domain and OS functions and a baseband processor (also known as “baseband radio processor”) which manages radio function. In various embodiments, the processor 705 controls the network apparatus 700 to implement the above described network entity behaviors (e.g., of the gNB) for decision testing in automated networks.

[00108] The memory 710, in one embodiment, is a computer readable storage medium. In some embodiments, the memory 710 includes volatile computer storage media. For example, the memory 710 may include a RAM, including DRAM, SDRAM, and/or SRAM. In some embodiments, the memory 710 includes non-volatile computer storage media. For example, the memory 710 may include a hard disk drive, a flash memory, or any other suitable non-volatile computer storage device. In some embodiments, the memory 710 includes both volatile and nonvolatile computer storage media.

[00109] In some embodiments, the memory 710 stores data relating to decision testing in automated networks. For example, the memory 710 may store parameters, configurations, resource assignments, policies, and the like as described above. In certain embodiments, the memory 710 also stores program code and related data, such as an OS or other controller algorithms operating on the network apparatus 700, and one or more software applications.

[00110] The input device 715, in one embodiment, may include any known computer input device including a touch panel, a button, a keyboard, a stylus, a microphone, or the like . In some embodiments, the input device 715 may be integrated with the output device 720, for example, as a touchscreen or similar touch-sensitive display. In some embodiments, the input device 715 includes a touchscreen such that text may be input using a virtual keyboard displayed on the touchscreen and/or by handwriting on the touchscreen. In some embodiments, the input device 715 includes two or more different devices, such as a keyboard and a touch panel.

[00111] The output device 720, in one embodiment, may include any known electronically controllable display or display device. The output device 720 may be designed to output visual, audible, and/or haptic signals. In some embodiments, the output device 720 includes an electronic display capable of outputting visual data to a user. Further, the output device 720 may be a component of a smart phone, a personal digital assistant, a television, a table computer, a notebook (laptop) computer, a personal computer, a vehicle dashboard, or the like.

[00112] In certain embodiments, the output device 720 includes one or more speakers for producing sound. For example, the output device 720 may produce an audible alert or notification (e.g., a beep or chime). In some embodiments, the output device 720 includes one or more haptic devices for producing vibrations, motion, or other haptic feedback. In some embodiments, all or portions of the output device 720 may be integrated with the input device 715. For example, the input device 715 and output device 720 may form a touchscreen or similar touch- sensitive display. In other embodiments, all or portions of the output device 720 may be located near the input device 715.

[00113] As discussed above, the transceiver 725 may communicate with one or more remote units and/or with one or more interworking functions that provide access to one or more PLMNs. The transceiver 725 may also communicate with one or more network functions (e.g., in the mobile core network 80). The transceiver 725 operates under the control of the processor 705 to transmit messages, data, and other signals and also to receive messages, data, and other signals. For example, the processor 705 may selectively activate the transceiver (or portions thereof) at particular times in order to send and receive messages.

[00114] The transceiver 725 may include one or more transmitters 730 and one or more receivers 735. In certain embodiments, the one or more transmitters 730 and/or the one or more receivers 735 may share transceiver hardware and/or circuitry. For example, the one or more transmitters 730 and/or the one or more receivers 735 may share antenna(s), antenna tuner(s), amplifier(s), fdter(s), oscillators), mixer(s), modulator/demodulator(s), power supply, and the like. In one embodiment, the transceiver 725 implements multiple logical transceivers using different communication protocols or protocol stacks, while using common physical hardware.

[00115] In one embodiment, the processor 705 receives a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; selects a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, creates a network isolation within the network or network testing environment; and selects a correct decision from the subset of the plurality of decisions according to the created network isolation. [00116] In one embodiment, the decisions of the plurality of decisions are alternatives to one another such that each decision does not influence another decision and can be evaluated separately.

[00117] In one embodiment, the decisions of the plurality of decisions are triggered in response to a condition, an event, or a combination of both, in the network.

[00118] In one embodiment, different decisions of the plurality of decisions are triggered by the same condition, event, or a combination of both, in the network.

[00119] In one embodiment, each created network isolation comprises a different and identical network slice instance for each decision.

[00120] In one embodiment, each created network isolation comprises a separation of users on the network into different groups where each group uses different decisions of the plurality of decisions.

[00121] In one embodiment, the processor 705 records performance of each of the created network isolations.

[00122] In one embodiment, the processor 705 reviews the recorded performance of each of the created network isolations according to a predefined set of key performance indicators.

[00123] In one embodiment, the processor 705 selects a second subset of the plurality of decisions fortesting based on the review of the recorded performance.

[00124] In one embodiment, the processor 705 drops decisions that have a recorded performance below a threshold.

[00125] In one embodiment, the processor 705 iteratively selects the next subset of the plurality of decisions for testing until an end condition is satisfied.

[00126] In one embodiment, the processor 705 sets the probability of being selected for each decision according to a probability distribution.

[00127] In one embodiment, each of the plurality of decisions comprises a different network policy.

[00128] In one embodiment, the network testing environment comprises a plurality of different network entities that are configured to perform the testing, the plurality of different network entities comprising a network slice instance, a network slice subnet instance, a network function, or a combination thereof.

[00129] Figure 8 is a flowchart diagram of a method 800 for decision testing in automated networks. The method 800 may be performed by a network entity as described herein, for example, the base unit 121, a gNB, and/or the network equipment apparatus 700. In some embodiments, the method 800 may be performed by a processor executing program code, for example, a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, a FPGA, or the like.

[00130] In one embodiment, the method 800 begins and receives 805 a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected. In one embodiment, the method 800 selects 810 a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment. In one embodiment, for each decision in the subset, the method 800 creates 815 a network isolation within the network or network testing environment. In one embodiment, the method 800 selects 820 a correct decision from the subset of the plurality of decisions according to the created network isolation, and the method 800 ends.

[00131] A first apparatus is disclosed for decision testing in automated networks. The first apparatus may include a network entity as described herein, for example, the base unit 121, a gNB, and/or the network equipment apparatus 700. In some embodiments, the first apparatus may include a processor executing program code, for example, a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, a FPGA, or the like.

[00132] In one embodiment, the first apparatus includes a transceiver and a processor coupled to the transceiver. In one embodiment, the processor is configured to cause the apparatus to receive a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; select a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, create a network isolation within the network or network testing environment; and select a correct decision from the subset of the plurality of decisions according to the created network isolation.

[00133] In one embodiment, the decisions of the plurality of decisions are alternatives to one another such that each decision does not influence another decision and can be evaluated separately.

[00134] In one embodiment, the decisions of the plurality of decisions are triggered in response to a condition, an event, or a combination of both, in the network.

[00135] In one embodiment, different decisions of the plurality of decisions are triggered by the same condition, event, or a combination of both, in the network.

[00136] In one embodiment, each created network isolation comprises a different and identical network slice instance for each decision. [00137] In one embodiment, each created network isolation comprises a separation of users on the network into different groups where each group uses different decisions of the plurality of decisions.

[00138] In one embodiment, the processor is configured to cause the apparatus to record performance of each of the created network isolations.

[00139] In one embodiment, the processor is configured to cause the apparatus to review the recorded performance of each of the created network isolations according to a predefined set of key performance indicators.

[00140] In one embodiment, the processor is configured to cause the apparatus to select a second subset of the plurality of decisions for testing based on the review of the recorded performance.

[00141] In one embodiment, the processor is configured to cause the apparatus to drop decisions that have a recorded performance below a threshold.

[00142] In one embodiment, the processor is configured to cause the apparatus to iteratively select the next subset of the plurality of decisions for testing until an end condition is satisfied.

[00143] In one embodiment, the processor is configured to cause the apparatus to set the probability of being selected for each decision according to a probability distribution.

[00144] In one embodiment, each of the plurality of decisions comprises a different network policy.

[00145] In one embodiment, the network testing environment comprises a plurality of different network entities that are configured to perform the testing, the plurality of different network entities comprising a network slice instance, a network slice subnet instance, a network function, or a combination thereof.

[00146] A first method is disclosed for decision testing in automated networks. The first method may be performed by a network entity as described herein, for example, the base unit 121, agNB, and/orthe network equipment apparatus 700. In some embodiments, the first method may be performed by a processor executing program code, for example, a microcontroller, a microprocessor, a CPU, a GPU, an auxiliary processing unit, a FPGA, or the like.

[00147] In one embodiment, the first method receives a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; selects a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, creates a network isolation within the network or network testing environment; and selects a correct decision from the subset of the plurality of decisions according to the created network isolation.

[00148] In one embodiment, the decisions of the plurality of decisions are alternatives to one another such that each decision does not influence another decision and can be evaluated separately.

[00149] In one embodiment, the decisions of the plurality of decisions are triggered in response to a condition, an event, or a combination of both, in the network.

[00150] In one embodiment, different decisions of the plurality of decisions are triggered by the same condition, event, or a combination of both, in the network.

[00151] In one embodiment, each created network isolation comprises a different and identical network slice instance for each decision.

[00152] In one embodiment, each created network isolation comprises a separation of users on the network into different groups where each group uses different decisions of the plurality of decisions.

[00153] In one embodiment, the first method records performance of each of the created network isolations.

[00154] In one embodiment, the first method reviews the recorded performance of each of the created network isolations according to a predefined set of key performance indicators.

[00155] In one embodiment, the first method selects a second subset of the plurality of decisions fortesting based on the review of the recorded performance.

[00156] In one embodiment, the first method drops decisions that have a recorded performance below a threshold.

[00157] In one embodiment, the first method iteratively selects the next subset of the plurality of decisions for testing until an end condition is satisfied.

[00158] In one embodiment, the first method sets the probability of being selected for each decision according to a probability distribution.

[00159] In one embodiment, each of the plurality of decisions comprises a different network policy.

[00160] In one embodiment, the network testing environment comprises a plurality of different network entities that are configured to perform the testing, the plurality of different network entities comprising a network slice instance, a network slice subnet instance, a network function, or a combination thereof.

[00161] Embodiments may be practiced in other specific forms. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore, indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.

Claims

27 An apparatus, comprising: a transceiver; and a processor coupled to the transceiver, the processor configured to cause the apparatus to: receive a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; select a subset of the plurality of decisions fortesting, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, create a network isolation within the network or network testing environment; and select a correct decision from the subset of the plurality of decisions according to the created network isolation. The apparatus of claim 1, wherein the decisions of the plurality of decisions are alternatives to one another such that each decision does not influence another decision and can be evaluated separately. The apparatus of claim 1, wherein the decisions of the plurality of decisions are triggered in response to a condition, an event, or a combination of both, in the network. The apparatus of claim 3, wherein different decisions of the plurality of decisions are triggered by the same condition, event, or a combination of both, in the network. The apparatus of claim 1, wherein each created network isolation comprises a different and identical network slice instance for each decision. The apparatus of claim 1, wherein each created network isolation comprises a separation of users on the network into different groups where each group uses different decisions of the plurality of decisions. The apparatus of claim 1, wherein the processor is configured to cause the apparatus to record performance of each of the created network isolations. The apparatus of claim 7, wherein the processor is configured to cause the apparatus to review the recorded performance of each of the created network isolations according to a predefined set of key performance indicators. The apparatus of claim 8, wherein the processor is configured to cause the apparatus to select a second subset of the plurality of decisions for testing based on the review of the recorded performance. The apparatus of claim 8, wherein the processor is configured to cause the apparatus to drop decisions that have a recorded performance below a threshold. The apparatus of claim 1, wherein the processor is configured to cause the apparatus to iteratively select the next subset of the plurality of decisions for testing until an end condition is satisfied. The apparatus of claim 1, wherein the processor is configured to cause the apparatus to set the probability of being selected for each decision according to a probability distribution. The apparatus of claim 1, wherein each of the plurality of decisions comprises a different network policy. The apparatus of claim 1, wherein the network testing environment comprises a plurality of different network entities that are configured to perform the testing, the plurality of different network entities comprising a network slice instance, a network slice subnet instance, a network function, or a combination thereof. A method, comprising: receiving a plurality of decisions for testing within a network or network testing environment, each decision having a probability of being selected; selecting a subset of the plurality of decisions for testing, the selected subset of decisions being tested simultaneously within the network or network testing environment; for each decision in the subset, creating a network isolation within the network or network testing environment; and selecting a correct decision from the subset of the plurality of decisions according to the created network isolation.