WO2020069647A1

WO2020069647A1 - System for deploying incremental network updates

Info

Publication number: WO2020069647A1
Application number: PCT/CN2019/098368
Authority: WO
Inventors: Wei Xu; Nagaravind CHALLAKERE; Yan Sun
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2018-10-03
Filing date: 2019-07-30
Publication date: 2020-04-09
Also published as: CN112805984A; CN112805984B

Abstract

A method for deploying incremental network updates is provided. A program is received including proposed updates to the computer network, and the program is analyzed to produce a network object model (NOM). Configuration changes to the computer network are generated for the proposed updates using the NOM, and the configuration changes and initial configuration are parsed to produce a synthesized network configuration. The synthesized network configuration is converted into a first-order logical representation, and the logical representation is verified to determine whether the configuration changes are bound by invariants. The verification results are analyzed using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the synthesized network configuration. The computer network configuration is updated using the analysis results.

Description

SYSTEM FOR DEPLOYING INCREMENTAL NETWORK UPDATES

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to and benefit of U.S. Provisional Application No. 62/740,471, filed on October 3, 2018, entitled “System for Deploying Incremental Network Updates” , which application is hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to computer network management, and in particular to systems and methods for deploying incremental network updates.

BACKGROUND

Network configurations are frequently in need of updating, and the updating of the configurations can pose even more difficult problems than the initial build-out of the network. Incremental updates to production network configurations can be difficult to implement correctly. Configuring diverse network devices (e.g., routers, switches, and middle boxes) using traditional configuration directives incrementally is as challenging as programming an operational distributed system in assembly language. In cloud scale networks, the problem becomes intractable. Several critical capabilities missing in current practice of network management include dependency, sequencing, transaction atomicity, and verifiability.

Currently, network changes are mostly performed via archaic, low-level configuration directives. This gap in higher-level network representation and deployment is becoming the bottleneck in scaling versatile networks. Network design is not a one-time activity, as network configuration continuously evolves. Network changes are currently performed via archaic low-level configuration directives, which often result in errors because of dependencies of related objects on disparate nodes or layers in the network.

SUMMARY

Methods, apparatus, and systems are provided for deploying incremental network updates. Various examples are now described to introduce a selection of concepts in a simplified form that are further described below in the detailed description. The Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the present subject matter provide for incremental updates to production network configurations. Incremental verification is used to make sure both original and new invariants are met, in various embodiments. The present subject matter uses incremental verification with a hierarchical object model and semantics of change from the object model to determine fault provenance in incremental updates. The present subject matter computes the ordering of atomic transaction, translates sub-actions into low-level configuration directives, and provisions the nodes and their component objects in the computed order.

According to a first aspect, the present subject matter relates to a method for updating a computer network configuration using one or more processors. The method includes receiving a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration. The program is analyzed to produce a network object model (NOM) . Configuration changes to the computer network are generated for the proposed updates using the NOM. A representation of the initial configuration is verified to determine whether the configuration changes are bound by invariants. Results of the verification are analyzed using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration. The computer network configuration is updated using results of the analysis.

In a first implementation form of the method according to the first aspect as such, updating the computer network includes parsing the configuration changes and the initial configuration of the computer network to produce a synthesized network configuration. The synthesized network configuration is converted into a first-order logical representation. The first- order logical representation is verified to determine whether the configuration changes are bound by invariants.

In a second implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, updating the computer network includes adding a node to the computer network or removing a node from the computer network.

In a third implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, updating the computer network configuration includes modifying the initial computer network configuration.

In a fourth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, modifying the initial configuration includes adding a new prefix.

In a fifth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, modifying the initial configuration includes upgrading an interface.

In a sixth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, verifying the logical representation to determine whether the configuration changes are bound by invariants includes using a satisfiability modulo theories (SAT/SMT) solver.

In a seventh implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, analyzing the program to produce a NOM includes determining atoms of the program.

In an eighth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, determining atoms of the program includes determining a dependency of the atoms.

In a ninth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, determining atoms of the program includes determining a sequence of operations of the atoms.

In a tenth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, determining atoms of the program includes determining physical interfaces used by the program.

In an eleventh implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, determining atoms of the program includes determining trunk interfaces used by the program.

In a twelfth implementation form of the method according to the first aspect as such or any preceding implementation of the first aspect, determining atoms of the program includes determining routing information used by the program.

According to a second aspect, the present subject matter relates to a system for updating a computer network configuration, the system comprising one or more processors connected to the computer network, the one or more processors configured to receive a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration. The program is analyzed to produce a network object model (NOM) . Configuration changes to the computer network are generated for the proposed updates using the NOM. A representation of the initial configuration is verified to determine whether the configuration changes are bound by invariants. Results of the verification are analyzed using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration. The computer network configuration is updated using results of the analysis. Thus, an efficient concept for providing incremental updates to production network configurations can be realized.

In a first implementation form of the system according to the second aspect as such, the one or more processors are further configured to parse the configuration changes and the initial configuration of the computer network to produce a synthesized network configuration, convert the synthesized network configuration into a first-order logical representation, and verify the first-order logical representation to determine whether the configuration changes are bound by invariants.

In a second implementation form of the system according to the second aspect as such or any preceding implementation of the second aspect, the one or more processors are further configured to determine atoms of the program to produce the NOM, and perform a dependency analysis, including determining relationships among network objects within the atoms.

In a third implementation form of the system according to the second aspect as such or any preceding implementation of the second aspect, the one or more processors are further configured to compute an object change set to produce the NOM.

In a fourth implementation form of the system according to the second aspect as such or any preceding implementation of the second aspect, the one or more processors are further configured to construct an incremental graph using the computed object change set.

In a fifth implementation form of the system according to the second aspect as such or any preceding implementation of the second aspect, the one or more processors are further configured to apply a topological sort to the incremental graph to compute an order of changing objects.

According to a third aspect, the present subject matter relates to a computer-readable medium storing computer instructions to update a computer network configuration, that when executed by one or more processors, cause the one or more processors to perform steps of receiving a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration. The program is analyzed to produce a network object model (NOM) . Configuration changes to the computer network are generated for the proposed updates using the NOM. A representation of the initial configuration is verified to determine whether the configuration changes are bound by invariants. Results of the verification are analyzed using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration. The computer network configuration is updated using results of the analysis.

In a first implementation form of the computer-readable medium according to the third aspect as such, updating the computer network configuration includes at least one of the following: adding a node to the computer network, removing a node from the computer network, and modifying the computer network configuration.

In a second implementation form of the computer-readable medium according to the third aspect as such or any preceding implementation of the third aspect, modifying the computer network configuration includes adding a new prefix.

In a third implementation form of the computer-readable medium according to the third aspect as such or any preceding implementation of the third aspect, the computer network is a cloud computer network.

Any one of the foregoing examples may be combined with any one or more of the other foregoing examples to create a new embodiment within the scope of the present disclosure.

This Summary is an overview of some of the teachings of the present application and not intended to be an exclusive or exhaustive treatment of the present subject matter. Further details about the present subject matter are found in the detailed description and appended claims. The scope of the present inventive subject matter is defined by the appended claims and their legal equivalents.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating the addition of tenants to a cloud computer network, according to various embodiments.

FIG. 2 is a flow diagram for a method for deploying incremental network updates, according to various embodiments.

FIG. 3 is a schematic diagram illustrating actions performed for computer network management, according to various embodiments.

FIG. 4 is a block diagram for a system for modeling and deploying incremental network updates, according to various embodiments.

FIG. 5A is a flow diagram illustrating steps used for a method for determining configuration changes when deploying incremental network updates, according to various embodiments.

FIG. 5B is a flow diagram illustrating steps used for a method for updating network configurations when deploying incremental network updates, according to various embodiments.

FIGS. 6A-6C illustrate code listings and schematic diagrams for deploying incremental network updates, according to an example embodiment.

FIGS. 7A-7B illustrate code listings and schematic diagrams for configuration synthesis and verification, according to various embodiments.

FIG. 8 is a schematic diagram illustrating changes derived from a program to add tenants to a cloud computer network, according to various embodiments.

FIG. 9 illustrates a code listing for identifying objects causing violations when deploying incremental network updates, according to an embodiment of the disclosure.

FIG. 10 is a schematic diagram illustrating circuitry for performing methods according to example embodiments.

DETAILED DESCRIPTION

In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration, specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the inventive subject matter, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present inventive subject matter. The following description of example embodiments is, therefore, not to be taken in a limited sense, and the scope of the present inventive subject matter is defined by the appended claims.

As used herein, the term “object” (or “network object” ) indicates configuration data (e.g., supported bandwidth, routing process definitions, virtual local area network (VLAN) association, or other network attributes) for a network entity (e.g., a network interface or a network process) or an interaction between network entities. As used herein, the term “node” indicates a network device. A node can be associated with different network objects, characterizing how the node operates within a network. As used herein, the term “network object model” (or NOM) indicates a model that encompasses multiple relationships between different network objects associated with one or more network nodes.

As used herein, the term “tenant” is used interchangeable with the term “tenant subnet, ” and it indicates one or more network entities that can share a network resource (or resources) with other subnets, but network traffic to a subnet end point is isolated from entities in other subnets.

Current network updating techniques are template-based and do not use inter-object dependency in configuration directives. Removing objects in a network may leave garbage configurations behind, because related objects are not updated accordingly. Garbage configurations are defined as stale configuration elements that are no longer necessary. In addition, improperly removing a shared object (i.e., configuration data shared between several nodes) can shut down sections of the network. Incremental operations, such as circuit migration, adding or removing full mesh internal border gateway protocol (iBGP) routers, adding or removing an elastic IP/virtual private network (EIP/VPN) , often requires strict ordering of actions. If the correct order is not followed, disruption of production network will likely occur. Also, incremental updates often require atomic operation capability. Upon a failure, the committed actions need to be cleanly rolled back in the correct sequence. Verification should be built into the network management process to guarantee both existing and new incremental invariants are met. As used herein, the term “invariant” indicates a predefined property of one or more nodes that must not be changed by a modification to network configuration. If an invariant is violated by a network modification (i.e., the predefined property of the one or more nodes is changed) , the modification can be rolled back to prevent the invariant violation.

Existing configuration description frameworks, such as OpenConfig, provide device-level models, but when such models are computed it is hard to specify dependencies at different network layers. Using techniques disclosed herein, existing configuration description framework models are extended at node level and hierarchical YANG (yet another next generation) models are defined to express network wide dependencies. Proper sequence of operations ordering ensures a disruption-free incremental update. In current deployment frameworks it is not possible to describe the order of operations short of sequential program directives. In the present subject matter, the representation of configuration-as-program enables application of proven program analysis techniques such as static analysis, reference counting, and garbage collection to provably determine the correct sequence of operations.

Incremental verification is used to make sure both original and new invariants are met, in various embodiments. The present subject matter uses incremental verification with a hierarchical object model and semantics of change from the object model to determine fault provenance in incremental updates. Even though fault determination can be challenging in a large scale network, a key observation that helps reduce the search-space is that, for a given invariant to be violated between configuration changes, it is possible to identify the cause of the issue if the network model supports incremental representation.

The capability of clean rollback on failure is rarely supported in traditional network management. Production networks commonly have many garbage configurations that can impede network performance. In the present subject matter, the operator can cause rollback using a high-level language for most network objects. Techniques disclosed herein can be used to compute the ordering of atomic transaction, translate sub-actions into low-level configuration directives, and provision the nodes and their component objects in the computed order.

FIG. 1 is a schematic diagram illustrating the addition of tenants to a cloud computer network, according to various embodiments. Referring to FIG. 1, there is illustrated a management interface 152, a route distribution function 154, layer 3 interfaces 156 and 166, layer 2

interfaces

158 and 168, and

physical interfaces

160, 162, 170, and 172.

The management interface 152, such as a user interface for an administrator, is used to control route distribution via the route distribution function 154 between interfaces (e.g., sharing of route information among interfaces) . In some aspects, the route distribution function 154 includes one or more routing tables that can be used to enable nodes to reach one or more subnets accessible via the route distribution function 154. In some aspects, the management interface 152 can control the route distribution function 154 by managing the routing tables associated with it.

The layer 3 interfaces 156 and 166 include logical network interfaces that are addressable, such as VLAN interfaces or other logical type interfaces. The layer 2

interfaces

158 and 168 are interfaces that can be configured to aggregate traffic from the

physical interfaces

160, 162, 170, and 172. The

physical interfaces

160, 162, 170, and 172 can represent network device ports with differing bandwidth requirements. The dependency of arrows shown in FIG. 1 (such as A→ B) denotes that A is dependent on B, and that B should be instantiated before A.

In a datacenter network, the same physical infrastructure can support one or more customer networks or subnets, which are all isolated from each other. As mentioned above, each of the customer networks or subnets are referred to herein as tenants or tenant subnets. For example, a first tenant subnet can include layer 3 interface 156, layer 2 interface 158, and physical interface 160. A second tenant subnet can include layer 3 interface 156, layer 2 interface 158, and physical interface 162 which can have a different bandwidth requirement from physical interface 160.

Challenges may arise when adding tenants associated with route distribution function 154, such as adding a tenant subnet that includes layer 3 interface 166, layer 2 interface 168, and physical interface 170 or 172 (shown with dashed borders to indicate their addition) . More specifically, the incremental operation of adding the new tenant subnet can result in changes to existing network objects and the order of implementing the incremental operation can be of importance in order to avoid invariant violations.

FIG. 2 is a flow diagram for a method 200 for deploying incremental network updates, according to various embodiments. At operation 202, a program is received by one or more processors including proposed updates to a computer network configuration. The computer network configuration at the time of receiving the program can be considered as an initial configuration. At operation 204, the program is analyzed by the one or more processors to produce a network object model (NOM) . At operation 206, configuration changes to the computer network are generated by the one or more processors for the proposed updates using the NOM.

Optionally, at operation 208, the configuration changes and the initial configuration are parsed by the one or more processors to produce a synthesized network configuration. In some aspects, the synthesized network configuration includes configuration routines for one or more network nodes as listed in the initial configuration, supplemented by one or more additional configuration routines associated with the configuration changes. The synthesized network configuration can be further converted by the one or more processors into a first-order logical representation. More specifically, individual configuration routines from the synthesized network configuration can be converted to corresponding first-order logic representations containing predicates, quantifiers, and variables.

At operation 210, a representation of the configuration (e.g., a first-order logical representation as generated at operation 208) is verified by the one or more processors to determine whether the configuration changes are bound by invariants. For example, to perform the verification, the one or more processors can parse the representation of the configuration and determine whether one or more predefined properties of network nodes are being affected by the configuration changes, where the predefined properties define the invariants. In some aspects, the invariants and the associated predefined properties that may not be changed by network configuration modifications can be obtained via the initial network configuration.

At operation 212, the verification results are analyzed by the one or more processors using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the configuration. At operation 214, the computer network configuration is updated by the one or more processors using the analysis results. Optionally, at operation 216, fault provenance can be performed to indicate the detected violations caused by the configuration changes.

The present subject matter improves upon existing approaches, where a model is generated and constraint violation with the model is detected. There are two distinct advantages with the present subject matter when compared to the previous approach: 1) incremental updates are the target problem and the present subject matter provides an operator with the power to isolate the impact of network changes clearly to changed parts of the configuration. In existing approaches to performing incremental network upgrades, the network is the model of how packets flow. There is no first-principle-based definition for existing configuration and proposed changes; 2) inter-object dependency: the present network model not only analyzes the impact of change on the “operators goals, ” but also analyzes the impact of configuration changes on existing configuration elements, for a pro-active validation of the network. For example, as a result of a proposed change, a network node can receive traffic intended for a new subset and forwarding already existing on the node has not been updated corresponding to the impact of the change, such as a second-order change. Because the present subject matter uses inter-object dependency from the network object model, these scenarios can be analyzed.

In various embodiments, the program received at operation 202 is a program such as program 602 of FIG. 6A, including a function call to a add_tenant () function. In various embodiments, an example program call received at operation 202 includes: Add_tenant ( “Core Router 1” , 10, 10.10.10.1, 10.10.10.64/28, yes, ospfv3) . The program is converted to a NOM using several calls, such as create_management_interface, with the objects created and their relationship being the NOM, in various embodiments. FIGS. 6B-6C provide details of how the NOM is converted to a list of configuration changes.

The present subject matter provides a framework in which an incremental configuration change to a network is isomorphic with a program that can be verified based on, e.g., initial network configurations indicating invariants (i.e., predefined properties of one or more nodes that may not be changed by a modification to network configuration) . Generally, proposed configuration changes are handled in terms of lines of configuration changes. In the present subject matter, a program is used to describe the proposed changes, which program can be verified based on predefined invariants that can be communicated with an initial network configuration. The evolution of the network can be modeled as the cumulative effect of applying a sequence of atomic changes. Network invariants derived from requirements can be used to guide the definition of these atomic changes. Compiler techniques, program analysis, and formal verification methodologies are used to roll out incremental updates in a large-scale network. Semantics of the proposed change are defined to verify that the syntactic changes to configuration satisfy the underlying invariants.

In very large scale networks, configuration error is common and human management is not feasible. Invariant language, or a high-level language is used to define invariants, is used to specify network behavior at a higher layer, covers end-to-end use cases at different levels of abstraction, and captures device-level configurations to be able to express network-level objects such as open shortest path first (OSPF) areas and VPN instances. In various embodiments, the present system, in response to receiving requested changes to a network configuration, produces possible intermediate representations such that if a network invariant is violated, the possible intermediate representations are identified that effected the change in network behavior.

FIG. 3 is a schematic diagram illustrating actions performed for computer network management, according to various embodiments. Referring to FIG. 3, in a first level of the diagram, proposed updates 302 are defined. The proposed updates 302 can be part of a received program (e.g., as received at operation 202 in FIG. 2) , such as programs 312, 314, 316, and 318, which can update an existing network configuration policy 310. The proposed updates 302 can be analyzed and inferred knowledge 304 can be generated, which can include network object models 324. In some aspects, logic models 322 can be generated based on the network object models 324, where the logic models 322 can include first-order logical representations as described at operation 208 in FIG. 2. The inferred knowledge 304 can be used to perform a network update 306 of network 326 based on one or more configuration changes 328. Example configuration changes 330 are associated with service rules and forwarding rules and are generated based on the inferred knowledge 304.

The proposed changes 302 can be based on programs that include adding or deleting tenants (EIP or VPN) , and the proposed changes can be made to prevent violation of invariants (INV) , including invariants INV0 for the existing network configuration policy 310. As shown in FIG. 3, the evolving network is represented as a sequence of programs (e.g., 312-318) , where each program represents an incremental update at one specific time. For example, "Add EIP1" program 312 (which includes additional variant INV1) and "Add VPN1" program 314 (which includes additional variant INV2) are two incremental updates. After "Add VPN1" , the present subject matter ensures that the two new invariants (INV1, INV2) and existent invariants (INV0) are all met. Program 316 adds a new tenant (EIP2) without including any additional invariants (NULL) . Program 318 deletes a tenant (EIP1) , which results in removing invariants INV1 using automated reversion 320.

The present subject matter uses inferred knowledge 304 (including logic models 322 and object models 324) to model the changes and update the network 326 based on the models. The present subject matter also enables capabilities such as automated transaction, automated deletion and micro-template-based incremental enforcement. In prior approaches, network operator make changes to the network manually, verify the constraint satisfiability, and then update each device in the correct order (as network objects are dependent) . Because the present subject matter solves the dependency problem, it can be coupled with a deployment platform (such as ansible/Napalm) to automatically push proposed changes that satisfy the invariants, such as automated transaction and automated deletion.

In various embodiments, production network configurations are used to produce object models through correlation analysis, and a control plane simulation is used to produce logic models. Various embodiments assist human operators to formulate invariants of the existent production network, and the incremental update is programmed. The program is compiled into an incremental object model, and the incremental object model is translated into a sequence of configuration groups. New invariants (INV1, INV2) and all existent invariants (INV0) are verified, in various embodiments. The present subject matter provides for atomic operation, providing for clean rollback upon failure. In addition, incremental delete is provided with automated reversion to provide for correct order of deletion, which is often but not always in reverse of the addition of the tenant. Automated reversion is when the network is returned to the previous known running state. When a change is applied, if an invariant is violated, then the change will be reverted. Due to the dependencies between network configuration elements, it is not a straightforward operation where a few lines of configuration are removed from a node. The order of operations matter and ensure that the network does not reach or propagate an erroneous state. For example, FIG. 6C illustrates a code listing (e.g., steps 1-4) for methods for adding and removing tenants from existing computer networks, such as the network shown in FIG. 3, according to various embodiments. In various embodiments, similar steps work for addition and removing tenants: dependency evaluation, ordering, and using a program to specify the operation and network object model are the same, only the intent of the operation will be “delete” instead of addition.

FIG. 4 is a block diagram for a system 400 for modeling and deploying incremental network updates, according to various embodiments. Referring to FIG. 4, the system 400 for deploying incremental network updates can include a program analyzer module 404, a synthesis module 406, a parser module 410, a solver module 412, an analysis module 414, a configuration update module 418, and a fault provenance module 416.

A network operator writes the proposed network updates as a program 402, which is a representation of the proposed updates in an imperative or high-level computer language. The system 400 receives the program 402 as an input, and the program analyzer module 404 determines atoms, their dependency and sequence of operations, to produce a network object model (NOM) 420, which is a hierarchical representation of changes for the transaction. The synthesis module 406 is configured to generate configuration changes 422 based on the NOM 420. Further details regarding the operation of the program analyzer module 404 and the synthesis module 406 are illustrated in FIG. 5.

Computer network configurations 408 are communicated to the parser module 410. The network configurations 408 can be received at the time the program 402 is received, and the configurations 408 can be considered as initial network configurations. The parser module 410 parses the configuration changes 422 and the initial configurations 408 to generate a representation of the initial configuration 408 based on the configuration changes 422. In some aspects, the parser module 410 can generate the representation of the initial configuration by parsing the configuration changes 422 with the initial configuration 408 to generate a synthesized network configuration. The parser module 410 converts the synthesized network configuration into a first-order logical representation (FOLR) 424 of the network configurations, which is communicated to the solver module 412.

The solver module 412 receives as input the FOLR 424 as well as additional input 428, which can include a query input (QUERY) and other input (ENV) associated with network configurations. The solver module 412 can be a satisfiability or satisfiability-modulo-theories (SAT/SMT) solver 412, and can be configured to test if the FOLR 424 is bound by the invariants, which are expressed as constraints in the framework. The QUERY input refers to an invariant description by an operator, which description is in a query form. In some aspects, the solver module 412 can receive a natural language invariant description from the operator, which the solver module can convert in a query form. The ENV input can be obtained through an interactive process with an administrator and can include natural language input that can describe extraneous conditions not derived from configurations. In some aspects, the solver module 412 can determine counter examples of network configurations rejecting satisfiability of invariants defined by the query input 428 (output as results 426) , which is indicative of the network configurations that are bound by invariants.

The analysis module 414 is configured to receive the NOM 420 as well as the results 426 and determine how the configuration changes caused violations associated with the invariants detected by the solver 412. The configuration update module 418 can perform updates on the network configurations 408 based on the results generated by the analysis module 414 (e.g., based on the detected violations of invariants caused by the configuration changes) . The fault provenance module 416 can provide an indication of the detected violations caused by the configuration changes. A more detailed description of the operation of the solver 412 and the analysis module 414 is provided in FIG. 5B.

FIG. 5A is a flow diagram illustrating steps used for a method 500 for determining configuration changes when deploying incremental network updates, according to various embodiments. In some aspects, the method 500 can be performed by the program analyzer module 404 and the synthesis module 406 within the system 400 of FIG. 4. The system receives a program 502 (which can be the same as program 402 in FIG. 4) from a network operator which can contain multiple sets of operations. The program analyzer 404 identifies portions of the program with related sets of configuration changes (or atoms 504) associated with one or more interfaces or other network nodes. As used herein, the term “atom” indicates a cohesive unit of configuration changes that are associated with inter-related interfaces or other nodes within a network. The program analyzer 404 can generate the network object model 506 based on the atoms 504. For example, the program analyzer 404 can generate the network object model 506 by aggregating network objects associated with the atoms 504.

In some aspects, the use of atoms 504 provides for transaction atomicity. For example, when adding a tenant to a network, physical interfaces, trunk interfaces, and routing information are created on a top-of-rack (TOR) switch. The corresponding addresses are updated in access control list (ACL) rules on an edge router or firewall, and all of these changes together can form an atom. In various embodiments, a dependency analysis is used to determine relationships among network objects within an atom to determine the network object model 506. An object change set is computed and an incremental graph is constructed for the atom using the dependency analysis. An incremental graph is a graph where the scope only includes objects that have changed with links to an existing configuration. Some objects may already exist when creating or adding tenants, and some objects might have dependencies and thus should not be de-referenced when performing a delete. A topological sort is applied to the incremental graph to deduce the ordering of changing objects into a sequence graph 508. In various embodiments, a proposed micro-template is instantiated with values of the object change set, and the micro-template is translated into vendor-specific configurations and provisioned to nodes using low-level directives 510 in an order stipulated by the sequence graph 508. In some aspects, the low-level directives 510 can be used (e.g., a connection with a lookup table) to determine the configuration changes 422. An example representation of a program, atoms, sequence graph, and low-level directives is illustrated in FIGS. 6A-6C.

FIG. 5B is a flow diagram illustrating steps used for a method 550 for updating network configurations when deploying incremental network updates, according to various embodiments. Referring to FIG. 5B, the method 550 can include

example operations

552, 554, 556, 558, 560, and 562 which can be performed by the solver module 412, the analysis module 414, the configuration update module 418, and the fault provenance module 416 of FIG. 4.

According to various embodiments, the present subject matter provides for incremental verification by detecting invariant violation. The object model is encoded as an Extended Finite State Machine (EFSM) , and network invariants are translated to derive queries and to identify fault provenance when an invariant is not satisfied using increment semantics from the network model. In various embodiments, identifying fault provenance is the inverse of network synthesis. When synthesizing network configurations, the broader network-wide configurations are defined before device-specific configurations can be generated. In identifying the possible configuration changes that violate a given invariant, the present subject matter first identifies device-level changes and then evaluates their network-wide effects. To reduce the program run-time, the present subject matter makes the following optimizations: reducing the number of candidate nodes and reducing the set of network objects to examine.

At operation 552, a query input is received (e.g., the input 428) that includes a translation of an operator’s invariant description. At operation 554, the solver module 412 can determine counter-examples of network configurations rejecting query satisfiability using the FOLR 424. The determined counter-examples indicate network configurations that are bound by invariants and can be output as results 426 to the analysis module 414.

At operation 556, network processing flows can be determined from the counter-examples, where the flows correspond to marker states in a representation (e.g., FOLR 424) of the initial network configuration 408. More specifically, the analysis module 414 can analyze counter-examples using marker-states from the FOLR 424. For example, node ingress, egress, route match, ACL/Route-filter decision and node accept/drop decisions in the FOLR 424 are relevant markers. From the counter-example, the analysis module 414 determines the network processing flows corresponding to these marker states. In this regard, a flow identifies a transition from one network state relevant to the query to the next network state relevant to the query. At operation 558, candidate nodes are selected from the processing flows. For example, a list of candidate nodes to examine is selected from the processing flows (e.g., Source, Destination, Transit, and No-Transit nodes) . To reduce the set of network objects to examine, an algorithm such as the algorithm shown in FIG. 9 is used to identifying changed objects that cause violations, and candidate nodes associated with such objects can be selected. At operation 560, for each of the selected candidate nodes, the analysis module 414 can determine node configuration impact on the invariants associated with the input query. At operation 562, based on the determined note configuration impact, the configuration update module 418 can update the network configuration 408, e.g., to implement the network update specified by the program 402 without affecting network invariants. At operation 562, the fault provenance module 416 can provide an indication of the nodes and corresponding note configuration impact (e.g., so that a network operator can manually perform incremental network updates using node configurations that do not impact the invariants) .

FIGS. 6A-6C illustrate code listings and schematic diagrams for deploying incremental network updates, according to an example embodiment. Referring to FIG. 6A, there is illustrated a code listing of a program 602 which can include incremental network updates. The program 602 can relate to adding a tenant subnet within a network. In some aspects, the program 602 can be similar to

programs

402 and 502 discussed herein above.

Referring to FIG. 6B, there is illustrated identification of atoms after program 602 is received (e.g., as also described at operation 202 in FIG. 2) . More specifically,

atoms

604, 606 are identified to split the program into sets of operations and configuration changes that are related. The first atom 604 includes operations to add a management interface 612, a layer 3 interface 616, an aggregate interface 617, and physical interfaces 618, 619. The second atom 606 includes operations to add a layer 3 interface 626, an aggregate interface 627 and

physical interfaces

628, 629. The

atoms

604, 606 access data from route distribution 614, and network objects associated with nodes and interfaces within the atoms can be aggregated to obtain a NOM. The NOM can identify objects that are changed due to the proposed updates within the program, and can be used to derive a sequence graph 608.

A sequence graph is a graphical representation of the order of configuration operations for an atom, and can be derived using a topological sort, to obtain a directed graph in which there is a connection from A to B that indicates that A predates B. As illustrated in FIG. 6B, elements 1-7 in sequence graph 608 can be used in the indicated order to add a tenant to the network. Element 7 refers to a sub-interface, which includes aggregated interfaces, and a loop, which is the same as a loopback. The sequence graph can be used to derive low-level directives (e.g., 610) , and the low-level directives can be used to obtain the configuration changes (e.g., as also described at operation 206 in FIG. 2) .

FIG. 6C illustrates example low-level directives 610, which can be used for determining configuration changes (e.g., configuration changes 422 in FIG. 4) . In some aspects, the low-level directives 610 can be mapped to corresponding configuration changes associated with each directive in a lookup table. In this regard, the synthesis module 406 can use such a lookup table to determine the configuration changes 422 based on the low-level directives, such as low-level directives 610.

In some aspects, generating a sequence graph from a NOM and using the sequence graph as intermediate representation enables correct-by-construction, control plane simulation, formal verification, and network synthesis. The model includes the capability to encode objects and their dependencies, as well as sequences of operations. In the present subject matter, these capabilities are used in both a top-down analysis (i.e., synthesis) of configurations and bottom-up analysis (i.e., verification) of an existing configuration in a datacenter.

In some aspects, the verification of these capabilities for an existing configuration is performed during the bottom-up analysis. The present subject matter implements hierarchical YANG models for network-wide topology and extended OpenConfig models for node-level configurations, in various embodiments. Both models include semantics to represent changed elements. Supported object types cover datacenter networking, and include physical interfaces such as Ethernet (ETH) , and virtual interfaces such as Ethernet trunk (ETH-TRUNK) , loopback, virtual local area network interface (VLANIF) , OSPF, border gateway protocol (BGP) , static routing, virtual routing and forwarding (VRF) , routing policy, firewall policy, virtual local area network (VLAN) , virtual extendible local area network (VxLAN) , and topology (L1, L2, L3) . In various embodiments, the model is extendable and the models’ language binding (such as Python, in an embodiment) provides model-based programmability.

FIGS. 7A-7B illustrate code listings and schematic diagrams for configuration synthesis (top-down analysis) and verification (bottom-up analysis) , according to various embodiments. Referring to FIG. 7A and FIG. 7B, the top-down analysis can be performed starting with the sequence graph 722, which is used to generate the low-

level directives

711, 712, 713, 714, 715, and 716 in FIG. 7B. The low-level directives in FIG. 7B can be used to generate the configuration changes 718. Based on the configuration changes, configuration updates can be performed so that a final configuration 720 can be generated.

The sequence graph 722 can be based on a task for adding a node to a network using sub-interface 707, in an embodiment. For interfaces ETH 701, ETH-TRUNK 702, VLANIF 703, LOOP 704, OSPF 705 and VPN-INSTANCE 706, corresponding low-level directives (711 for ETH, 712 for ETH-TRUNK, 713 for VLANIF, 714 for LOOP, 715 for OSPF, and 716 for VPN-INSTANCE) are provided for the sequence graph 722 in FIG. 7B.

In some aspects, the bottom-up analysis can be used for verification of the updated configurations 720. More specifically, the bottom-up analysis can start with the updated configurations 720, which can be used to generate the configuration changes 718. The configuration changes 718 can be used for generating a sequence graph 722 associated with one or more tasks that have been added to the configuration 720 during the top-down analysis. The sequence graph 722 can then be assessed on whether any unintended tasks are being performed or tasks that violate existing invariants.

FIG. 8 is a schematic diagram illustrating changes derived from a program to add tenants to a cloud computer network, according to various embodiments. In the depicted embodiment, an incremental task is demonstrated to add new nodes to a cloud computer network. At 802, an operation is conducted to add NODE1, where adding NODE1 includes adding tenant OSPF 814, LOOP1 804, objects including virtual interfaces VLANIF1 806 and ETH-TRUNK1 808, and physical interfaces ETH1 810 and ETH2 812. At 822, an operation is conducted to add NODE2, including tenant OSPF 814, LOOP1 804, objects including virtual interfaces VLANIF2 826, ETH-TRUNK2 828, and physical interfaces ETH3 830 and ETH4 832. While the system performs the incremental tasks, the sub-task ordering and dependency on shared sub-tasks are determined from the model. The program dependency analysis technique is used to analyze the impacting objects of the program, in various embodiments. Impacting objects refer to objects to be modified to ensure that new objects are valid, such as when adding an OSPF process a new loopback interface should be created. An object change set is computed using dependency analysis and an incremental graph is constructed from the change set. In various embodiments, a topological sort is then applied to the incremental graph to deduce the ordering of changing objects into the sequence graph. In various embodiments, a template concept is used at a smaller granularity, referred to herein as a micro-template. The micro-template is instantiated with values of the object change set, and the micro-template is translated using a vendor-specific translation program into vendor-specific configurations and provisioned to nodes in the order stipulated by the sequence graph.

In various embodiments, the present subject matter can be used to safely remove a tenant from a computer network. While removing a tenant can be a reverse of adding a tenant, problems can be encountered in determining where to start the removal process and a correct order of operations. Another problem with removal of a tenant is object sharing. Shared objects should not be removed if they are still referred to by other objects. In various embodiments, techniques such as a garbage collector and a smart-pointer are used to achieve incremental delete in a fully automated manner, without creating garbage configurations. Garbage configurations can be detected and eliminated without violating an invariant, in various embodiments. The present approach uses the ordering information of the object model. As stated above, performing an incremental add of a tenant follows a particular sequence to create network objects and to build the object model. The process can be reversed to perform an automatic incremental delete. In one example, sink objects are tracked for each incremental add operation. Sink objects are network elements in the sequence graph which only have one parent node, which represents a network object that was previously present. An incremental delete utilizes these sink objects and removes child objects of the sink objects following the sequence graph, removing the objects created during the incremental addition. To manage shared objects, a reference count (refcnt) variable is used for each object that tracks the number of live references to the object. If the parent object of each object in the reference graph is removed, the refcnt value is decreased by one. If the refcnt value is equal to zero, the object can be safely removed, in various embodiments.

For a program-directed incremental delete, such as deleting a tenant, the present subject matter identifies network dependencies using a dependency tree and deallocates resources safely, reducing the possibility of human error. For an automated incremental delete, unused network objects (having zero references) can be identified when parsing an existing configuration. The impact of automated incremental deletion is tested by generating queries, and if no invariant violation is detected, then the object can be safely deleted.

Various embodiments of the present subject matter provide for transaction atomicity, which allows for a clean rollback on failures for incremental add and delete operations. In various embodiments, transaction primitives are used to mark transaction scope for incremental add and delete operations. Various embodiments provide model-driven, bottom-up analysis of raw configurations. Bottom-up analysis includes extraction, correlation and sequencing, in various embodiments. Extraction includes turning monolithic configuration into discrete objects. Correlation refers to correlating objects to deduce a dependency graph. Sequencing includes a topological sort of the dependency graph to impose a sequence onto the model. Given physical topology and raw configurations, the present subject matter converts the production network into an accompanying object model. This object model is encoded into a logic model for SMT solver-based verification, in various embodiments.

FIG. 9 illustrates a code listing for identifying objects causing violations when deploying incremental network updates, according to an embodiment of the disclosure. More specifically, the algorithm 900 is used to identifying changed objects that cause violations, and candidate nodes associated with such objects can be selected (e.g., during the processing discussed in connection with FIG. 5B) . In the depicted embodiment, DP is an object model of the synthesized dataplane (or collection of forwarding behavior at each node) , which is also referred to as the collapsed state (where the network configuration collapses to a set of forwarding entries at each node) ; T is the topology; Q is the query representing an assertion from the invariant definition; and N is the network model. Candidate nodes are selected from the set of flows described above. For each candidate node, the present subject matter examines the impact of its configuration on the result of the query. In this manner, provisioning errors are detected where the resources are allocated but not used. In various embodiments, T is the current topology of the network, DP is the synthesized network configuration (e.g., at operation 208) , Q is an object with functions for testing invariants, and N supplements T in describing the network. C _routes is a map that stores deleted routes, in various embodiments. According to various embodiments, each Q is an invariant, such that the function Tracefailure is called once for each Q.

Thus, the present subject matter provides an efficient system to deploy updates to a network in real time without modifying the base configuration significantly. The system includes methods to efficiently model the intended change and synthesize network commands, and also includes an algorithm to locate fault in the case of errors. Benefits of the proposed subject matter further include reducing time delay for add/delete and corresponding debugging, which reduces monetary costs from lengthy network outages that can also damage brand and reputation.

FIG. 10 is a schematic diagram illustrating circuitry for performing methods according to example embodiments. All components need not be used in various embodiments. For example, the computing devices may each use a different set of components and storage devices.

One example computing device in the form of a computer 1000 may include a processing unit or processor 1002, memory 1003, cache 1007, removable storage 1011, and non-removable storage 1012, all coupled by a bus 1020. Although the example computing device is illustrated and described as a computer 1000, the computing device may be in different forms in different embodiments. For example, the computing device may be a router or other computing device including the same or similar elements as illustrated and described with regard to FIG. 10. Further, although the various data storage elements are illustrated as part of the computer 1000, the storage may also or alternatively include cloud-based storage accessible via a network, such as the Internet or server-based storage.

In FIG. 10, memory 1003 may include volatile memory 1014 and/or non-volatile memory 1008. Computer 1000 may include –or have access to a computing environment that includes –a variety of computer-readable media, such as volatile memory 1014 and/or non-volatile memory 1008, removable storage 1011 and/or non-removable storage 1012. Computer storage includes random access memory (RAM) , read only memory (ROM) , erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM) , flash memory or other memory technologies, compact disc read-only memory (CD ROM) , Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions. Storage can also include networked storage such as a storage area network (SAN) .

Computer 1000 may include or have access to a computing environment that includes an input interface 1006, an output interface 1004, and a communication interface 1016. In various embodiments, communication interface 1016 includes a transceiver and an antenna. Output interface 1004 may include a display device, such as a touchscreen, that also may serve as an input device. The input interface 1006 may include one or more of a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated within or coupled via wired or wireless data connections to the computer 1000, or other input devices. The computer 1000 may operate in a networked environment using a communication connection to connect to one or more remote computers, such as database servers. The remote computer may include a personal computer (PC) , server, router, network PC, a peer device or other common network node, or the like. The communication connection may include a Local Area Network (LAN) , a Wide Area Network (WAN) , cellular, WiFi,

or other networks.

Computer-readable instructions, i.e., a program 1018, comprise instructions stored on a computer-readable medium that are executable by the processor 1002 of the computer 1000. The terms “computer-readable medium” and “storage device” do not include carrier waves to the extent carrier waves are deemed too transitory. In one example, the processor 1002 executes the program 1018 to implement methods for deploying incremental network updates. In some aspects, the program 1018 can implement one or more modules configured to perform functionalities discussed herein. For example, the program 1018 can implement the program analyzer 404, the synthesis module 406, the analysis module 414, and the fault provenance module 416. Even though not illustrated in FIG. 10, the program 1018 can further implement other modules performing functionalities disclosed herein, such as the solver module 412 and the configuration update module 418.

In an example embodiment, the computer 1000 includes a reception module receiving a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration, a model module analyzing the program to produce a network object model (NOM) , a change module generating configuration changes to the computer network for the proposed updates using the NOM, a verification module verifying a representation of the initial configuration to determine whether the configuration changes are bound by invariants, a results module analyzing results of the verification using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration, and an update module updating the computer network configuration using results of the analysis. In some embodiments, the computer 1000 may include other or additional modules for performing any one of or combination of steps described in the embodiments. Further, any of the additional or alternative embodiments or aspects of the method, as shown in any of the figures or recited in any of the claims, are also contemplated to include similar modules.

Although a few embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Other embodiments may be within the scope of the following claims.

Although the present disclosure has been described with reference to specific features and embodiments thereof, it is evident that various modifications and combinations can be made thereto without departing from the scope of the disclosure. The specification and drawings are, accordingly, to be regarded simply as an illustration of the disclosure as defined by the appended claims, and are contemplated to cover any and all modifications, variations, combinations or equivalents that fall within the scope of the present disclosure.

Claims

A method for updating a computer network configuration using one or more processors, the method comprising:

receiving a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration;

analyzing the program to produce a network object model (NOM) ;

generating configuration changes to the computer network for the proposed updates using the NOM;

verifying a representation of the initial configuration to determine whether the configuration changes are bound by invariants;

analyzing results of the verification using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration; and

updating the computer network configuration using results of the analysis.
The method of claim 1, further comprising:

parsing the configuration changes and the initial configuration of the computer network to produce a synthesized network configuration;

converting the synthesized network configuration into a first-order logical representation; and

verifying the first-order logical representation to determine whether the configuration changes are bound by invariants.
The method of any of claims 1 through 2, wherein updating the computer network configuration includes adding a node to the computer network or removing a node from the computer network.
The method of any of claims 1 through 3, wherein updating the computer network configuration includes modifying the initial configuration.
The method of claim 4, wherein modifying the initial configuration includes adding a new prefix.
The method of claim 4, wherein modifying the initial configuration includes upgrading an interface.
The method of claim 2, wherein verifying the logical representation to determine whether the configuration changes are bound by invariants includes using a satisfiability modulo theories (SAT/SMT) solver.
The method of any of claims 1 through 7, wherein analyzing the program to produce the NOM includes determining atoms of the program.
The method of claim 8, wherein determining atoms of the program includes determining a dependency of the atoms.
The method of claim 8 or claim 9, wherein determining atoms of the program includes determining a sequence of operations of the atoms.
The method of any of claims 8 through 10, wherein determining atoms of the program includes determining physical interfaces used by the program.
The method of any of claims 8 through 11, wherein determining atoms of the program includes determining trunk interfaces used by the program.
The method of any of claims 8 through 12, wherein determining atoms of the program includes determining routing information used by the program.
A system for updating a computer network configuration, the system comprising:

one or more processors connected to the computer network, the one or more processors configured to:

receive a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration;

analyze the program to produce a network object model (NOM) ;

generate configuration changes to the computer network for the proposed updates using the NOM;

verify a representation of the initial configuration to determine whether the configuration changes are bound by invariants;

analyze results of the verification using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration; and

update the computer network configuration using results of the analysis.
The system of claim 14, wherein the one or more processors are further configured to:

parse the configuration changes and the initial configuration of the computer network to produce a synthesized network configuration;

convert the synthesized network configuration into a first-order logical representation; and

verify the first-order logical representation to determine whether the configuration changes are bound by invariants.
The system of claim 14, wherein the one or more processors are further configured to:

determine atoms of the program to produce the NOM; and

perform a dependency analysis, including determining relationships among network objects within the atoms.
The system of any of claims 14 through 16, wherein the one or more processors are further configured to compute an object change set to produce the NOM.
The system of claim 17, wherein the one or more processors are further configured to:

construct an incremental graph using the computed object change set; and

apply a topological sort to the incremental graph to compute an order of changing objects.
A computer-readable medium storing computer instructions to update a computer network configuration, that when executed by one or more processors, cause the one or more processors to perform steps of:

receiving a program including proposed updates to the computer network configuration, the computer network configuration at the time of receiving the program being an initial configuration;

analyzing the program to produce a network object model (NOM) ;

generating configuration changes to the computer network for the proposed updates using the NOM;

verifying a representation of the initial configuration to determine whether the configuration changes are bound by invariants;

analyzing results of the verification using the NOM to determine how the configuration changes caused violations associated with the invariants and detected when testing the representation of the initial configuration; and

updating the computer network configuration using results of the analysis.
The computer-readable medium of claim 19, wherein updating the computer network configuration includes at least one of: adding a node to the computer network, removing a node from the computer network, or modifying the computer network configuration;

wherein modifying the computer network configuration includes adding a new prefix or upgrading an interface.