WO2023218227A1

WO2023218227A1 - Methods and apparatus of performing isolated testing on a virtualised cloud layer

Info

Publication number: WO2023218227A1
Application number: PCT/IB2022/054432
Authority: WO
Inventors: Oussama JEBBAR; Maria Toeroe
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2022-05-12
Filing date: 2022-05-12
Publication date: 2023-11-16

Abstract

A method of performing isolated testing on a virtualised cloud layer in a production environment, the virtualised cloud layer comprising virtual components configured to be executed to provide corresponding production services in the production environment, the method comprising: marking each virtual component with a production key to identify the virtual components as production virtual components executed in the production environment; marking each production service with a production key to identify the production services as services provided in the production environment; receiving a test configuration defining a virtual component under test; creating a test virtual component based on the virtual component under test defined in the test configuration; creating a test service based on one of the production services; selecting a test case comprising the test service and the test virtual component; and executing the test case.

Description

METHODS AND APPARATUS OF PERFORMING ISOLATED TESTING ON A VIRTUALISED CLOUD LAYER

Technical Field

Embodiments of the present disclosure relate to methods and apparatus of performing isolated testing on a virtualised cloud layer, and in particular methods and apparatus for executing a test case in isolation from production traffic in a production environment.

Background

Testing a service in its production environment while mitigating disruption may be referred to as live testing. Test interferences are one of the main challenges of live testing. One countermeasure taken to overcome this challenge is called test isolation.

Kubernetes is an open-source containers orchestrator used to deploy containerised applications to provide services. Kubernetes’ orchestration is guided by a configuration specified using a set of concepts.

The first concept is the “pod”. A pod is a logical entity that encapsulates a set of containers that together may provide a set of functionalities. As a result, a pod is specified by the container images used for its containers, the ports it exposes, as well as a set of user defined labels that are used for management purposes such as for the placement of pods. A label may be a set of key value pairs that is used to identify/organise objects or sets of objects. Users/clients may then use the labels (e.g. by specifying selectors) to select and group subsets of objects for various purposes.

To ensure proper handling of production load and fault tolerance, pods of identical specifications may be grouped into sets. These sets may be referred to as ReplicaSets or Deployments. A ReplicaSet/Deployment may specify the required number of pods that should be available at all times. Moreover, each ReplicaSet/Deployment specifies a selector used to match the labels of pods to be managed by a given ReplicaSet/Deployment.

To expose the functionalities provided by pods, the concept of “service” is used. A service specifies the internet protocol (IP) address and port (or domain name) through which the service is accessible. Services also specify selectors that are used to match the labels of pods that may provide the service (i.e. endpoints). Services provided by pods may functionally depend on one another and compose a functionality of the system. Services may invoke one another either through the IP address of the service or using its domain name.

Several solutions exist for specifying and setting of functional dependencies between services. For services that use HTTP or HTTPS, Ingress is a tool often used to setup traffic routing through such services. Services may also invoke one another without the need for third party tools. Such invocations may be configured by setting some environment variables at the dependent container level to the sponsor service, or by setting parameters in specific configuration files inside the containers.

Techniques for traffic distribution are commonly used to perform testing in production. Canary testing is one of the most commonly used techniques for testing in production. Canary testing often takes place at the same time as (as part of) an upgrade. As a result, pods operating a new version of a configuration are considered as pods under test. During canary testing, pods under test start receiving a small portion of the production traffic. As long as no errors manifest, this portion of production traffic is increased until the pods of the old version stop handling any production traffic. To distribute the traffic between old pods and pods under test (i.e. new pods), several approaches may be used, such as:

• Manipulating the ReplicaSet or Deployment through pod labels such that the portion of pods running the new version of the configuration matches the portion of production traffic that should be handled by the version being canary tested. The bigger this portion gets the more scaling actions are taken to replace pods of old version by pods of the new version. This process continues until the upgrade is completed or an error manifest.

• Using tools such as Istio, one may specify the portion of the production traffic that should go to the new version. Then this specified portion continues to increase until an error manifests or no more old pods exist.

Typical test approaches have some limited capability for setting the placement of pods, such as the pods under test, through node selectors. However, they do not allow setting a route of the test traffic when the functionality being tested is composed of multiple services as they rely on a round robin load balancing logic provided by Kubernetes. Therefore, typical test approaches are not adequate to implement the test isolation necessary for typical test methods.

Figure 1 a typical test configuration being deployed using a typical test approach. The test configuration comprises serving pods (illustrated by cog symbols) and pods under test (illustrated by cog symbols having a border). The serving pods and pods under test are deployed on nodes (Nodel , Node2, Node3, Node4, Node5, Node6, Node? and Node8) and are associated with certain ReplicaSets (RS1 , RS2, RS3, RS4, RS5 and RS6). Functionality required by production traffic is provided as three services (Srvd , Srvc2 and Srvc3) illustrated with block arrows. Dependencies between services are illustrated by line arrows. These dependencies result in invocations, represented by the dashed arrows, between the ReplicaSets providing those services.

New versions of the services are deployed in the test configuration as pods under test. In order to test the new versions of the services in production, 50% of the production traffic is directed to the pods under test and the remaining 50% of production traffic is directed to the serving pods. Such a test configuration may be deployed using Kubernetes provided concepts only and/or using Istio (or a similar tool).

As the typical test configuration illustrated in Figure 1 relies on production traffic to test new versions of a feature, it not only limits the types of features that may be tested in production but also restricts the deployability of a test configuration. Therefore, problems of such a test configuration include the following:

• No test isolation exists as production traffic is routed to the serving pods as well as the pods under test. Typical approaches may reduce the number of users impacted by test interferences, but it does not help alleviate the risk of interference.

• Test coverage is limited to scenarios that happen to occur with certain production traffic. Potentially disastrous but seldom occurring scenarios might not be tested at all.

• Tests may not be executed properly. For example, in the test configuration of Figure 1 , if a test case consisted of checking whether certain functionality operates properly through pods under test in Nodel , Node5, and Node6, routing of production traffic through these pods under test may not be guaranteed as Kubernetes assumes that the pods under test play the same role as the serving pods and serve the same service.

Summary

It is an object of the present disclosure to provide methods of performing isolated testing on a virtualised cloud layer for executing a test case in isolation from production traffic in a production environment.

Aspects of embodiments provide a network node, methods and computer programs which at least partially address one or more of the challenges discussed above.

An aspect of the disclosure provides a method of performing isolated testing on a virtualised cloud layer in a production environment. The virtualised cloud layer comprises virtual components configured to be executed to provide corresponding production services in the production environment. The method comprises marking each virtual component with a production key to identify the virtual components as production virtual components executed in the production environment. The method further comprises marking each production service with a production key to identify the production services as services provided in the production environment. The method further comprises receiving a test configuration defining a virtual component under test. The method further comprises creating a test virtual component based on the virtual component under test defined in the test configuration. The method further comprises creating a test service based on one of the production services. The method further comprises selecting a test case comprising the test service and the test virtual component. The method further comprises executing the test case.

Advantageously, the use of separate test virtual components and test services as part of a deployment of a test configuration mitigates the risk of test interference in a deployment environment. Optionally, a test case may be executed in isolation from the virtual components configured to be executed to provide corresponding production services in the production environment and the production services.

Optionally, executing the virtual components in the production environment comprises routing production traffic to the production virtual components only.

Optionally, the method may further comprise generating test traffic, and executing one or more test cases may comprise routing the test traffic only to the test virtual component.

Another aspect of the disclosure provides a network node configured to performing isolated testing on a virtualised cloud layer in a production environment. The virtualised cloud layer comprising virtual components configured to be executed to provide corresponding production services in the production environment. The network node comprising processing circuitry and a memory containing instructions executable by the processing circuitry. The network node is operable to mark each virtual component with a production key to identify the production virtual components as virtual components executed in the production environment. The network node is further operable to mark each production service with a production key to identify the production services as services provided in the production environment. The network node is further operable to receive a test configuration defining a virtual component under test. The network node is further operable to create a test virtual component based on the virtual component under test defined in the test configuration. The network node is further operable to create a test service based on one of the production services. The network node is further operable to select a test case comprising the test service and the test virtual component. The network node is further operable to execute the test case.

Another aspect of the disclosure provides a manager of a network node configured to performing isolated testing on a virtualised cloud layer in a production environment. The virtualised cloud layer comprising virtual components configured to be executed to provide corresponding production services in the production environment. The manager of a network node comprising processing circuitry and a memory containing instructions executable by the processing circuitry. The manager of a network node is operable to mark each virtual component with a production key to identify the production virtual components as virtual components executed in the production environment. The manager of a network node is further operable to mark each production service with a production key to identify the production services as services provided in the production environment. The manager of a network node is further operable to receive a test configuration defining a virtual component under test. The manager of a network node is further operable to create a test virtual component based on the virtual component under test defined in the test configuration. The manager of a network node is further operable to create a test service based on one of the production services. The manager of a network node is further operable to select a test case comprising the test service and the test virtual component. The manager of a network node is further operable to execute the test case.

Another aspect of the disclosure provides a computer-readable medium comprising instructions for performing the methods set out above, which may provide equivalent benefits to those set out above.

Further aspects provide apparatuses and computer-readable media comprising instructions for performing the methods set out above, which may provide equivalent benefits to those set out above.

Brief Description of Drawings

For a better understanding of the present disclosure, and to show how it may be put into effect, reference will now be made, by way of example only, to the accompanying drawings, in which:

Figure 1 is a schematic diagram illustrating a typical test configuration deployed on a virtualised cloud layer;

Figure 2 is a flowchart illustrating a method of performing isolated testing on a virtualised cloud layer;

Figure 3A is a schematic diagram of a network node for performing isolated testing on a virtualised cloud layer;

Figure 3B is another schematic diagram of a network node for performing isolated testing on a virtualised cloud layer;

Figure 4 is a schematic diagram illustrating an embodiment of the method of performing isolated testing on a virtualised cloud layer;

Figure 5 is a flowchart illustrating an embodiment of the method of performing isolated testing on a virtualised cloud layer; and

Figure 6 is another schematic diagram illustrating an embodiment of the method of performing isolated testing on a virtualised cloud layer.

Detailed Description

The following sets forth specific details, such as particular embodiments for purposes of explanation and not limitation. It will be appreciated by one skilled in the art that other embodiments may be employed apart from these specific details. In some instances, detailed descriptions of well-known methods, nodes, interfaces, circuits, and devices are omitted so as not obscure the description with unnecessary detail. Those skilled in the art will appreciate that the functions described may be implemented in one or more nodes using hardware circuitry (e.g., analog and/or discrete logic gates interconnected to perform a specialized function, ASICs, PLAs, etc.) and/or using software programs and data in conjunction with one or more digital microprocessors or general purpose computers that are specially adapted to carry out the processing disclosed herein, based on the execution of such programs. Nodes that communicate using the air interface also have suitable radio communications circuitry. Moreover, the technology can additionally be considered to be embodied entirely within any form of computer-readable memory, such as solid- state memory, magnetic disk, or optical disk containing an appropriate set of computer instructions that would cause a processor to carry out the techniques described herein.

Hardware implementation may include or encompass, without limitation, digital signal processor (DSP) hardware, a reduced instruction set processor, hardware (e.g., digital or analog) circuitry including but not limited to application specific integrated circuit(s) (ASIC) and/or field programmable gate array(s) (FPGA(s)), and (where appropriate) state machines capable of performing such functions.

In terms of computer implementation, a computer is generally understood to comprise one or more processors, one or more processing modules or one or more controllers, and the terms computer, processor, processing module and controller may be employed interchangeably. When provided by a computer, processor, or controller, the functions may be provided by a single dedicated computer or processor or controller, by a single shared computer or processor or controller, or by a plurality of individual computers or processors or controllers, some of which may be shared or distributed. Moreover, the term “processor” or “controller” also refers to other hardware capable of performing such functions and/or executing software, such as the example hardware recited above. Embodiments of the present disclosure provide methods of performing isolated testing on a virtualised cloud layer for executing a test case in isolation from production traffic in a production environment. A test case comprises a test virtual component and a test service. The test case may be configured to be executed in isolation from production virtual components that provide corresponding production services in a production environment.

When referring to a specific type of virtualised cloud layer in the following description, reference will be made to a Kubernetes virtualised cloud layer. It will be understood that embodiments of the present disclosure may be applied to any virtualisation layer. Examples of other types of virtualisation layers to which embodiments of the present disclosure may apply include a hypervisor-based virtualisation layer such as the OpenStack virtualisation layer, container-based virtualisation layer similar to Kubernetes, and a Java Virtual Machine (JVM) virtualisation layer.

Figure 2 is a flowchart illustrating a method of performing isolated testing on a virtualised cloud layer performed by a network node (e.g. a core network node, a radio access node, a base station node, a user equipment (UE) node, etc.). The isolated testing is performed in a production environment. A production environment may be a real-world environment in which the virtualised cloud layer is deployed (i.e. executed) for use by users.

In some embodiments, the method of performing isolated testing on a virtualised cloud layer may be performed by a manager of the network node in addition to or instead of the network node. The manager of the network node may be, for example, a mobility management entity (MME).

The virtualised cloud layer comprises plural virtual components each of which are configured to provide a corresponding production service in the production environment. The plural virtual component may be configured directly or indirectly.

The virtual components may be pods defining containers in a Kubernetes system. Alternatively, the virtual components may be virtual machines (VMs) in an OpenStack system. Each virtual component may have one or more corresponding production services. Production services may also be shared between plural virtual components. Each production service may represent a functionality provided by (or at least partially by) the virtual component to which it corresponds in the production environment. Each production service may specify data such as an IP address, a port, and/or a domain name through which the production service is accessible.

Figures 3A and 3B show network nodes 300A and 300B in accordance with certain embodiments. The network nodes 300A and 300B are examples of devices that may perform the method of Figure 2. The method of performing isolated testing on a virtualised cloud layer may be performed in a communication system. The communication system may be a telecommunications system such as, for example, a 3GPP 5G network (i.e. a 5th generation new radio (5G NR) network).

In step S202, each virtual component is marked with a production key to identify the virtual components as production virtual components executed in the production environment (i.e. to provide production services). Once a virtual component is marked with a production key, the virtual component may be identified as a production virtual component for use in providing a corresponding production service to a user within or outside the production environment. Marking each virtual component may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a virtual component marker 352 of the network node 300B.

In step S204, each production service is marked with a production key to identify the production services as services provided in the production environment. Once a production service is marked with a production key, the production service may be identified as a service which is provided to a user (e.g. via a corresponding virtual component) in the production environment. Marking each production service may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a production service marker 354 of the network node 300B.

Each production key may be a key value pair used to identify and/or organise corresponding production virtual components and production services.

In step S206, a test configuration is received which defines a virtual component under test. The test configuration may further define an environment and/or context of the virtual component under test (e.g. for collocated components, whether there are any and what kind). The test configuration may be received from, for example, a test planner (i.e. a manager entity which based on some criteria is capable of selecting test cases from a test repository and tailoring them to the production environment which needs to be tested). The test planner may be an administrator/system integrator who may want to run some tests for diagnostics, or a process which is triggered under some circumstances.

The virtual component under test may be selected from among the virtual components configured to provide corresponding production services. That is, the test configuration may define a virtual component under test selected from among the configured virtual components (i.e. the virtual components configured to be executed to provide corresponding production services in the production environment). Alternatively or additionally, the test configuration may define other virtual components selected from among other configured virtual components collocated with the virtual component under test. For example, in certain embodiments, the virtualised cloud layer may comprise different virtual components (i.e. other virtual components which provide other functionality) which need to be collocated with the virtual component under test. Receiving the test configuration may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a receiver 356 of the network node 300B.

In step S208, a test virtual component is created based on the virtual component under test defined in the test configuration. The test virtual component may be created to be identical to the virtual component under test. The test virtual component may be marked with a test key in order to distinguish the production virtual components, which are marked with a production key, from the test virtual component. Creating the test virtual component may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a test virtual component creator 358 of the network node 300B.

The test virtual component may be a created as a completely new virtual component in the virtualised cloud layer. Alternatively, the test virtual component may be created by reconfiguring one of the already running virtual components (i.e. from the previous test configuration or a production virtual component, which needs to be turned into a test virtual component). In certain embodiments, a single test configuration may define different virtual components under test. Different test virtual components may subsequently be created based on the different virtual components under test defined in the single test configuration. The term “different” may be taken to mean the providing of different functionality.

In step S210, a test service is created based on one of the production services. For example the test service may be created based on one of the production services that corresponds to the virtual component under test (i.e. the production service corresponding to the configured virtual component selected as the virtual component under test). In embodiments where the virtual component under test is selected from among virtual components configured to execute corresponding production services, the test service may be created to be identical to the corresponding production service of the selected virtual component. In other embodiments, the test service may be predefined in the test configuration (e.g. based on the functionality available to be provided by the configured virtual components in the virtualised cloud layer). The test service may be marked with a test key in order to distinguish from the production service which is marked with a production key. Creating the test service may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a test service creator 360 of the network node 300B.

In some embodiments, plural test configurations may be received. In such embodiments, each test configuration may define one or more different virtual component under test. For example, if three test configurations are received, the first test configuration may define a single virtual component under test selected from among the configured virtual components, the second test configuration may define two interdependent virtual components under test selected from among the configured virtual components, and third test configuration may define a single virtual component under test selected from among other virtual component (i.e. a virtual component other than the configured virtual components).

In embodiments where plural test configurations are received, after the above discussed test configuration (i.e. an initial test configuration) has been processed (i.e. a first test component and a first test service have been created), a next test configuration may be selected from among the (remaining) plural test configurations. In the embodiment discussed above, the second test configuration defining two interdependent virtual components may be selected as the next test configuration for processing. The next test configuration may be processed in a similar manner to the above discussed test configuration (i.e. the initial test configuration). That is, one or more different test virtual component may be created based on the one or more different virtual component under test defined in the selected next test configuration, and by creating one or more different test services based on one or more of the production services. In the embodiment of selecting the second test configuration as the next test configuration, two interdependent test virtual components may be created based on the selected configured virtual components and two interdependent test services may be created based on production services corresponding to the configured virtual components.

In some embodiments, the method may further comprise determining whether each test configuration is one of: an ordered test configuration and an unordered test configuration. Test configurations determined to be ordered test configurations may be processed consecutively. That is, an ordered test configuration may be a test configuration the processing of which depends on processing of preceding and/or subsequent test configurations. For example, a given ordered test configuration may depend on four other (different) test configurations all of which are processed in succession. The order of succession may be dependent on the function of the virtual components under test defined in the respective test configurations or may be based on the same component under test needing to be tested under different circumstances/states. For example, the given test configuration may be the third test configuration to be processed in the order of succession. Test configurations determined to be unordered test configurations may be processed in parallel (e.g. simultaneously) (i.e. the test virtual components and test services are created for all these unordered test configurations). Advantageously, executing unordered test configurations in parallel improves the processing efficiency and overall performance of the method.

The method may further comprise marking the test virtual component (or each test virtual component) with a test key to identify the test virtual component (or each test virtual component) as a virtual component(s) to be tested in the production environment. Once a test virtual component is marked with a test key, the test virtual component may be identified as a test virtual component for use in testing only (i.e. not for providing a corresponding production service to a user in the production environment).

The method may further comprise marking the test service (or each test service) with a test key to identify the test service (or each test service) as a test service(s) to be tested in the production environment. Once a test service is marked with a test key, the test service may be identified as a test service for use in testing only (i.e. not for providing a service to the user via a corresponding virtual component).

Advantageously, test keys are used to identify virtual components and services for use in testing only and production keys are used to identify virtual components for use in providing corresponding production services only. Test keys and production keys thereby ensure test component remain isolated from production components (i.e. configured virtual components and production services).

Each test key may be a key value pair used to identify and/or organise corresponding test virtual components and test services.

In step S212, a test case is selected, wherein the test case comprises the test service and the test virtual component (i.e. the test case exercises the test virtual component via the test service). That is, the test virtual component created in step S208 and the test service created in step S210 may be prerequisites for selecting a test case. Selecting the test case may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by a selector 352 of the network node 300B.

A next test case may be selected based on the one or more different test virtual components and the one or more different test services of the (selected) next test configuration. Therefore, the next test case may comprise the one or more different test virtual components and the one or more different test services. Following on from the above embodiment of selecting the second test configuration as the next test configuration, the next test case may comprise the two interdependent test virtual components and two interdependent test services corresponding to the selected configured virtual components. The next test case may subsequently be executed thereby performing isolated testing in the production environment. According to certain embodiments, each test case may further define one or more collocated virtual components configured to be executed to provide corresponding production services in the production environment. The collocated virtual components may be a ReplicaSets or Deployment, as discussed above. The next test case may further comprise the one or more collocated virtual components defined in a corresponding test configuration. Similarly, the initial test configuration may further define one or more collocated virtual components configured to be executed to provide corresponding production services in the production environment.

In some embodiments, a dependency may exist between at least two production services. A dependency between services may indicate that the at least two services are required in order to achieve an overall functionality desired by a user. For example, in order to achieve a functionality of X, first and second interdependent production services may both be required. In such embodiments, the method may comprise determining a dependency between at least two test services based on a dependency between at least two production services. That is, two test services may be determined to have the same or a similar dependency to two corresponding production services. When selecting a test case, the test case may additionally or alternatively comprise the at least two dependent test services and corresponding test virtual components (i.e. the test virtual components configured to provide the at least two dependent test services).

In step S214, the test case is executed. That is, the test case may be executed in the production environment by configuring the test virtual component to provide the test service in the production environment. Executing the test case may be performed, for example, by the processor 302 of the network node 300A running a program stored on the memory 304 in conjunction with the interfaces 306, or may be performed by an executor 352 of the network node 300B.

In accordance with certain embodiments, the test case may be executed in isolation from the virtual components configured to be executed to provide corresponding production services in the production environment and the production services.

In embodiments comprising plural test cases, executing the next test case may comprise identifying existing test services to keep, identifying existing test virtual components to keep, removing superfluous test services, and removing superfluous test virtual components. Existing test virtual components and existing test services may be test virtual component and test services created during processing of one or more previous test configurations. Superfluous test virtual components and test services may be previously created test virtual components and test services which are not required to execute the functionality required for a present test case. Removing superfluous test virtual components and test services may comprise deleting them from the virtualised cloud layer.

Advantageously, by removing superfluous test virtual components and test services, storage space is made available for processing of further test configurations. Furthermore, the risk of inadvertently executing a test case comprising superfluous test virtual components and test services is mitigated, which thereby improves processing efficiency.

Furthermore, by identifying test components and/or test services to maintain/re- create, the next test configuration provides the advantage of saving time and reducing potential disturbance to the system.

The method may further comprise determining if further test configurations have been received. That is, after the (initial) test configuration has been processed and/or the corresponding (initial) test case has been executed, the method may proceed to determine whether at least one further test configuration has been received. The method may further determine if subsequent test configurations have been received after the next test configuration has been processed and/or the next test case has been executed.

If it is determined that no further test configurations have been received (e.g. after the initial test case has been executed and/or after the next test case has been executed), the method may remove one or more test service and one or more test virtual component, remove production keys from the virtual components, and/or remove production keys from the production services. Removing one or more test service and one or more test virtual component may comprise deleting (i.e. terminating) at least one created test virtual component and at least one test service from the virtualised cloud layer. In some embodiments, all the test service and test virtual component may be removed. Removing production keys from the virtual components and the production services may comprise deleting the production keys such that the virtual component and production service are no longer marked. In certain embodiments, executing the test case (or each test case) may comprise generating test traffic and routing the test traffic only to the test virtual components. That is, for a given test case, test traffic may be routed to a corresponding test virtual component in order to provide a corresponding test service of that test virtual component. The test traffic may be routed only to the test virtual component and not to the production virtual components. The routing of test traffic to the test virtual component only may be controlled by the production keys and/or test keys. For example, test traffic may be configured to be routed only to components marked with test keys (i.e. the test virtual components). The test traffic may be routed away from any components marked with a production key.

Executing the virtual components in the production environment may comprise routing production traffic to the production virtual components only. That is, execution of the production virtual components may be performed by routing production traffic to the production virtual components only and not to the test virtual components. The routing of production traffic to the production virtual components only may be controlled by the production keys and/or the test keys. For example, production traffic may be configured to be routed only to components marked with production keys.

In some embodiments, the test traffic and production traffic may be processed according to marking at the same time. Advantageously, simultaneous processing of test traffic and production traffic according to marking ensures minimal impact on a user experience during testing while providing a means for complete and accurate testing.

Furthermore, errors that happen in the production environment cannot be easily recreated in a lab environment due to the different context. Therefore, the present disclosure provides a means for comprehensive testing of test traffic and test components in a production environment without interfering with production traffic and production components.

Production traffic may be network traffic generated by users accessing the virtualised cloud layer in order to utilise the functionality of certain virtual components.

Test traffic may be artificial network traffic generated based on, for example, historical production traffic and/or theoretical network traffic. Advantageously, the use of artificially generated test traffic maximises the range of test scenarios the test virtual components may be exposed to during testing.

Advantageously, routing test traffic to test virtual components only and routing production traffic to production virtual components only ensured that test cases are executed in the production environment in isolation from production services and production virtual components.

Furthermore, by using separate test services as part of a deployment of test configurations, the risk of test interference is alleviated for the services being tested. Moreover, the test traffic goes through only and exactly the test virtual components intended to compose the test configuration. Thus, the test configuration is properly deployed, and the isolation of test traffic from production traffic is achieved.

Specific embodiments of performing isolated testing on a virtualised cloud layer by a network node will now be discussed.

Functionalities of a system may need to be tested under various test configurations. Such test configurations could be representative of the runtime configuration states under which the services that compose the functionality being tested may be provided. A test configuration specifies the locations of test pods (i.e. test virtual components) and their collocated pods (i.e. pods of other ReplicaSets or Deployments hosted on the same node). A functional test may involve a single pod under test while a stress test usually involves more than one pod under test. Depending on the test method, more than one test configuration may be deployed at a time, but, in any case, a pod under test may only be part of a single test configuration. Under these circumstances to provide test isolation during testing it is desirable to control a route test traffic takes.

A brief summary of the method according to the present disclosure will now be provided followed by a more detailed description with reference to Figures 4 to 6.

The method of the present disclosure deploys a test configuration by creating additional services and test pods for test traffic, which are similar to those used for production traffic. Using additional keys in labels and selectors, test and production entities may be distinguished to ensure that test traffic is isolated and handled by test pods only. The steps performed to execute a test session consisting of partially ordered test configurations are as follows:

• An existing production deployment is marked as such using an additional key “role” in the pod labels and service selectors.

• A batch of unordered test configurations is selected for deployment and execution within which for each test configuration: new test services are created for each ReplicaSet/Deployment based on their existing services. The names of the test services and the “role” key of their selectors include the test configuration’s identifier. new test pods are created according to the test configuration based on the existing pod templates. These test pods have in their labels the “role” key include the test configuration’s identifier.

• When all test configurations of a batch have been deployed, test cases requiring these test configurations are executed.

• After execution of the test cases, tear down of the deployed test configurations is performed. If there are remaining test configurations, a new batch is selected and deployed, as described in the steps above. If the test session has been completed, production markings of the deployment are removed and the test results are reported.

Figure 4 shows various entities in play when using the method of the present disclosure. In the embodiment of Figure 4, the system under test is composed of a Kubernetes managed cluster deployed on a set of physical or virtual nodes. The nodes expose some application services provided and protected by pods (i.e. virtual components) according to the configuration. It is assumed that only a single pod of a given ReplicaSet or Deployment (i.e. (set of) configured virtual components) is located on each node. That is, pods instantiated on a node may belong to different ReplicaSets and/or Deployments. The entity responsible of deploying the test configurations and running the test cases may be referred to as a Test Execution Framework (TEF). The TEF interacts with Kubernetes and acts on entities such as pods and services in order to ensure test isolation while conducting a test execution in production (i.e. execution of a test case). The TEF may be deployed in any location as long as that location has proper access to the Kubernetes cluster master. Such access consists of network connectivity to the master nodes of the Kubernetes cluster, and proper placement, in the node(s) hosting the TEF, of Kubernetes configuration files that indicate the IP and port of kube-api-server and the required authentication certificates. Therefore, the TEF nodes may be either part of the Kubernetes cluster the TEF is testing or in any other location meeting the above mentioned conditions.

Accordingly, a set of partially ordered test configurations to be deployed is received as an input. Unordered test configurations may be deployed simultaneously. Each test configuration specifies the type and number of test pods (i.e. as virtual components under test) per ReplicaSet/Deployment (i.e. (set of) configured virtual components) protecting a service that composes the functionality being tested. The test configuration also specifies the placement of each pod under test involved in its realisation.

Figure 5 illustrates the overall flow of the present disclosure. When a test session is launched (S502), a key is added (S504), e.g. “role”, to labels of existing pods that may be used to distinguish production pods (i.e. configured virtual components) from test pods (i.e. test virtual components) and , in case of production pods assign them the value “production” (i.e. production key), for example. The method then adds (S506) a similar key to the selectors of existing services (i.e. production services) that may be used to ensure that production traffic goes to production pods only and does not go to test pods, and assign the value “production” to these selectors (i.e. production key).

The next step is the deployment of a batch of unordered test configurations. Firstly, for each test configuration, a test service is created (S508) for each ReplicaSet/Deployment. A test service may be identical to a service used for the production traffic except for the name of the test service and the related selector, i.e., the test key (vs production key) in the selector that distinguishes them. To ensure the difference, an ID of the test configuration may be appended to the name of the service as well as assigned to the “role” key of the test selector. Next, the test pods are created (S508) for the deployment of the test configuration. The test pods may be created through a sequence of actions as follows:

• Create a pod template for each test pod, similar to the template of the pods used in production.

• Add a node selector to the created template to specify the node on which the pod under test is to be deployed.

• Add a “role” key to the label in the template of the pod to match that in the selector of its associated service. • If the production pods are managed by a Deployment : delete the Deployment managing the pods in production with cascade option set to “false” or “orphan”.

• Delete the ReplicaSet managing the pods in production with cascade option set to “false” or “orphan”. (Note that this step applies whether the pods are managed by a Deployment or a ReplicaSet as pods of a Deployment are also managed by a ReplicaSet associated with the Deployment).

• If the production pods were managed by a Deployment: add the key “pod- template-hash” to the label in the templates of test pods, its value should match the value of the same key in the labels of the pods in production.

• Instantiate the test pods.

• Increase the number of replicas in the yaml file of the deleted ReplicaSet and possibly Deployment by the number of test pods that were instantiated for it.

• Re-create the deleted ReplicaSet and possibly Deployment using the modified yaml file.

Certain steps illustrated in Figure 5 may be omitted depending on the Kubernetes controller (e.g. ReplicaSet, Deployment) managing the pods. In the embodiment of Figure 5, it is assumed that the production pods and the test pods are managed by the same Kubernetes controller; however, other options may be possible. For example, a Deployment may be associated with two ReplicaSets: one for the production pods and another for the test pods.

The dependencies between the test services of the test configuration are then set (S508) in a similar way to the dependencies between the services of the production traffic.

In step S510, test cases involving test services and test virtual components created in steps S508 are run (i.e. test cases are executed).

In step S512, the method checks for more test configurations. If more test configurations are identified, the result of step S512 is “Yes” and the method proceeds to step S514. At step S514, the method identifies whether any previously created test services and test pods should be kept for use with the present test configuration. Based on the result of step S514, unneeded (i.e. superfluous) test services and test pods are removed at step S516 before the method proceeding to S508 again. If no more test configurations are identified, the result of step S512 is “No” and the method proceeds to step S518. At step S518, the following elements are removed: test services, test pods, “role” key from the labels of existing pods. The method then proceeds to step S520 where the test session ends.

Figure 6 illustrates the deployment of a test configuration (tConfl ) using the disclosed method. The test configuration (tConfl) comprises test pods (illustrated by cog symbols with a border) of ReplicaSets RS2, RS3, and RS5 on Nodel , Node5, and Node6 respectively. Assuming that the production pods and production services have already been marked as such with the appropriate production keys, the first step of deploying tConfl is to create test services, namely Srcv2-tConf1 , Srvc3-tConf1 , and Srvc5-tConf1 based on the original services Srvc2, Servc3 and Servc5. The second step is to create the test pods on Nodel, Node5 and Node6 for the ReplicaSets RS2, RS3, and RS5 respectively by increasing the number of pods to be instantiated in the ReplicaSets, creating the test pods, and waiting for the ReplicaSets to claim them as the labels of the test pods match the selector of the ReplicaSet.

The matchings between ReplicaSets, pods, and services are done using labels and selectors. T able 1 , below, illustrates an embodiment setup of labels and selectors that may be used to deploy the test configuration of Figure 6.

Table 1 - Labels and selectors for objects in play in the embodiment of Figure 6

The ReplicaSets have selectors (rows 1-3 in Table 1) that match the labels of corresponding pods (both production (rows 7-9) and under test (rows 13-15)) related to a specific service as they only use one key for the matching (i.e. the key “app”). In the embodiment of Figure 6, the same ReplicaSet is used for the orchestration of pods serving both the production and the test traffic for a given service.

The pods of these ReplicaSets have a label with the “role” key, which takes one of two values “production” or “test-tConf1”. The pods with a role “production” (rows 7-9) are used for serving the production traffic and handling user requests. The pods with a role “test-*” where “*” is the Confld value (rows 13-15) are used for handling the test traffic of the given test configuration.

The services that represent the production traffic (rows 4-6) then have selectors that match the labels of the serving pods and only those, thus the use of both keys in the selector. Similarly, the services that represent the test traffic (rows 10-12) have selectors that match the labels of the test pods and only those.

In the third step, the dependencies shown by dashed line arrows between the test services are set up according to the dependencies between the production services shown by solid line arrows. Test traffic is illustrated with the block arrows having solid lines and production traffic is illustrated with the block arrows having dashed lines. Serving pods are illustrated by cog symbols and test pods are illustrated by cog symbols having a border.

The embodiment of Figure 6 and Table 1 illustrates the deployment of one test configuration. However, as indicated above, unordered test configurations may be deployed simultaneously. In this case the setup of labels and selectors may be modified as follows:

• For each test configuration, one test service is created per ReplicaSet/Deployment for which the Confld of the test configuration is used as the values of the “role” key in the label as shown in Table 1 for tConfl (rows 10-12).

• For each test service created, test pods are created which match the test configuration and with labels that match the selector of their associated test service as shown in Table 1 for tConfl (rows 13-15). • Dependencies between the test pods are setup to match the functional path specified in the test configuration. In other words, the functional dependencies between test pods of the same test configuration are configured.

Once the test configurations of a selected batch are deployed, the TEF executes the test cases that need to run under these deployed test configurations.

When the testing under the selected batch of test configurations is completed, the TEF tears down a test configuration, as follows:

• If the production pods are managed using a Deployment: delete the deployment managing the pods in production with cascade option set to “false” or “orphan”.

• Delete the ReplicaSet managing the pods in production with cascade option set to “false” or “orphan”.

• Terminate and remove the test pods that need to be removed.

• Decrease the number of replicas in the yaml file of the deleted ReplicaSet/Deployment by the number of test pods that were removed for it.

• Re-create the ReplicaSet/Deployment using the modified yaml file.

If there are remaining test configurations to deploy, the TEF ensures the transition to the next batch of test configurations by first tearing down the not needed (i.e. superfluous) test pods of the currently deployed test configurations and creating the needed test pods of the next batch. The identification of what needs to be removed and what needs to be kept in the tear down of test configurations may be done in several ways such as:

• Delete all entities that were created for the current batch and create from scratch the entities needed for the next batch. In this case, after running the tests in each iteration, the set of “test services to keep” and the set of “test pods to keep” will be equal to the empty set; or

• Compare the currently deployed batch of test configurations with the batch of test configurations for the next iteration. Reuse the test services and test pods from previous iteration in order to setup for the next iteration.

Advantageously, when separate test services are used as part of a deployment of a test configuration (as illustrated in Figure 6), the risk of test interference is mitigated. Moreover, the test traffic is routed through only and exactly the test pods intended to compose the test configuration. Thus, the test configuration is properly deployed and test isolation is achieved.

It will be understood that the detailed examples outlined above are merely examples. According to embodiments herein, the steps may be presented in a different order to that described herein. Furthermore, additional steps may be incorporated in the method that are not explicitly recited above. For the avoidance of doubt, the scope of protection is defined by the claims.

Claims

1. A method of performing isolated testing on a virtualised cloud layer in a production environment, the virtualised cloud layer comprising virtual components configured to be executed to provide corresponding production services in the production environment, the method comprising: marking each virtual component with a production key to identify the virtual components as production virtual components executed in the production environment; marking each production service with a production key to identify the production services as services provided in the production environment; receiving a test configuration defining a virtual component under test; creating a test virtual component based on the virtual component under test defined in the test configuration; creating a test service based on one of the production services; selecting a test case comprising the test service and the test virtual component; and executing the test case.

2. The method according to claim 1, wherein the method further comprises: receiving plural test configurations, each test configuration defining one or more different virtual component under test; selecting a next test configuration from among the plural test configurations; creating one or more different test virtual component based on the one or more different virtual component under test defined in the selected next test configuration; creating one or more different test services based on one or more of the production services; selecting a next test case comprising: the one or more different test virtual components, and the one or more different test services; and executing the next test case.

3. The method according to claim 1 or claim 2, wherein: a test configuration further defines one or more collocated virtual components configured to be executed to provide corresponding production services in the production environment, and the test case and/or the next test case further comprises one or more collocated virtual components.

4. The method according to claim 2 or claim 3, the method further comprising: determining whether each test configuration is one of: an ordered test configuration and an unordered test configuration, wherein: test configurations determined to be ordered test configurations are processed consecutively, and test configurations determined to be unordered test configurations are processed in parallel.

5. The method according to any of claims 2 and 4, wherein executing the next test case comprises: identifying existing test services to keep; identifying existing test virtual components to keep; removing superfluous test services; and removing superfluous test virtual components.

6. The method according to any preceding claim, the method further comprising: marking one or more test virtual components with a test key to identify the one or more test virtual components as components to be tested in the production environment; marking one or more test service with a test key to identify the one or more test services as services to be tested in the production environment.

7. The method according to any preceding claim, the method further comprising: determining if further test configurations have been received; and if no further test configurations have been received: removing one or more test service and one or more test virtual component, removing production keys from the production virtual components, and/or removing production keys from the production services.

8. The method according to any preceding claim, wherein: a test configuration defines a virtual component under test selected from among the configured virtual components.

9. The method according to claim 8, wherein: a test service is based on a production service corresponding to the configured virtual component selected as the virtual component under test.

10. The method according to any of claims 1 to 7, wherein: a test configuration defines a virtual component under test selected from among other configured virtual components.

11. The method according to any preceding claim, wherein a test case is executed in isolation from the production virtual components configured to be executed to provide corresponding production services in the production environment and the production services.

12. The method according to any preceding claim, wherein executing the virtual components in the production environment comprises routing production traffic to the production virtual components only.

13. The method according to any preceding claim, the method further comprising: generating test traffic, wherein executing one or more test cases comprises routing the test traffic only to the test virtual component.

14. The method according to any preceding claim, the method further comprising: creating different test virtual components based on different virtual components under test defined in a single test configuration.

15. The method according to any of claims 2 to 14, the method further comprising: determining a dependency between at least two test services based on a dependency between at least two production services; selecting a test case comprising the at least two dependent test services and corresponding test virtual components.

16. The method according to any preceding claim, wherein the virtualised cloud layer is at least one of an OpenStack virtualisation layer and a Kubernetes virtualisation layer.

17. A network node configured to performing isolated testing on a virtualised cloud layer in a production environment, the virtualised cloud layer comprising virtual components configured to be executed to provide corresponding production services in the production environment, the network node comprising processing circuitry and a memory containing instructions executable by the processing circuitry, whereby the network node is operable to: mark each virtual component with a production key to identify the virtual components as production virtual components executed in the production environment; mark each production service with a production key to identify the production services as services provided in the production environment; receive a test configuration defining a virtual component under test; create a test virtual component based on the virtual component under test defined in the test configuration; create a test service based on one of the production services; select a test case comprising the test service and the test virtual component; and execute the test case.

18. The network node according to claim 18, wherein the network node is further operable to perform the method according to any of claims 1 to 16.

19. A computer-readable medium comprising instructions which, when executed on a computer, cause the computer to perform a method in accordance with any of claims 1 to 16.