WO2021123100A1 - System and method for testing of system under test - Google Patents
System and method for testing of system under test Download PDFInfo
- Publication number
- WO2021123100A1 WO2021123100A1 PCT/EP2020/086936 EP2020086936W WO2021123100A1 WO 2021123100 A1 WO2021123100 A1 WO 2021123100A1 EP 2020086936 W EP2020086936 W EP 2020086936W WO 2021123100 A1 WO2021123100 A1 WO 2021123100A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- test
- testing system
- actors
- under test
- testing
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/22—Detection or location of defective computer hardware by testing during standby operation or during idle time, e.g. start-up testing
- G06F11/26—Functional testing
- G06F11/261—Functional testing by simulating additional hardware, e.g. fault simulation
Definitions
- the present disclosure relates, generally, to a testing system and method, and more particularly, to a system and method for testing a system under test (SUT).
- SUT system under test
- test automation is the use of software separate from the system being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes.
- Testing of products or systems using protocols for communication may be highly complex and challenging due to, inter alia, complex internal and external dependencies and complex timing.
- a server in a network for mobile communication may be dependent on a number of hardware and software entities and layers, including the control layer, connectivity layer, service layer etc.
- test environment In which simulators represent entities surrounding the system under test.
- simulators represent entities surrounding the system under test.
- simulators must be carefully configured to work with specific test cases and may be difficult to control during test execution.
- test environment and test cases Other issues related to the development of the test environment and test cases are data and address management. In complex protocol-based systems, there may be a continuous need to update and maintain the test data, including data being used and communicated in the system and verification data, as well as the addresses of the system under test.
- the present disclosure relates to a testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is configured to simulate independent processes; wherein each actor comprises logic representing a fraction of the function or entity that the actor represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses, optionally with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names.
- the interface names of the intermediate network interfaces may point to networks.
- the testing system introduces a number of separations, which in different combinations may contribute to a highly flexible and controllable system, which at the same time simplifies the development and test process.
- the testing system may comprise flow separation, logic separation, data separation and address separation.
- Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain.
- Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors.
- Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor (i.e. sender or receiver), at test case level or at testing system level.
- Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses. The combination of separations has the purpose of simplifying the development of test cases and the testing environment.
- the system under test may be, but is not limited to, for example, a server, such as a proxy server for mobile network, a database server, a computing server etc.
- servers may communicate with a number of different entities, including, but not limited to, mobile devices, a home subscriber server, worldwide web server and rights server. These entities may be complex entities themselves and communicate with the SUT using various protocols, such as a radius protocol, a diameter protocol or an HTTP protocol.
- the testing of systems may therefore be highly complex.
- the presently disclosed testing system for testing a system under test in a protocol-based communication environment may encapsulate the system under test and provide improved control through the separations. A user can easily track and debug by observing, in a sequence diagram, sequences of requests and responses between the system under test and the actors. While the system under test is isolated, the user has direct control of data, addresses and behaviour of the actors, with full reusability of data and/or addresses and/or actor functionality.
- the present disclosure further relates to a computer-implemented method for testing a system under test, the method comprising: simulating a plurality of actors representing functions and/or entities outside the system under test; executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is simulated as a number of independent processes; wherein each actor comprises logic representing a fraction of the function or entity that the actor represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names.
- Fig. 1 shows an example of a system under test.
- Fig. 2 shows an embodiment of the presently disclosed testing system for testing a system under test.
- Fig. 3 shows an example of requests and responses between the system under test and the actors in a sequence diagram.
- Fig. 4 shows a further example of requests and responses between the system under test and the actors in a sequence diagram.
- Fig. 5 shows an embodiment of the presently disclosed computer-implemented method for testing a system under test in a protocol-based communication environment.
- Fig. 6 shows an example of a configuration of addresses and interfaces according to the presently disclosed testing system.
- Fig. 7 shows an example of a graphical presentation of communication addresses being distributed to actors.
- the present disclosure relates to a testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result.
- the verification of an expected result against an actual result may, within the present disclosure, be construed broadly and may include, for example, a verification of test data, but may also include a verification of the functioning of a technical function or that that the system under test remains in a predefined state.
- the verification that the system under test remains in a predefined state may be, for example, a verification that the system is still in an operational state.
- the testing system may comprise any combination of flow separation, logic separation, data separation and address separation.
- Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain.
- Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors.
- Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor (i.e. sender or receiver), at test case level or at testing system level.
- Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses. Further embodiments and details of the separations are provided below.
- An actor is an entity for simulating a process.
- An actor may be seen as a simulated process.
- An actor carries out a task, typically a task that is related to another actor or to the system under test.
- the task may be a technical task, for example a task involving sending/receiving requests and responses to other actors and/or the system under test.
- An actor will typically simulate a technical function, such as a technical function in a server.
- the simulation could involve, but is not limited to, carrying out a function related to a permission or restriction, or fetching and/or delivering data such as images.
- Actors may be configured to simulate a wide range of technical processes.
- An actor may simulate a logical process.
- the testing system is configured to set up dependencies between the plurality of actors, including internal dependencies between the plurality of actors, and the system under test.
- a user may, through a user interface, define the dependencies between the plurality of actors and the system under test.
- the testing system comprises an execution engine configured to schedule execution of the plurality of actors.
- a user can set up all dependencies, and then set up an execution order. Before a test is executed, it may be useful to supply the plurality of actors with test data and/or communication addresses. This can be done manually by a user, or partly or fully by the testing system.
- the system and the execution engine will then be able to execute a test or series of tests, wherein the dependencies of the plurality of actors and the system under test, the execution order and the provided test data and/or communication addresses are taken into account.
- the testing system may be further configured to execute a test suite comprising a plurality of tests or test sequences.
- the system under test may be a system in a protocol-based communication environment and/or a system in a communication environment based on communication over buses, such as USB-based communication. More specifically, the system under test may be a server such as a proxy server, a database server, a computing server, a mail server, a communication server, a media server, a web server gateway, a server of a network, such as a mobile network or a local network, or a network of networks such as the Internet or any other technical application that would benefit from the presently disclosed testing system.
- the system under test may also be a service, preferably a technical service.
- the system under test may be a real system, i.e. an actual system comprising physical parts and, optionally, software for controlling the physical parts.
- the system under test may be a server.
- the system under test may be a simulated system, preferably wherein technical functions, such as technical functions of a server, are simulated.
- the plurality of actors that are simulated by the testing system may comprise communication devices, such as mobile communication devices, and/or a home subscriber server and/or a worldwide web server and/or a rights server.
- the plurality of actors communicates with the system under test through a communication protocol, such as a radius protocol and/or a diameter protocol and/or an HTTP protocol.
- the system may involve communication between the actors internally and between the actors and system under test by means of request and response messages. Accordingly, each actor may be configured to generate at least one response to the system under test in response to at least one request from the system under test and/or configured to generate at least one response to another actor in response to at least one request from the other actor. Moreover, the system under test may be configured to generate at least one response to at least one request of at least one of the actors.
- One way of starting a test may be to configure the system such that one of actors generates an initial stimulus to the system under test, which may then trigger a sequence of steps and/or technical functions in the system, carried out by the system under test and/or the actors.
- the system under test starts the test.
- a test within the present disclosure may comprise a complete sequence of requests and responses between the system under test and the plurality of actors and/or internally between the actors. An example of such a sequence is shown in Fig.
- the testing system may be configured to display the complete sequence of requests and responses between the system under test and the actors in a sequence diagram, as shown in the example in Fig. 3.
- the sequence diagram may be displayed on any display.
- a user can easily follow the execution and localize any errors or unexpected results or behaviour.
- the system under test and the actors are represented as vertical lines and the requests and responses between the system under test and the actors are represented as horizontal arrows arranged in a time sequence from the top to the bottom of the display.
- Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors.
- the plurality of actors may be configured to simulate independent processes.
- an actor may comprise a set of instructions executable by the processor.
- Each actor may execute its own specific sequences comprising its own requests and responses. In practice this means that instead of controlling a schedule of a complete sequence of request and responses, the execution engine can use and re-use the actors.
- the actors may thereby be seen as fully reusable building blocks.
- a specific actor can be re-used in a number of instances.
- an actor may be implemented in any suitable software language.
- all actors may be executed in parallel.
- the presently disclosed testing allows a user to create, configure and schedule any number of actors to be simulated in the testing system.
- Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain. Full simulators add complexity to the testing environment in terms of installation and maintenance. By scaling down the actors so that they represent only the functions that are needed for a specific system under test and specific test cases, considerable flexibility, robustness and controllability can be achieved.
- each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test.
- at least one actor may further comprise logic representing faulty behaviour of the entity that the actor represents.
- the actors are scaled down to comprise only functionality that is directly related to communication with the system under test. The behaviour of the actors may be individually configurable through the testing system by a user.
- Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor, at test case level or at testing system level.
- the testing system may comprise test data.
- Test data may refer to any data that is used in the system and that can be verified.
- Test data can be, for example, an acknowledge response of a certain message, or some data that is collected from a database.
- the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system.
- the testing system comprises a user interface for controlling test data at actor level and/or at a testing system top level and/or at test case level.
- a further advantage of the presently disclosed testing system relates to the actor being configured, through a graphical user interface, to request the test data that the actor needs.
- the plurality of actors is configured to indicate to the testing system and/or to a user of the testing system which data the actors need to be simulated in the test.
- the plurality of actors is configured to indicate to the testing system and/or to a user of the testing system which verification data the actors need to be able to run against the test data.
- An actor may use data by referring to a value or to a data name. If an actor refers to a data name, data from a test or test case is used.
- Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses.
- the testing system comprises a number of communication addresses, such as IP addresses, used by the actors and/or system under test, wherein the communication addresses are accessed through intermediate network interfaces having interface names, wherein the interface names are accessed from test addresses provided in the test.
- the interface names may point to networks.
- the communication addresses may be real IP addresses.
- the communication addresses may be addresses of a network interface card, typically a limited number of available communication addresses. Communication address may be seen as actual addresses and may comprise an IP address and a port number, for example 1.2.3.4:8080.
- Each intermediate interface may have a name and a pointer to a communication address.
- Each test address may in turn comprise a pointer to an intermediate interface in the form of the name of the intermediate interface.
- an IP address is an address of a system in a network.
- a port is an address of a service within the system.
- the combination of an IP address and a port defines an address of a particular service on a particular system.
- Test addresses are the addresses that are used in tests or test cases.
- the test addresses may point to the intermediate interfaces.
- each test address may further comprise a port.
- the presently disclosed testing system may be configured to distribute the communication addresses to the system under test and the plurality of actors. If the testing system is executed from a computer, the number of available communication addresses to be used may be limited. Therefore, the communication addresses may be reused by different test addresses which use the same addresses but different ports.
- the presently disclosed testing system may be configured to perform a distribution to minimize the use of available communication addresses.
- test addresses and interfaces referred to in the test are converted to real communication addresses according to a distribution process.
- a network may, for example, be of the type IPv4 or IPv6, and may take names such as N 1 , N2, N3 or any suitable name.
- the intermediate interfaces for the networks are defined. For example, an intermediate interface may be named 11.
- the network with which the intermediate interface is associated may also be specified, for example, N1.
- the test addresses are set up.
- a test address will have a name, such as A1 , A2, A3 etc.
- the test address should also point to one of the interfaces, for example, I2.
- the test address may have a port, such as 8080 or 1234.
- the testing system now has a complete set of virtual addresses and ports.
- test When the test is executed, it is typically executed in an environment which has only a limited number of communication addresses.
- a computer may have one or a limited number of network interface cards with limited communication addresses.
- the test system Before executing the test, the test system may therefore be configured to distribute the communication addresses to test addresses used by actors.
- the communication addresses may be reused.
- the distribution scheme may make use of the fact that an IP address and a port together define an address of a particular service on a particular system. A conflict of addresses can therefore be remedied by using a different port for the same address.
- the present disclosure further relates to a computer-implemented method for testing a system under test, the method comprising: simulating a plurality of actors representing functions and/or entities outside the system under the test; executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result.
- the method may comprise any combination of flow separation, logic separation, data separation and address separation.
- the plurality of actors is simulated as independent processes, which may refer to the flow separation.
- each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test, which may refer to logic separation.
- the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system. This may refer to data separation.
- the testing system comprises a number of communication addresses, such as IP addresses, used by the actors and/or system under test, wherein the communication addresses are accessed through intermediate network interfaces having interface names, wherein the interface names are accessed from test addresses provided in the test. This may refer to address separation.
- the method may be performed using any embodiment of the presently disclosed testing system for testing a system under test. Accordingly, the method may perform any step which the presently disclosed testing system for testing a system under test is configured to perform.
- the invention further relates to a computer program having instructions which, when executed by a computing device or computing system, cause the computing device or system to carry out any embodiment of the presently disclosed computer-implemented method for testing a system under test.
- Fig. 1 shows an example of testing system (100) for testing a system under test (200) in a protocol-based environment.
- the system under test (200) communicates with a mobile device (300) using a communication protocol in the form of radius and an application protocol HTTP.
- the system under test (200) communicates with further entities: a home subscriber server (HSS) (300) through a diameter protocol, a worldwide web (WWW) (300) server using an HTTP application protocol and a rights server (300) through a socket connection.
- HSS home subscriber server
- WWW worldwide web
- Fig. 2 shows an embodiment of the presently disclosed testing system (100) for testing a system under test (200).
- the testing system (100) comprises a processing unit (110), and a memory (120) having instructions stored thereon that, when executed by the processor, carries out any embodiment of the presently disclosed computer- implemented method or any step described as part of the presently disclosed testing system (100).
- the testing system (100) further comprises a test (400) which may comprise, for example, a test sequence, or be executed as a number of test cases.
- the testing system (100) further comprises a number of actors (400) that are simulated in the testing system (100).
- Fig. 3 shows an example, in a sequence diagram, of requests and responses between the system under test (300) and the actors (400) in the example of Fig. 1.
- the actors (mobile, HSS, Rights, www) are represented as vertical lines (600).
- the requests and responses between the system under test (100) and the actors (400) are represented as horizontal arrows (500) arranged in a time sequence starting at the top of the diagram.
- Fig. 4 shows, in a sequence diagram, a further example of requests and responses between the system under test and the actors, wherein a further mobile device (mobile 2) is present.
- Fig. 5 shows an embodiment of the presently disclosed computer-implemented method (700) for testing a system under test (100).
- the method (700) comprises the steps of: simulating a plurality of actors representing functions and/or entities outside the system under the test (710); and executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result (720).
- Fig. 6 shows an example of a configuration of addresses and interfaces according to the presently disclosed testing system.
- a test address has a name (for example A1, A2 etc.). The test address also points to a specific interface. In addition, the test address has a port, such as 8080.
- An interface has a name (for example 11, I2 etc.). The interface is also associated with a specific network, such as N1, N2 etc.
- a network has a name (for example N1, N2 etc.) and is typically associated with a type, such as IPv4 or IPv6. According to type and availability, the network can be paired with real network interface cards in the distribution process.
- the interface names of the intermediate network interfaces may point to networks.
Landscapes
- Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The present disclosure relates to a testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is configured to simulate independent processes; wherein each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses, used by the actors and/or system under test, wherein the communication addresses are accessed through intermediate network interfaces having interface names, wherein the interface names are accessed from test addresses provided in the test. The present disclosure further relates to a computer-implemented method for testing a system under test.
Description
System and method fortesting of system under test
The present disclosure relates, generally, to a testing system and method, and more particularly, to a system and method for testing a system under test (SUT).
Background
In system testing, test automation is the use of software separate from the system being tested to control the execution of tests and the comparison of actual outcomes with predicted outcomes. Testing of products or systems using protocols for communication may be highly complex and challenging due to, inter alia, complex internal and external dependencies and complex timing. As an example, a server in a network for mobile communication may be dependent on a number of hardware and software entities and layers, including the control layer, connectivity layer, service layer etc.
As an alternative to testing in a complete real environment, a test environment is typically used, in which simulators represent entities surrounding the system under test. Using simulators, however, adds complexity to the testing environment. For example, the simulators need to be installed, maintained, and versions of the various simulators need to be controlled and aligned. Moreover, simulators must be carefully configured to work with specific test cases and may be difficult to control during test execution.
Other issues related to the development of the test environment and test cases are data and address management. In complex protocol-based systems, there may be a continuous need to update and maintain the test data, including data being used and communicated in the system and verification data, as well as the addresses of the system under test.
Summary
The present disclosure relates to a testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result,
wherein the plurality of actors is configured to simulate independent processes; wherein each actor comprises logic representing a fraction of the function or entity that the actor represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses, optionally with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names. The interface names of the intermediate network interfaces may point to networks.
The testing system introduces a number of separations, which in different combinations may contribute to a highly flexible and controllable system, which at the same time simplifies the development and test process.
The testing system may comprise flow separation, logic separation, data separation and address separation. Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain. Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors. Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor (i.e. sender or receiver), at test case level or at testing system level. Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses. The combination of separations has the purpose of simplifying the development of test cases and the testing environment.
The system under test may be, but is not limited to, for example, a server, such as a proxy server for mobile network, a database server, a computing server etc. Such servers may communicate with a number of different entities, including, but not limited to, mobile devices, a home subscriber server, worldwide web server and rights server. These entities may be complex entities themselves and communicate with the SUT
using various protocols, such as a radius protocol, a diameter protocol or an HTTP protocol. The testing of systems may therefore be highly complex. The presently disclosed testing system for testing a system under test in a protocol-based communication environment may encapsulate the system under test and provide improved control through the separations. A user can easily track and debug by observing, in a sequence diagram, sequences of requests and responses between the system under test and the actors. While the system under test is isolated, the user has direct control of data, addresses and behaviour of the actors, with full reusability of data and/or addresses and/or actor functionality.
The present disclosure further relates to a computer-implemented method for testing a system under test, the method comprising: simulating a plurality of actors representing functions and/or entities outside the system under test; executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is simulated as a number of independent processes; wherein each actor comprises logic representing a fraction of the function or entity that the actor represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names.
Description of drawings
Fig. 1 shows an example of a system under test.
Fig. 2 shows an embodiment of the presently disclosed testing system for testing a system under test.
Fig. 3 shows an example of requests and responses between the system under test and the actors in a sequence diagram.
Fig. 4 shows a further example of requests and responses between the system under test and the actors in a sequence diagram.
Fig. 5 shows an embodiment of the presently disclosed computer-implemented method for testing a system under test in a protocol-based communication environment.
Fig. 6 shows an example of a configuration of addresses and interfaces according to the presently disclosed testing system.
Fig. 7 shows an example of a graphical presentation of communication addresses being distributed to actors.
Detailed description
The present disclosure relates to a testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result.
The verification of an expected result against an actual result may, within the present disclosure, be construed broadly and may include, for example, a verification of test data, but may also include a verification of the functioning of a technical function or that that the system under test remains in a predefined state. The verification that the system under test remains in a predefined state may be, for example, a verification that the system is still in an operational state.
The testing system may comprise any combination of flow separation, logic separation, data separation and address separation. Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain. Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors. Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor (i.e. sender or receiver), at test case level or at testing system level. Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses. Further embodiments and details of the separations are provided below.
An actor, according to the present disclosure, is an entity for simulating a process. An actor may be seen as a simulated process. An actor carries out a task, typically a task that is related to another actor or to the system under test. The task may be a technical task, for example a task involving sending/receiving requests and responses to other actors and/or the system under test. An actor will typically simulate a technical function, such as a technical function in a server. The simulation could involve, but is not limited to, carrying out a function related to a permission or restriction, or fetching and/or delivering data such as images. Actors may be configured to simulate a wide range of technical processes. An actor may simulate a logical process.
According to one embodiment of the presently disclosed testing system, the testing system is configured to set up dependencies between the plurality of actors, including internal dependencies between the plurality of actors, and the system under test. A user may, through a user interface, define the dependencies between the plurality of actors and the system under test. Preferably, the testing system comprises an execution engine configured to schedule execution of the plurality of actors. Accordingly, in one embodiment, a user can set up all dependencies, and then set up an execution order. Before a test is executed, it may be useful to supply the plurality of actors with test data and/or communication addresses. This can be done manually by a user, or partly or fully by the testing system. The system and the execution engine will then be able to execute a test or series of tests, wherein the dependencies of the plurality of actors and the system under test, the execution order and the provided test data and/or communication addresses are taken into account. The testing system may be further configured to execute a test suite comprising a plurality of tests or test sequences.
Application of the testing system
The system under test may be a system in a protocol-based communication environment and/or a system in a communication environment based on communication over buses, such as USB-based communication. More specifically, the system under test may be a server such as a proxy server, a database server, a computing server, a mail server, a communication server, a media server, a web server gateway, a server of a network, such as a mobile network or a local network, or a network of networks such as the Internet or any other technical application that would benefit from the presently disclosed testing system. The system under test may also be
a service, preferably a technical service. The system under test may be a real system, i.e. an actual system comprising physical parts and, optionally, software for controlling the physical parts. As an example, the system under test may be a server.
Alternatively, the system under test may be a simulated system, preferably wherein technical functions, such as technical functions of a server, are simulated.
The plurality of actors that are simulated by the testing system may comprise communication devices, such as mobile communication devices, and/or a home subscriber server and/or a worldwide web server and/or a rights server. In one embodiment of the presently disclosed testing system, the plurality of actors communicates with the system under test through a communication protocol, such as a radius protocol and/or a diameter protocol and/or an HTTP protocol.
The system may involve communication between the actors internally and between the actors and system under test by means of request and response messages. Accordingly, each actor may be configured to generate at least one response to the system under test in response to at least one request from the system under test and/or configured to generate at least one response to another actor in response to at least one request from the other actor. Moreover, the system under test may be configured to generate at least one response to at least one request of at least one of the actors.
One way of starting a test may be to configure the system such that one of actors generates an initial stimulus to the system under test, which may then trigger a sequence of steps and/or technical functions in the system, carried out by the system under test and/or the actors. Alternatively, it is possible that the system under test starts the test. A test within the present disclosure may comprise a complete sequence of requests and responses between the system under test and the plurality of actors and/or internally between the actors. An example of such a sequence is shown in Fig.
3.
The testing system may be configured to display the complete sequence of requests and responses between the system under test and the actors in a sequence diagram, as shown in the example in Fig. 3. The sequence diagram may be displayed on any display. By linking the execution of the system under test and simulation of actors to a visual presentation in this manner, a user can easily follow the execution and localize
any errors or unexpected results or behaviour. In one embodiment of the graphical user interface, the system under test and the actors are represented as vertical lines and the requests and responses between the system under test and the actors are represented as horizontal arrows arranged in a time sequence from the top to the bottom of the display.
Flow separation
Flow separation may refer to the separation of steps and/or functionality in a test sequence into smaller local sequences associated with the actors. The plurality of actors may be configured to simulate independent processes. As an example, an actor may comprise a set of instructions executable by the processor. Each actor may execute its own specific sequences comprising its own requests and responses. In practice this means that instead of controlling a schedule of a complete sequence of request and responses, the execution engine can use and re-use the actors. The actors may thereby be seen as fully reusable building blocks. A specific actor can be re-used in a number of instances. In practice, an actor may be implemented in any suitable software language. In one specific embodiment, all actors may be executed in parallel. The presently disclosed testing allows a user to create, configure and schedule any number of actors to be simulated in the testing system.
Logic separation
Logic separation may refer to separating and simplifying functionality of entities that are conventionally handled by complete simulators, which are typically relatively complex to control and maintain. Full simulators add complexity to the testing environment in terms of installation and maintenance. By scaling down the actors so that they represent only the functions that are needed for a specific system under test and specific test cases, considerable flexibility, robustness and controllability can be achieved.
In one embodiment of the presently disclosed testing system, each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test. In order to test how the system under test behaves in case of errors in the system, at least one actor may further comprise logic representing faulty behaviour of the entity that the actor represents. In one specific embodiment, the actors are scaled down to comprise only functionality that
is directly related to communication with the system under test. The behaviour of the actors may be individually configurable through the testing system by a user.
Data separation
Data separation may refer to the possibility to control test data on all abstraction layers, i.e. at a specific actor, at test case level or at testing system level.
The testing system may comprise test data. Test data may refer to any data that is used in the system and that can be verified. Test data can be, for example, an acknowledge response of a certain message, or some data that is collected from a database. In one embodiment, the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system. In one embodiment, the testing system comprises a user interface for controlling test data at actor level and/or at a testing system top level and/or at test case level.
A further advantage of the presently disclosed testing system relates to the actor being configured, through a graphical user interface, to request the test data that the actor needs. This has the advantage that user can easily see what must be provided to make the actor functional for a test setup. Accordingly, in one embodiment of the presently disclosed testing system, the plurality of actors is configured to indicate to the testing system and/or to a user of the testing system which data the actors need to be simulated in the test. Similarly, the plurality of actors is configured to indicate to the testing system and/or to a user of the testing system which verification data the actors need to be able to run against the test data. An actor may use data by referring to a value or to a data name. If an actor refers to a data name, data from a test or test case is used.
Address separation
Address separation may refer to indirect access of communication addresses through intermediate interfaces having names, wherein the names of the interfaces are used in test sequence addresses. In one embodiment of the presently disclosed system, the testing system comprises a number of communication addresses, such as IP addresses, used by the actors and/or system under test, wherein the communication addresses are accessed through intermediate network interfaces having interface names, wherein the interface names are accessed from test addresses provided in the
test. The interface names may point to networks. Typically, the communication addresses may be real IP addresses. The communication addresses may be addresses of a network interface card, typically a limited number of available communication addresses. Communication address may be seen as actual addresses and may comprise an IP address and a port number, for example 1.2.3.4:8080.
Each intermediate interface may have a name and a pointer to a communication address. Each test address may in turn comprise a pointer to an intermediate interface in the form of the name of the intermediate interface.
Generally, an IP address is an address of a system in a network. A port is an address of a service within the system. Hence, the combination of an IP address and a port defines an address of a particular service on a particular system. Some ports, such as HTTP, are predefined, whereas others are flexible.
Test addresses are the addresses that are used in tests or test cases. The test addresses may point to the intermediate interfaces. In addition, each test address may further comprise a port. The presently disclosed testing system may be configured to distribute the communication addresses to the system under test and the plurality of actors. If the testing system is executed from a computer, the number of available communication addresses to be used may be limited. Therefore, the communication addresses may be reused by different test addresses which use the same addresses but different ports. The presently disclosed testing system may be configured to perform a distribution to minimize the use of available communication addresses.
When a test is executed, the test addresses and interfaces referred to in the test are converted to real communication addresses according to a distribution process.
One way of setting up the testing system may be: In a first step, networks are defined. A network may, for example, be of the type IPv4 or IPv6, and may take names such as N 1 , N2, N3 or any suitable name. In a second step, the intermediate interfaces for the networks are defined. For example, an intermediate interface may be named 11. The network with which the intermediate interface is associated may also be specified, for example, N1. In a third step, the test addresses are set up. A test address will have a name, such as A1 , A2, A3 etc. The test address should also point to one of the interfaces, for example, I2. In addition, the test address may have a port, such as 8080 or 1234. The testing system now has a complete set of virtual addresses and ports.
When the test is executed, it is typically executed in an environment which has only a limited number of communication addresses. As an example, a computer may have one or a limited number of network interface cards with limited communication addresses. Before executing the test, the test system may therefore be configured to distribute the communication addresses to test addresses used by actors. The communication addresses may be reused. In case of conflicts the distribution scheme may make use of the fact that an IP address and a port together define an address of a particular service on a particular system. A conflict of addresses can therefore be remedied by using a different port for the same address.
The present disclosure further relates to a computer-implemented method for testing a system under test, the method comprising: simulating a plurality of actors representing functions and/or entities outside the system under the test; executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result.
The method may comprise any combination of flow separation, logic separation, data separation and address separation. In particular, according to one embodiment of the computer-implemented method, the plurality of actors is simulated as independent processes, which may refer to the flow separation. In one embodiment, each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test, which may refer to logic separation. In a further embodiment of the method, the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system. This may refer to data separation. In a further embodiment, the testing system comprises a number of communication addresses, such as IP addresses, used by the actors and/or system under test, wherein the communication addresses are accessed through intermediate network interfaces having interface names, wherein the interface names are accessed from test addresses provided in the test. This may refer to address separation.
A person skilled in the art will recognize that the method may be performed using any embodiment of the presently disclosed testing system for testing a system under test.
Accordingly, the method may perform any step which the presently disclosed testing system for testing a system under test is configured to perform.
The invention further relates to a computer program having instructions which, when executed by a computing device or computing system, cause the computing device or system to carry out any embodiment of the presently disclosed computer-implemented method for testing a system under test.
Detailed description of drawings
The invention will be described in the following in greater detail, with reference to the accompanying drawings. The drawings are exemplary and intended to illustrate some of the features of the presently disclosed testing system and method, and are not to be construed as limiting to the presently disclosed invention.
Fig. 1 shows an example of testing system (100) for testing a system under test (200) in a protocol-based environment. The system under test (200) communicates with a mobile device (300) using a communication protocol in the form of radius and an application protocol HTTP. The system under test (200) communicates with further entities: a home subscriber server (HSS) (300) through a diameter protocol, a worldwide web (WWW) (300) server using an HTTP application protocol and a rights server (300) through a socket connection.
Fig. 2 shows an embodiment of the presently disclosed testing system (100) for testing a system under test (200). The testing system (100) comprises a processing unit (110), and a memory (120) having instructions stored thereon that, when executed by the processor, carries out any embodiment of the presently disclosed computer- implemented method or any step described as part of the presently disclosed testing system (100). The testing system (100) further comprises a test (400) which may comprise, for example, a test sequence, or be executed as a number of test cases. The testing system (100) further comprises a number of actors (400) that are simulated in the testing system (100).
Fig. 3 shows an example, in a sequence diagram, of requests and responses between the system under test (300) and the actors (400) in the example of Fig. 1. The actors (mobile, HSS, Rights, www) are represented as vertical lines (600). The requests and responses between the system under test (100) and the actors (400) are represented as horizontal arrows (500) arranged in a time sequence starting at the top of the
diagram. Fig. 4 shows, in a sequence diagram, a further example of requests and responses between the system under test and the actors, wherein a further mobile device (mobile 2) is present. Fig. 5 shows an embodiment of the presently disclosed computer-implemented method (700) for testing a system under test (100). The method (700) comprises the steps of: simulating a plurality of actors representing functions and/or entities outside the system under the test (710); and executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result (720).
Fig. 6 shows an example of a configuration of addresses and interfaces according to the presently disclosed testing system. A test address has a name (for example A1, A2 etc.). The test address also points to a specific interface. In addition, the test address has a port, such as 8080. An interface has a name (for example 11, I2 etc.). The interface is also associated with a specific network, such as N1, N2 etc. A network has a name (for example N1, N2 etc.) and is typically associated with a type, such as IPv4 or IPv6. According to type and availability, the network can be paired with real network interface cards in the distribution process. The interface names of the intermediate network interfaces may point to networks.
Claims
1. A testing system for testing a system under test, the testing system comprising a processing unit configured for executing instructions to: - simulate a plurality of actors representing functions and/or entities outside the system under test; execute a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is configured to simulate independent processes; wherein each actor comprises logic representing a fraction of the function or entity that the actor represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names.
2. The testing system according to claim 1 , wherein the system under test is a system in a protocol-based communication environment and/or a system in a communication environment based on communication over buses, such as USB- based communication.
3. The testing system according to any one of the preceding claims, wherein the testing system is configured to set up dependencies between the plurality of actors, including internal dependencies between the plurality of actors, and the system under test.
4. The testing system according to any one of the preceding claims, wherein the testing system comprises an execution engine configured to schedule execution of the plurality of actors.
5. The testing system according to any one of the preceding claims, wherein the testing system is further configured to supply the plurality of actors with test data and communication addresses.
6. The testing system according to any one of the preceding claims, wherein each actor is configured to generate at least one response to the system under test in response to at least one request from the system under test.
7. The testing system according to any one of the preceding claims, wherein the system under test is configured to generate at least one response to at least one request of at least one of the actors.
8. The testing system according to any one of the preceding claims, wherein one of actors generates an initial stimulus to the system under test.
9. The testing system according to any one of the preceding claims, wherein the test comprises a complete sequence of requests and responses between the system under test and the plurality of actors.
10. The testing system according to claim 9, wherein the testing system is configured to display the complete sequence of requests and responses between the system under test and the actors in a sequence diagram.
11. The testing system according to claim 10, wherein the system under test and the actors are represented as vertical lines and wherein the requests and responses between the system under test and the actors are represented as horizontal arrows arranged in a time sequence.
12. The testing system according to any one of the preceding claims, wherein the verification of an expected result against an actual result comprises a verification of data and/or a verification of a technical function of the system under test and/or a verification that the system under test remains in a predefined state, such as an operational state.
13. The testing system according to any one of the preceding claims, wherein the system under test is a server such as a proxy server, a database server, a
computing server, a mail server, a communication server, a media server, a web server gateway, a server of a network, such as a mobile network or a local network, or a network of networks such as the Internet.
14. The testing system according to any one of the preceding claims, wherein the system under test is a service.
15. The testing system according to any one of the preceding claims, wherein the plurality of actors comprises communication devices, such as mobile communication devices, and/or a home subscriber server and/or a worldwide web server and/or a rights server.
16. The testing system according to any one of the preceding claims, wherein the plurality of actors communicates with the system under test through a communication protocol, such as a radius protocol and/or a diameter protocol and/or an HTTP protocol.
17. The testing system according to any one of the preceding claims, wherein each actor comprises a set of instructions executable by the processor.
18. The testing system according to claim 17, wherein each actor executes only its own specific sequences comprising its own requests and responses.
19. The testing system according to any one of claims 17-18, wherein the sets of instructions of all actors are executed in parallel.
20. The testing system according to any one of the preceding claims, wherein at least one actor further comprises logic representing faulty behaviour of the system entity.
21. The testing system according to any one of the preceding claims, wherein behaviour of the actors is individually configurable, by a user, through the testing system.
22. The testing system according to any one of the preceding claims, wherein the actors comprise only functionality directly related to communication with the system under test.
23. The testing system according to any one of the preceding claims, wherein the testing system comprises a user interface for controlling test data at actor level and/or at a testing system top level and/or at test case level.
24. The testing system according to any one of the preceding claims, wherein the plurality of actors is configured to indicate, to the testing system and/or to a user of the testing system, the data that needs to be simulated in the test.
25. The testing system according to any one of the preceding claims, said testing system being configured to execute a test suite comprising a plurality of test sequences.
26. The testing system according to any one of the preceding claims, wherein the testing system is configured to distribute the communication addresses to the system under test and the plurality of actors.
27. The testing system according to claim 26, wherein addresses of the communication addresses are reused by different test addresses that use the same addresses but different ports.
28. The testing system according to any one of the preceding claims, wherein each test address further comprises a port.
29. The testing system according to any one of the preceding claims, wherein the communication addresses are IP addresses, preferably IP addresses with port numbers.
30. The testing system according to any one of the preceding claims, wherein each intermediate interface has a name and a pointer to a communication address.
31. The testing system according to any one of the preceding claims, wherein each test address comprises a pointer to an intermediate interface in the form of the name of the intermediate interface.
32. The testing system according to any one of the preceding claims, wherein the communication addresses are associated with test addresses when the test is executed, wherein the communication addresses with port numbers are actual addresses and the test addresses are virtual addresses in the testing system.
33. A computer-implemented method for testing a system under test, the method comprising: simulating a plurality of actors representing functions and/or entities outside the system under test; - executing a test comprising interaction between the actors and the system under test, wherein an expected result is verified against an actual result, wherein the plurality of actors is simulated as independent processes; wherein each actor comprises logic representing a fraction of the function or entity that it represents, the fraction being a minimum fraction needed for executing the test; wherein the test comprises test data, wherein the testing system is configured to allow a user, by selection, to specify and apply the test data for one of the actors, for the test or globally for all simulations within the testing system; and wherein the testing system comprises a number of communication addresses, such as IP addresses with port numbers, used by the actors and/or system under test, wherein the communication addresses are generated from test addresses pointing to intermediate network interfaces having interface names.
34. A computer program having instructions which, when executed by a computing device or computing system, causes the computing device or computing system to carry out the method for testing a system under test according to claim 33.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP19218798.7 | 2019-12-20 | ||
EP19218798 | 2019-12-20 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021123100A1 true WO2021123100A1 (en) | 2021-06-24 |
Family
ID=69061087
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/EP2020/086936 WO2021123100A1 (en) | 2019-12-20 | 2020-12-18 | System and method for testing of system under test |
Country Status (1)
Country | Link |
---|---|
WO (1) | WO2021123100A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0453152A2 (en) * | 1990-04-17 | 1991-10-23 | AT&T Corp. | Computer program specification system |
US6832184B1 (en) * | 2000-03-02 | 2004-12-14 | International Business Machines Corporation | Intelligent work station simulation—generalized LAN frame generation simulation structure |
US20120158395A1 (en) * | 2010-12-15 | 2012-06-21 | ZanttZ, Inc. | Network stimulation engine |
WO2018144036A1 (en) * | 2017-02-06 | 2018-08-09 | Visa International Service Association | Simulator for system testing |
-
2020
- 2020-12-18 WO PCT/EP2020/086936 patent/WO2021123100A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0453152A2 (en) * | 1990-04-17 | 1991-10-23 | AT&T Corp. | Computer program specification system |
US6832184B1 (en) * | 2000-03-02 | 2004-12-14 | International Business Machines Corporation | Intelligent work station simulation—generalized LAN frame generation simulation structure |
US20120158395A1 (en) * | 2010-12-15 | 2012-06-21 | ZanttZ, Inc. | Network stimulation engine |
WO2018144036A1 (en) * | 2017-02-06 | 2018-08-09 | Visa International Service Association | Simulator for system testing |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
Rosenkranz et al. | A distributed test system architecture for open-source IoT software | |
CN111819538A (en) | Workpiece lifecycle management on cloud computing systems | |
US10409632B2 (en) | Emulation of hardware components based on interrupt design | |
Sandobalin et al. | An infrastructure modelling tool for cloud provisioning | |
US9348771B1 (en) | Cloud-based instrument driver system | |
US11347531B2 (en) | Generalized virtualization platform for systems using hardware abstraction software layers | |
US20150113423A1 (en) | System and method to configure a field device | |
CN110825985B (en) | Data acquisition system, method, device, control equipment and proxy equipment | |
CN112242920B (en) | Cloud platform deployment method, device and node | |
US20140157057A1 (en) | Code-free testing framework | |
CN106897223A (en) | A kind of software kit delivery method and device based on continuous integrating | |
CN106775951A (en) | A kind of operation management method and device of Android application | |
JP5708078B2 (en) | Verification device, verification method, and verification program | |
WO2021123100A1 (en) | System and method for testing of system under test | |
EP3734379A1 (en) | Method and system for generating control programs in a cloud computing environment | |
CN110471767A (en) | A kind of dispatching method of equipment | |
US9195475B2 (en) | Data relay by server | |
Muli et al. | Virtual validation-a new paradigm in controls engineering | |
CN116157774A (en) | Method and system for providing engineering of industrial equipment in cloud computing environment | |
EP3014562A1 (en) | Simulating sensors | |
Singh et al. | it@ intel | |
US20240094694A1 (en) | Virtual Deployment of Distributed Control Systems for Control Logic Testing | |
Singh et al. | A reliability growth model for object oriented software developed under concurrent distributed development environment | |
Smith et al. | The Careful Puppet Master: Reducing risk and fortifying acceptance testing with Jenkins CI | |
Alptekin et al. | Design for ARINC 653 conformance: Architecting independent validation of a safety-critical RTOS |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 20824578 Country of ref document: EP Kind code of ref document: A1 |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
32PN | Ep: public notification in the ep bulletin as address of the adressee cannot be established |
Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 07.10.2022) |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 20824578 Country of ref document: EP Kind code of ref document: A1 |