CN109416653B - Selecting unit tests for system analysis - Google Patents
Selecting unit tests for system analysis Download PDFInfo
- Publication number
- CN109416653B CN109416653B CN201780024091.4A CN201780024091A CN109416653B CN 109416653 B CN109416653 B CN 109416653B CN 201780024091 A CN201780024091 A CN 201780024091A CN 109416653 B CN109416653 B CN 109416653B
- Authority
- CN
- China
- Prior art keywords
- unit
- tests
- test
- unit tests
- unit test
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 292
- 238000004458 analytical method Methods 0.000 title description 5
- 238000000034 method Methods 0.000 claims abstract description 76
- 230000002452 interceptive effect Effects 0.000 claims abstract description 13
- 238000013507 mapping Methods 0.000 claims abstract description 4
- 230000006870 function Effects 0.000 claims description 42
- 238000003066 decision tree Methods 0.000 claims description 30
- 238000004891 communication Methods 0.000 claims description 2
- 230000000977 initiatory effect Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 14
- 238000010998 test method Methods 0.000 description 30
- 230000010354 integration Effects 0.000 description 8
- 238000012552 review Methods 0.000 description 6
- 238000003745 diagnosis Methods 0.000 description 5
- 230000004044 response Effects 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000008859 change Effects 0.000 description 2
- 238000009635 antibiotic susceptibility testing Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000012512 characterization method Methods 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000006855 networking Effects 0.000 description 1
- 239000010979 ruby Substances 0.000 description 1
- 229910001750 ruby Inorganic materials 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Computer Hardware Design (AREA)
- Debugging And Monitoring (AREA)
Abstract
The present invention provides a process for analyzing a system comprising a plurality of physical entities executing an interactive software application to provide a plurality of services. The process comprises the following steps: unit tests are determined for the software application, wherein at least a portion of the software application has been assigned more than one unit test, and if an application is unavailable, the unit tests assigned to the application fail. Mapping the determined unit tests to the plurality of services to build a data structure, wherein the data structure indicates which unit tests failed when a certain service is unavailable. Then, a subset of the mapped unit tests is executed to analyze availability of services in the system, wherein members of the subset are dynamically determined during execution based on whether a previous unit test in the subset failed or passed.
Description
Technical Field
The present invention relates to analyzing a system having a plurality of physical entities executing an interactive software application to provide a plurality of services. More particularly, the invention relates to selecting unit tests for analysis.
Background
A system, such as a cloud platform, having multiple physical entities executing an interactive software application may be implemented by including multiple networked computers. Each computer (also referred to as a "server") provides services having various functions to a client controlled by a user. As processing power, networking, and storage technologies have increased, these systems have become more powerful and more complex. Due to the increased complexity, there is an increasing demand for effective diagnosis of system faults.
A software application can be verified using a test suite comprising a plurality of unit tests that are developed as part of the software development process with the aim of targeted individual testing of the smallest testable component, also referred to as a unit, of the software application in conjunction with associated control data. The execution unit test can effectively check the integration condition of the code in the software development process, and can verify the modification of the code by a developer so as to ensure the quality of the code.
Disclosure of Invention
Unit testing in a system with multiple physical entities executing interactive software applications that provide multiple services typically does not provide any information about non-responsive or faulty services. Instead, a list of pass and fail tests is generated after the unit test is performed. The manifest can help locate software failures or discover problems with code modules, however, the manifest itself cannot be used to diagnose the failure service. In addition, there may be a large number of unit tests available, and as code libraries continue to grow, the number of unit tests may increase further. Thus, it can take a long time to perform all available cell tests.
Therefore, system operators typically customize tests to diagnose the faulty service. However, if the system is constantly evolving, frequently updated and modified, the customized test is often outdated. Thus, these tests are also continually modified/updated, e.g., each time a new software release is released. This approach is costly for the operator, who expends significant resources to keep service diagnostics exhaustive and timely. Therefore, there is a need for another less costly method of diagnosing a malfunctioning service in a system having multiple physical entities executing interactive software applications.
According to a first aspect of the present invention there is provided a computer readable medium having stored thereon computer readable instructions for analysing a system having a plurality of physical entities executing one or more interactive software applications to provide a plurality of services, the execution of the computer readable instructions causing the system to: shutting down a first service and performing selected unit tests to determine failed first unit tests due to unavailability of the first service; shutting down a second service and performing the selected unit test to determine a failed second unit test due to unavailability of the second service; and initiating the service and executing a subset of the first and second unit tests to analyze the system, wherein members of the subset are dynamically determined during execution based on whether a previous unit test in the subset failed or passed.
It is noted here that the term "physical entity" as used in the present description and claims refers in particular to a networked computer, wherein the computer may comprise a processor, a memory and a network interface. In addition, it should be noted that the term "interactive software application" as used in the present specification and claims especially refers to a software application that performs a certain operation using the other party. For example, one software application instructs another software application to perform an operation and waits for the operation to be performed.
Furthermore, it is to be noted that the term "service" as used in the present description and claims refers in particular to a procedure which a client may (spontaneously) invoke and which provides data, for example to the client or a third party, wherein the data is based on a request of the client. In addition, it should be noted that the terms "fail" and "pass" as used in the present specification and claims refer to, inter alia, a failure or a pass as determined by a unit test. That is, if a service is turned off and the unit test passes, the unit test cannot be used to determine whether the service is available. On the other hand, if a service is closed and the unit test fails, the unit test can be used to determine if the service is available.
Thus, available unit tests (i.e., unit tests developed during software development) may be used to diagnose unavailable services.
In a first possible implementation form of the first aspect, the unit test of execution corresponds to a node on a path in a decision tree, leaves in the decision tree indicating unavailable services in the system.
It is noted here that the term "decision tree" as used in the present description and claims refers in particular to a tree-like decision graph or model, as well as the possible outcome of said decision, related to whether a unit test failed or passed, and to whether a service is assumed to be available or not.
Therefore, only the relevant unit test, i.e., the unit test that must be performed to determine whether the service is available, is performed.
In a second possible implementation form of the first aspect, the nodes on the path are ordered according to a ratio of the number of unavailable services that can be detected by the corresponding unit test to an amount of resources required to perform the unit test.
It should be noted that, as used herein, the term "amount of resources required to perform the unit test" as used in the specification and the claims refers to, in particular, the amount of processing resources or processing time required for one unit test.
Since the unit test that generates more information/consumes less resources is preferentially performed, the overall performance can be improved.
In a third possible implementation form of the first aspect, the computer-readable medium stores computer-readable instructions, and execution of the computer-readable instructions causes the system to: assigning a relevance score to the function of the available unit test call; and deselecting available unit tests in the following cases: only the ignored function with the relevance score below the threshold is called; or only a function with a relevancy score greater than or equal to the threshold is called, which function is also called by other ones of the selected unit tests.
It should be noted that the term "function" as used in the present specification and claims refers to a supporting method for performing unit test.
Thus, unit tests may be screened to avoid performing irrelevant or redundant unit tests.
In a fourth possible implementation form of the first aspect, the relevancy score is based on a number of times the function is called in a file encoded by the unit test of the set of unit tests.
Therefore, the degree of correlation of the function can be determined based on the uniqueness of use of the function.
In a fifth possible implementation form of the first aspect, the more times the function is called in a file encoded by the unit test in the unit test set, the lower the relevancy score.
Thus, it can be assumed that frequently called functions are less important than less frequently called functions.
In a sixth possible implementation form of the first aspect, the computer-readable medium stores computer-readable instructions that, when executed, cause the system to select available unit tests that call only functions that are also called by other unit tests of the selected unit tests.
Therefore, selection of redundant cell tests may be avoided.
In a seventh possible implementation form of the first aspect, the computer readable medium stores computer readable instructions that, when executed, cause the system to select an available unit test having a performance score lower than other unit tests.
Thus, overall performance can be improved since a set of unit tests that call the same (related) function are grouped into a unit test that requires the least resources.
In an eighth possible implementation form of the first aspect, the computer-readable medium stores computer-readable instructions, and execution of the computer-readable instructions causes the system to: determining members of the subset from the redundant unit tests of the first and second unit tests based on performance scores assigned to the redundant unit tests, wherein failure patterns of the redundant unit tests for unavailable services are the same.
Thus, overall performance can be further improved since a set of redundant unit tests that produce the same service availability information when executed are grouped into a unit test that requires the least resources.
In a ninth possible implementation form of the first aspect, the performance score is based on a performance time of the redundant unit test.
Therefore, the overall system analysis execution time can be reduced.
In a tenth possible implementation form of the first aspect, the computer-readable medium is included in a computing device, the computing device further includes a processor, and the computing device is located in a communication network.
According to a second aspect of the invention, there is provided a method of analysing a system having a plurality of physical entities executing an interactive software application to provide a plurality of services. The method comprises the following steps: determining unit tests for the software application, wherein at least a portion of the software application has been assigned more than one unit test, and if an application is unavailable, the unit tests assigned to the application fail; mapping the determined unit tests to the plurality of services to build a data structure indicating which unit tests failed when a service is unavailable; and executing a subset of the mapped unit tests to analyze availability of services in the system, wherein members of the subset are dynamically determined during execution based on whether a previous unit test in the subset failed or passed.
Thus, as described above, available unit tests (i.e., unit tests developed during software development) may be used to diagnose unavailable services.
In a first possible implementation form of the second aspect, the unit test of execution corresponds to a node on a path in a decision tree, leaves in the decision tree indicating unavailable services in the system.
Therefore, as described above, only the relevant unit test needs to be performed, that is, the unit test must be performed to determine whether the service is available.
In a second possible implementation form of the second aspect, the nodes on the path are ordered according to a ratio of the number of unavailable services that can be detected by the corresponding unit test to the amount of resources required to perform the unit test.
Therefore, as described above, since the unit test that generates more information or consumes less resources is preferentially performed, the overall performance can be improved.
In a third possible implementation form of the second aspect, the determining a unit test for the software application further comprises: determining a function call to be executed by the available unit test; assigning a relevance score to the function; ignoring unit tests that only call functions with relevance scores below a threshold; and disregarding only unit tests that call functions having a relevancy score greater than or equal to the threshold, and which are also called by other ones of the determined unit tests.
Thus, as described above, unit tests may be screened to avoid performing irrelevant or redundant unit tests.
In a fourth possible implementation form of the second aspect, the relevancy score is based on a number of times the function is called in the available units test.
Therefore, as described above, the degree of correlation of the function can be determined based on the use uniqueness of the function.
In a fifth possible implementation form of the second aspect, determining unit tests for the software application comprises: determining a function call to be executed by the available unit test; and ignoring available unit tests that only call functions that are also called by other ones of the determined unit tests.
Therefore, as described above, selection of redundant cell tests may be avoided.
In a sixth possible implementation form of the second aspect, for unit tests that call the same function, unit tests with performance scores lower than other unit tests are ignored for deduplication.
Therefore, as described above, since a set of unit tests calling the same (related) function is classified as a unit test requiring the least resources, the overall performance can be improved.
In a seventh possible implementation form of the second aspect, the members of the subset are selected from a plurality of mapped unit tests based on performance scores assigned to the plurality of mapped unit tests, wherein failure patterns of the plurality of mapped unit tests for unavailable services are the same.
Thus, as described above, since a set of redundant unit tests that produce the same service availability information when executed are grouped into a unit test that requires the least resources, the overall performance can be further improved.
Drawings
FIG. 1 illustrates a block diagram of an exemplary system involved in the execution of a process;
FIG. 2 shows a flow chart of a first stage of the process;
FIG. 3 is an abstract syntax tree to be used in a first stage according to an example;
FIGS. 4a to 4c illustrate the usage of the abstract syntax tree shown in FIG. 3 for different target parameters;
FIG. 5 illustrates an example of a deselect redundant cell test;
FIG. 6 shows a flow chart of a second stage of the process;
FIG. 7 illustrates an exemplary mapping between unit tests and services resulting from the second stage;
FIG. 8 illustrates several sets of redundant isomorphic unit tests;
fig. 9 shows a decision tree based on the relationships established at the second stage.
Detailed Description
The following exemplary process may be implemented by software, hardware, or a combination thereof, with the purpose of efficiently diagnosing a system, such as a cloud platform, having multiple physical entities executing one or more interactive software applications to provide a variety of services by reusing unit tests used to test software applications during a software development phase.
The process may include three stages, as will be explained in more detail below at the time of description. In the first stage, the number of available unit tests can be reduced by deselecting redundant unit tests. It should be noted that when the redundant cell test is removed, irrelevant support methods can be determined and ignored. In a second phase, a relationship between the remaining unit tests and all services required for system operation may be determined, wherein the remaining unit tests may be screened based on the determined relationship. In a third phase, a decision tree may be generated to diagnose system faults. Test results show that in some cases it is sufficient to perform only 4% to 5% of the available unit tests to test the availability of all services.
Fig. 1 illustrates an exemplary system 10 involved in the execution of the process. The system 10 includes a plurality of physical entities 12 that execute interactive Software (SW) Applications (APPs) 14. The SW APP 14 may communicate with each other using a Hypertext Transfer Protocol (HTTP), a Remote Procedure Call (RPC), a token, a message queue, and the like. The interaction SW APP 14 may provide services 16, and these services 16 are critical to the operation of the system 10. For example, if one of the services 16 fails to function properly, the system 10 may be unable to process one or more (or even any) requests from the client 18.
The system 10 also includes a software development infrastructure 20. The software development infrastructure 20 may include one or more computers and software that a human software developer may use to implement, update, and maintain the interactive SW APP 14. For example, the software development infrastructure 20 may include a version control database 22. The version control database 22 may be stored in a computer and may include software code as well as a record of modifications of the SW APP 14. In particular, the version control database 22 may store all code related to the current implementation of the SW APP 14 at the physical entity 12.
In addition, the software development infrastructure 20 can include a code review module 24. The code review module 24 may provide a platform for reviewing the code of the SW APP 14. For example, a developer or operator may use the code review module 24 to accept or reject code modifications and provide feedback. Code review module 24 may provide a web-based Graphical User Interface (GUI) through which a developer or operator may review code. The code form may be one or more human-readable programming languages, such as Java, C #, HTML 5, JavaScript, R, Swift, C + +, PHP, Ruby, Python, and so forth.
Further, software development infrastructure 20 may include a code integration module 26. The code integration module 26 may be used to control the integration of code into a version control Database (DB) 22. For example, prior to integration, all available unit tests 28 may be performed to verify the integrity of the code. The unit test 28 may be used to test the smallest testable component, also referred to as a unit, of the SW APP 14 in combination with associated control data. For example, unit test 28 may send an HTTP request to physical entity 12 and receive an HTTP response from physical entity 12. It may determine whether unit test 28 passed or failed based on the response. If the unit test 28 fails, the corresponding code (and code associated therewith) may not be integrated while it is flagged for further review by the software developer. This may help to detect problems at an early stage, so that the SW APP 14 runs error-free.
The system 10 may also include a fault diagnosis unit 30 for determining whether the service 16 is operating properly. Such diagnostics may be performed periodically based on operator-determined timeframes and may supplement the tests performed by software development infrastructure 20 to discover problems not discovered when software development infrastructure 20 performs unit tests 28.
The fault diagnosis unit 30 may include a unit test manager 32. The unit test manager 32 may be used to opt out of irrelevant and/or redundant unit tests 28 (to diagnose the service 16 as a whole). Failure diagnostic unit 30 may also include a test service mapper 34. Test service mapper 34 may determine data indicating which unit failed the test when a service 16 is unavailable. Prior to this, the test service mapper 34 may send a signal, such as a Unix signal, to one or more physical entities 12, causing the one or more physical entities 12 to disable (stop/shut down) the service 16.
Once the service 16 is unavailable, the test service mapper 34 may send a request, such as an HTTP request, to one or more software applications 14. This may be performed repeatedly for all selected unit tests 2. Since each request requires one or more particular services 16 to be running, when a service 16 is unavailable, it can be determined which of the selected unit tests 28 failed. In particular, whether a certain unit test 28 passes or fails may be determined by examining (HTTP) responses of one or more physical entities 12. A passing unit test 28 may be classified as being unable to test the availability of the service 16, while a failing unit test 28 is classified as being able to test the availability of the service 16.
After determining which of the selected unit tests 28 are capable of (and can be used to) test that a particular service 16 is not available, a signal, such as a Unix signal, may be sent to one or more physical entities 12 causing the one or more physical entities 12 to re-enable (launch) the service 16. This may be repeated for all or a portion of the services 16. In addition, the dependencies between the selected unit tests 28 and the services 16 may be stored in a data structure, such as a matrix or array. These dependencies may indicate the ability of unit test 28 to determine whether a particular service 16 is functioning properly.
The fault diagnosis unit 30 may also include a fault detector 36. Fault detector 36 may use the data structure and construct a decision tree to select the most relevant unit tests 28 to diagnose services 16 in system 10. However, the decision tree may only be rebuilt when needed, i.e., if the set of selected unit tests 28 changes, the same decision tree may be used multiple times to diagnose the system service 16. For example, the operator may define scheduled diagnostics during selected periods of the day (periods of low system load, e.g., nighttime periods, set sleep periods for system 10, or when new code is to be integrated). For example, fault detector 36 may send a (HTTP) request to SW APP 14 and receive a (HTTP) response.
Analysis of the exchange of (HTTP) messages may result in a failure report indicating the unavailable service 16 (if any). If the system service 16 is not available, maintenance operations may be taken on it, for example, canceling the most recent change in code. Additionally, upon code integration, the report may be sent to the code integration module 26, and the code integration module 26 may determine whether to accept or reject the code change.
FIG. 2 shows a flow chart of a first stage of the above-described process 38, which may be performed by the unit test manager 32. Unit test manager 32 may receive as input a series of existing unit tests 28 written during the development phase and output a reduced number of unit tests 28 to test service mapper 34, where test service mapper 34 may perform the second phase of operations. As shown in FIG. 2, at 40, the unit test manager 32 may begin the first phase by: a source file/module is determined that includes code associated with the available unit tests 28.
At 42, a source file/module including code related to the available units tests 28 may be code analyzed and an Abstract Syntax Tree (AST) may be constructed. For example, the code may include unit test methods that may invoke other methods, which may be referred to hereinafter as support methods. The support methods may call various functions of the SW APP 14. To determine the unit test method and its supporting methods, AST can be constructed (see "abstract syntax trees" by Kuhn, Thomas and Olivier Thomann in Eclipse Corner arms 20 (2006)). An AST may be a characterization of the source code/module in the form of a tree, where nodes in the AST may represent builds (classes, unit test methods, or support methods) performed in the source code/module.
The AST may be traversed to determine the unit test method and its supporting methods. For each unit test method, a set of its supporting methods may be created. The AST can be constructed by a compiler in a series of steps as follows:
● scanning the location where the source code reads the character stream;
● grouping the characters to form a token; and
●, the token is converted into a grammar structure, which may be referred to as a parse tree.
The parse tree is then converted to an AST. It is noted herein that at least a portion of the programming languages already support the construction of ASTs. Py can be used to construct AST, for example, when using Python.
At 44, the constructed AST is screened. The supporting methods may be scored according to the relevance of the determined supporting methods. For example, if a method is applicable only to a particular language and may not play a direct role in the testing of the services 16 of the system 10, the supporting method may be selected. The support methods may be marked as "relevant" or "irrelevant" based on the Inverse Document Frequency (IDF). Therefore, the support methods with lower IDF, i.e., the support methods existing in multiple source files/modules, may be considered irrelevant. For example, the IDF supporting method (x) may be calculated according to the following equation:
idf (x) log (# source file/module/# source file/module including support method x) (1)
Thus, as described in more detail below in connection with the example shown in fig. 3, the support methods present in multiple source files/modules may have a lower IDF. Fig. 3 shows four modules. Module 1 contains the definitions of the Unit test methods U1, U2, and U3. Module 2 contains the definitions of the Unit test methods U4, U5, and U6. Module 3 contains the definitions of the Unit test methods U7, U8, and U9. Module 4 contains the definitions of the Unit test methods U10, U11, and U12. The following IDF can be determined from equation (1):
IDF(Assert)＝log(4/4)＝0
IDF(S1)＝log(4/1)＝log(4)
IDF(S2)＝log(4/2)＝log(2)
IDF(S3)＝log(4/1)＝log(4)
thus, S1, S2, and S3 may be marked as related, while "Assert" may be marked as not related and deselected. It is emphasized here that the threshold values under different frames may also be different. Thus, for best results, one may try to use different IDF thresholds and may choose one from them to remove most or all of the supported methods that are specific to a particular language only, but not or only a few of the commonly related user-defined methods.
With continued reference to FIG. 2, at 46, the supporting methods marked as "relevant" may be deduplicated. For example, if a subset of unit test methods call the same supporting method, this indicates that there are redundant unit test methods that can be deselected. The deselection operation may be performed based on a modified set overlay algorithm. The modified set coverage algorithm may select the minimum number of unit test methods such that the selected unit test method invokes all supporting methods invoked by any unit test method in the source file/module. For example, in FIG. 3, the subsets { U7, U8} and { U7, U9} in module 3 call all supported methods { S6, S7, S8} in the module. Thus, any one subset may be selected to invoke all supported methods. In addition, the set cover algorithm may select the subset with the shortest execution time.
The modified set covering algorithm may involve the following parameters:
● time: the time required to perform the unit tests in the subset. This is related to the "cost" in the set covering algorithm, which can help select the most efficient subset of unit test methods. For example, if more than one subset of unit test methods calls all supported methods in one source file/module, the subset with the shortest execution time may be selected.
● coverage rate: the minimum coverage (unit:%) that needs to be achieved by the support method. This means that the operator can select a unit test method based on the percentage of supported methods required. For example, in FIG. 3, there are three support methods { S6, S7, S8} in Module 3. If 100% coverage is to be achieved, the subset U7, U8 or U7, U9 is selected to cover all supported methods. To achieve 30% coverage, only { U7} needs to be selected, since only one third of the support methods need to be invoked.
The mathematical definition of the modified set cover algorithm may be:
inputting:
● support a set of methods: s ═ S1，s2，…，sn}
One of ● m unit test sets: u ═ U1，U2，…，Um} such that:
●
● expected coverage, c ∈ [0, 1]
● set of times required to perform each unit test: t ═ T1，t2，…，tmAre such that tiIs UiThe execution time of.
Modified set overlay algorithm may be used to determine the set
Such that:
●∑tjminimum size
●
●j∈J
In addition, the modified set overlay algorithm can be schematically represented by the following pseudo-code:
1.Umin←{}
2.
for all unit test method U do
One such unit test method was found: for each newly added support method, the cost α is lowest:
b.Umin←Umin∪U
3. output Umin。
The algorithm represented in pseudo code works as follows:
1.Uminrepresenting a collection of unit tests. The initial value is an empty set.
2. When the coverage condition is not met (i.e., the number of supported methods in the set is less than a desired value):
a. find such a unit test method U: and the cost alpha of each support method of the U is the lowest, wherein alpha is the ratio of the execution time of the unit test method to the number of the newly added support methods.
b. Adding unit test method U to UminIn (1).
3. Output Umin。
In this regard, fig. 4a shows the AST of module 3 after the irrelevant support method (Assert) is removed. The AST includes three unit test methods: u7, U8, and U9. Hereinafter, it is assumed that the execution time of U7 is 15 seconds, the execution time of U8 is 8 seconds, and the execution time of U9 is 9 seconds. If a modified set coverage algorithm is applied to the AST in fig. 4a to remove the redundant cell test method and achieve 100% desired coverage, all supporting methods must occur. In this case, as shown in fig. 4b, U7 and U8 are chosen because U7 and U8 cover all the different support methods and are the lowest cost. Thus, the modified set cover algorithm removes U9.
If the modified aggregate coverage algorithm is applied to the AST in fig. 4a to remove the redundant cell test method and achieve 50% desired coverage, only two of the three support methods need be performed (since 2/3> 50%). Both U8 and U9 include two different support methods. Since U8 costs the least, U8 was selected to select U7 and U9. It should be noted that the percentage values selected above are merely exemplary, and the operator may set the coverage rate according to actual needs.
Still referring to FIG. 2, at 48, the redundant unit test 28 in the source file/module may be removed. For example, as shown in FIG. 5, when module 2 and module 4 (of FIG. 3) are involved, the Unit test method U5 in module 2 calls the support method { S4, S5 }. Similarly, module 4 includes Unit test methods U10, U11, and U12, calling the support method sets { S4}, { S10, S4}, and { S12, S5}, respectively. The support method deduplication subsystem ensures that a subset of the unit test methods in module 4 call the support methods { S4, S5, S10, S12 }. This means that module 4 has covered all the supported methods called by U5 (i.e., { S4, S5 }). Therefore, U5 is redundant and can be deselected. The method can be applied to all unit test methods to remove redundant unit test methods in the entire source file/module.
The flow chart for the second phase of the above process may be completed by the test service mapper 34 as shown in fig. 6. Test service mapper 34 may receive data from unit test manager 32 regarding associated and deduplicated unit tests 28. At 52, test service mapper 34 may determine a relationship between received unit tests 28 and services 16. The relationship may be determined based on the ability of the unit test to detect whether a particular service 16 is functioning properly. The following steps may be repeatedly performed for all services 16:
● disable (stop/close) service s. This may simulate a situation where the service 16 fails.
● run a reduced number of sets of unit tests 28. The unit test 28 that depends on service s will fail. The unit test 28 that does not depend on the service s will pass. This result may be recorded.
● re-enables (starts) the service s.
Only one service 16 may be disabled at a time. If more than one service 16 is disabled at a time and one unit test fails, it cannot be determined which service 16 caused the unit test 28 to fail. Thus, the above steps may be repeated for all services 16 critical to the operation of the system 10. Relationships between unit tests 28 and services 16 may be established based on the results of the test runs and represented in a matrix, as shown in FIG. 7.
In fig. 7, each box represents the relationship between one unit test 28 and one service 16. The relationship may be represented by the expression R (Ui, service j), where Ui represents the unit test 28 and service j represents the service 16. R (Ui, service j) ═ 0 indicates that Ui failed when service j is unavailable. If Ui is executed in system 10 and passes successfully, then service j may be considered to be operating normally. This means that Ui depends on service j. However, if Ui fails, service j may not be available, although it cannot be determined one hundred percent that service j is not available in this case.
R (Ui, service j) ═ 1 indicates that Ui passes after service j is disabled. Therefore, Ui does not depend on service j. In other words, Ui cannot be used to test whether service j is available. Therefore, the execution of the unit test 28 cannot unambiguously determine whether the service j is available. For example, as shown in the matrix of fig. 7, if U5 fails, service 2, service 3, service 4, or service 5 may not be available, but it cannot be determined whether service 1 is available. However, if U5 passes, it indicates that service 2, service 3, service 4, and service 5 are available. These relationships can be used to build a decision tree in the third phase. Before this, in 54 of FIG. 6, the number of unit tests 28 may be further reduced by screening.
As shown in FIG. 7, each unit test 28 may be represented by a binary string pattern derived from rows representing services 16, with the exception of rows representing execution times. For example, U1 is represented by 01010, U2 is represented by 11101, and so on. Unit tests 28 in which the binary strings are identical may be referred to as isomorphic tests. That is, the relationship between isomorphic tests and services 16 is the same, and thus the availability of the same service 16 may be detected. Thus, the isomorphic test is a redundant test. To remove these redundant tests, the homogeneous unit tests 28 may be grouped and one unit test 28 selected from (e.g., the unit test 28 with the least execution time selected), as shown in FIG. 8.
For example, the execution time of the unit test U1 may be 10 seconds, the execution time of the unit test U2 may be 9 seconds, the execution time of the unit test U3 may be 13 seconds, the execution time of the unit test U4 may be 19 seconds, and the execution time of the unit test U5 may be 3 seconds. Thus, the unit tests U1, U5, and U2 may be selected.
Only the selected unit tests in the second phase are available for building the decision tree in the third phase. In the first two phases, the number of unit tests 28 is reduced and a relationship is established between the unit tests 28 and the services 16 provided by the system 10 to be diagnosed. In a third stage, these relationships can be used to build a decision tree. The decision tree may be used to detect a failed service 16 in normal operation of system 10, i.e., a failed service 16 that is concurrent with a service request by client 18.
The tree structure of the decision tree may test an attribute for each internal node thereof, one attribute value for each branch, and a category or label for each leaf node. The path from the root node to the leaf node may represent a classification rule. In this case, each internal node may represent a selected one of the unit tests 28 in the second phase. Each branch may represent the result of one unit test 28 and each leaf may represent one or more unavailable services 16.
The decision tree may be constructed by sequential fault diagnosis. This may be based on the attribute information gain to generate a decision tree. The information gain for each unit test can be calculated by the following formula:
testability of information gain for Unit test/execution time of Unit test (2)
Testability of Unit test-the number of untested services that the Unit test can test
Given the relationship shown in FIG. 9, the information gain can be calculated using equation (2), as follows:
information gain: u1 ═ 1/8;
information gain: u2 ═ 2/7;
information gain: u3 ═ 2/5;
information gain: u4 ═ 2/4.
The unit test 28 with the highest information gain may be selected and the node may be added to the decision tree in which the unit test 28 (here U4) is located. The node corresponding to the result of the unit test 28 may then be added to the decision tree. As described in the second stage, the unit test Ui has a relationship of R (Ui, service j) being 0, and if Ui passes, service j is considered to be operating normally. This relationship is used to build a decision tree. After the unit is selected for testing, two branches can be newly built in the decision tree. The branching of U4 may indicate that service 1 and service 2 are functioning properly. Thus, service 3 is considered as an undetected service for the next node. The information gains for U1, U2, and U3 are 1/8, 1/7, and 1/5, respectively, for the pass branch on U4.
Since the information gain of U3 is highest, U3 may be selected as the next node. Based on the above relationship, if U3 passes, service 1 and service 3 may be considered available. The previous node has determined that service 1 and service 2 are functioning properly. Therefore, all services are determined to be running normally by the node through the branches. However, if the test fails, it may indicate that any of the services 16 may not be available. Since the previous node has determined that service 1 and service 2 are normal, the failed branch of the node indicates that service 3 is in a failed state. This process may be repeated to build other parts of the decision tree shown in FIG. 9. The decision tree includes four attributes representing four unit tests 28 shown as internal nodes. The path from the root node to the leaf node represents the test order, where the leaf node represents the failure service 16. To diagnostically run the system 10, the decision tree may be traversed based on the results of the unit tests 28 until a leaf node is reached.
For example, the decision tree in FIG. 9 begins with U4. If the EU test U4 is performed and passes, then move to the next node including U3 and perform the EU test U3. If U3 fails at this time, it may indicate that service 3 is in a failed state; if unit test U3 passes, it may indicate that all services 16 are functioning properly. However, as previously mentioned, in some cases, there may be more than one possible service 16 in a failed state. For example, if U4 fails, the node including U3 may fail. A failure of U3 may result in a failure of the node comprising U2. Similarly, a U2 failure may result in a U1 failure. If U1 passes, it may indicate that service 1 or service 2 is not available or that neither service 1 nor service 2 is available.
In the example shown in fig. 9, generally, only log (n) (about) unit tests need to be run to detect the faulty service, and n unit tests need not be run, where n is the total number of unit tests after the isomorphic unit tests are removed. Thus, the decision tree enables a user to detect an unavailable service 16 without having to perform all available unit tests 28.
Claims (19)
1. A computer-readable medium storing computer-readable instructions for analyzing a system having a plurality of physical entities executing one or more interactive software applications to provide a plurality of services, execution of the computer-readable instructions causing the system to:
shutting down a first service and performing selected unit tests to determine failed first unit tests due to unavailability of the first service;
starting the first service;
shutting down a second service and performing the selected unit test to determine a failed second unit test due to unavailability of the second service; and
initiating the second service and executing a subset of the first and second unit tests to analyze the system, wherein members of the subset are dynamically determined during execution based on whether a previous unit test in the subset failed or passed.
2. The computer-readable medium of claim 1, wherein the first and second unit tests correspond to nodes on paths in a decision tree, leaves in the decision tree indicating unavailable services in the system.
3. The computer-readable medium of claim 2, wherein the nodes on the path are ordered according to a ratio of a number of unavailable services detectable by the corresponding unit test to an amount of resources required to perform the unit test.
4. A computer-readable medium as claimed in any one of claims 1 to 3, having stored thereon computer-readable instructions, execution of which causes the system to:
assigning a relevance score to the function of the available unit test call; and
available unit tests in the following cases are selected:
only the ignored function with the relevance score below the threshold is called; or
Only functions with relevance scores greater than or equal to the threshold are called, which functions are also called by other ones of the selected unit tests, wherein the selected unit test is a selected unit test among the available unit tests.
5. The computer-readable medium of claim 4, wherein the relevancy score is based on a number of times a function called by the available unit test in a file encoded by the available unit test in a unit test set is called.
6. The computer-readable medium of claim 5, wherein the more times the functions called by the available unit tests in the file encoded by the available unit tests in the unit test set are called, the lower the relevancy score.
7. A computer-readable medium as claimed in any one of claims 1 to 3, having stored thereon computer-readable instructions, execution of which causes the system to:
deselecting available unit tests that only call functions that are also called by other ones of the selected unit tests, wherein the selected unit test is a selected one of the available unit tests.
8. The computer-readable medium of claim 5 or 6 having stored thereon computer-readable instructions, wherein execution of the computer-readable instructions causes the system to:
the available unit tests with performance scores lower than the other unit tests are deselected.
9. The computer-readable medium of any one of claims 1-3 having stored thereon computer-readable instructions, wherein execution of the computer-readable instructions causes the system to:
determining members of the subset from redundant unit tests of the first and second unit tests based on performance scores assigned to the redundant unit tests, wherein failure patterns of the redundant unit tests for unavailable services are the same.
10. The computer-readable medium of claim 9, wherein the performance score is based on a time of execution of the redundant unit test.
11. A communications network comprising a computing device comprising a processor and a computer readable medium according to any one of claims 1 to 10.
12. A method of analyzing a system having a plurality of physical entities executing an interactive software application to provide a plurality of services, the method comprising:
determining unit tests for the software application, wherein at least a portion of the software application has been assigned more than one unit test, and if an application is unavailable, the unit tests assigned to the application fail;
mapping the determined unit tests to the plurality of services to build a data structure, wherein the data structure indicates which unit tests failed when a certain service is unavailable; and
executing a subset of the mapped unit tests to analyze availability of services in the system, wherein members of the subset are dynamically determined during execution based on whether a previous unit test in the subset failed or passed.
13. The method of claim 12, wherein the unit test of execution corresponds to a node on a path in a decision tree, wherein leaves in the decision tree indicate unavailable services in the system.
14. The method of claim 13, wherein the nodes on the path are ordered according to a ratio of a number of unavailable services detectable by the corresponding unit test to an amount of resources required to perform the unit test.
15. The method of any of claims 12 to 14, wherein determining a unit test for the software application further comprises:
determining a function call to be executed by the available unit test;
assigning a relevance score to the function;
ignoring unit tests that only call functions with relevance scores below a threshold; and
ignoring unit tests that only call functions having a relevancy score greater than or equal to the threshold, and which are also called by others of the determined unit tests, wherein the unit test determined for the software application is the unit test determined among the available unit tests.
16. The method of claim 15, wherein the relevancy score is based on a number of times the function called by the available unit test is called in the available unit test.
17. The method of any one of claims 12 to 14, wherein the unit test determined for the software application is a unit test determined among available unit tests, and wherein determining a unit test for the software application comprises:
determining a function call to be executed by the available unit test; and
ignoring available unit tests that only call functions that are also called by other ones of the determined unit tests.
18. The method of claim 15, wherein for unit tests that call the same function, unit tests with performance scores lower than other unit tests are ignored for deduplication.
19. The method of any of claims 12 to 14, wherein the members of the subset are selected from a plurality of mapped unit tests based on performance scores assigned to the plurality of mapped unit tests, wherein failure patterns of the plurality of mapped unit tests for the unavailable service are the same.
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| PCT/EP2017/060360 WO2018202275A1 (en) | 2017-05-02 | 2017-05-02 | Unit test selection for system analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109416653A CN109416653A (en) | 2019-03-01 |
| CN109416653B true CN109416653B (en) | 2020-11-27 |
Family
ID=58668888
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201780024091.4A Active CN109416653B (en) | 2017-05-02 | 2017-05-02 | Selecting unit tests for system analysis |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN109416653B (en) |
| WO (1) | WO2018202275A1 (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9032361B2 (en) * | 2011-12-15 | 2015-05-12 | Tata Consultancy Services Limited | Agile unit and regression testing framework for domain specific languages |
| CN104899141A (en) * | 2015-06-05 | 2015-09-09 | 北京航空航天大学 | Test case selecting and expanding method facing network application system |
-
2017
- 2017-05-02 CN CN201780024091.4A patent/CN109416653B/en active Active
- 2017-05-02 WO PCT/EP2017/060360 patent/WO2018202275A1/en active Application Filing
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9032361B2 (en) * | 2011-12-15 | 2015-05-12 | Tata Consultancy Services Limited | Agile unit and regression testing framework for domain specific languages |
| CN104899141A (en) * | 2015-06-05 | 2015-09-09 | 北京航空航天大学 | Test case selecting and expanding method facing network application system |
Non-Patent Citations (1)
| Title |
|---|
| 软件回归测试用例选取方法与研究;王小丽;《空间控制技术与应用》;20100630;第36卷(第3期);第1-4页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2018202275A1 (en) | 2018-11-08 |
| CN109416653A (en) | 2019-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Yilmaz et al. | Covering arrays for efficient fault characterization in complex configuration spaces | |
| Xu et al. | POD-Diagnosis: Error diagnosis of sporadic operations on cloud applications | |
| US8990778B1 (en) | Shadow test replay service | |
| US8230401B2 (en) | Performing regression tests based on test case effectiveness | |
| Wu et al. | Performance diagnosis in cloud microservices using deep learning | |
| US7890808B2 (en) | Testing software applications based on multiple data sources | |
| US10185650B1 (en) | Testing service with control testing | |
| US10452435B2 (en) | Dynamic build pipeline execution | |
| WO2012073686A1 (en) | Dependability maintenance device, dependability maintenance system, malfunction supporting system, method for controlling dependability maintenance device, control program, computer readable recording medium recording control program | |
| King et al. | Towards self-testing in autonomic computing systems | |
| King et al. | An integrated self-testing framework for autonomic computing systems. | |
| US11194703B2 (en) | System testing infrastructure for analyzing soft failures in active environment | |
| US11169910B2 (en) | Probabilistic software testing via dynamic graphs | |
| TWI626538B (en) | Infrastructure rule generation | |
| Marin et al. | Towards testing future web applications | |
| US11609842B2 (en) | System testing infrastructure for analyzing and preventing soft failure in active environment | |
| US20040148590A1 (en) | Hierarchical test suite | |
| Camacho et al. | Chaos as a Software Product Line—a platform for improving open hybrid‐cloud systems resiliency | |
| Rava et al. | A review on prioritization techniques in regression testing | |
| CN109416653B (en) | Selecting unit tests for system analysis | |
| Long et al. | Coordinated collaborative testing of shared software components | |
| Jebbar et al. | Architecture for the automation of live testing of cloud systems | |
| Zhang et al. | Fault localization based on dynamic slicing via JSlice for Java programs | |
| WO2021171117A1 (en) | Executing tests in deterministic order | |
| Moraes et al. | UML-based modeling of robustness testing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| TR01 | Transfer of patent right |
Effective date of registration: 20220211 Address after: 550025 Huawei cloud data center, jiaoxinggong Road, Qianzhong Avenue, Gui'an New District, Guiyang City, Guizhou Province Patentee after: Huawei Cloud Computing Technologies Co.,Ltd. Address before: 518129 Bantian HUAWEI headquarters office building, Longgang District, Guangdong, Shenzhen Patentee before: HUAWEI TECHNOLOGIES Co.,Ltd. |
|
| TR01 | Transfer of patent right |