CN113568839A

CN113568839A - Method, device, equipment and medium for software testing and statistical test coverage rate

Info

Publication number: CN113568839A
Application number: CN202110886810.1A
Authority: CN
Inventors: 杨京
Original assignee: Jingdong Technology Holding Co Ltd
Current assignee: Jingdong Technology Holding Co Ltd
Priority date: 2021-08-03
Filing date: 2021-08-03
Publication date: 2021-10-29

Abstract

The present disclosure relates to a method, an apparatus, a device and a medium for software testing and statistical test coverage, wherein the software testing method comprises: acquiring a difference code, wherein the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software; compiling and instrumentation the difference codes to obtain an executable file with a probe, wherein the probe is used for recording track information experienced when the byte codes in the executable file execute a program; testing the executable file based on a preset test case, and obtaining track information experienced by the bytecode in the testing process based on the probe record; and comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result about the difference code.

Description

Method, device, equipment and medium for software testing and statistical test coverage rate

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for testing software and counting test coverage.

Background

In the delivery process of the software, the software quality is guaranteed through means such as unit test, function test, performance test, compatibility test, interface automation test and the like. Regardless of the test method, it is important to determine whether the test case design is accurate and comprehensive. At present, only a normal path is often used for code walk-through in unit testing, all abnormal scenes cannot be covered by functional testing, and serious production faults can be caused by missing testing.

In the course of implementing the disclosed concept, the inventors found that there are at least the following technical problems in the related art: in the related technology, most of the full-scale codes of the target software are tested based on the test cases, so that test coverage rate information of the test cases for the target software is obtained, and whether the test case design is accurate and comprehensive is further measured according to the test coverage rate information, the full-scale code testing mode has the problem of low testing efficiency on one hand, and on the other hand, for various types of test cases, the reliability of the test is difficult to evaluate, and whether the test cases cover all code logics cannot be determined, for example, a black box testing method is usually adopted for functional testing, and test missing can occur for code logics which are not covered; the unit test usually only needs a normal path to carry out code walk, and also has the problem that all code logics cannot be covered.

Disclosure of Invention

To solve the above technical problem or at least partially solve the above technical problem, embodiments of the present disclosure provide a method, an apparatus, a device, and a medium for software testing and statistics of test coverage.

In a first aspect, embodiments of the present disclosure provide a method for testing software. The software testing method comprises the following steps: acquiring a difference code, wherein the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software; compiling and instrumentation the difference codes to obtain an executable file with a probe, wherein the probe is used for recording track information experienced when the byte codes in the executable file execute a program; testing the executable file based on a preset test case, and obtaining track information experienced by the bytecode in the testing process based on the probe record; and comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result about the difference code.

According to an embodiment of the present disclosure, comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result about the difference code, including: analyzing track information of the bytecode experience to obtain information of the candidate source code covered in the process of running the test case, wherein the information represents position information of the executed source code and the content of the correspondingly executed source code; screening target information covering the updated source code from the information of the candidate source code; determining the number of the covered updating source codes according to the target information; and comparing the number of the covered updating source codes with the structure of the updating source codes to obtain the test coverage rate.

According to an embodiment of the present disclosure, the compiling and instrumentation of the difference code to obtain an executable file with a probe includes: determining the instrumentation mode of the executable file to be an offline instrumentation mode or an runtime instrumentation mode based on the type of the test case; under the condition that the instrumentation mode is an offline instrumentation mode, inserting a probe in the differential code compiling process, wherein the compiled byte code structure with the probe forms the executable file with the probe; under the condition that the instrumentation mode is an operation instrumentation mode, compiling the difference codes to obtain an executable file; and in the process of deploying and loading the executable file in the test environment, inserting a probe into the byte code of the executable file so as to obtain the executable file with the probe.

According to an embodiment of the present disclosure, the determining, based on the type of the test case, that the instrumentation mode of the executable file is an offline instrumentation mode or a runtime instrumentation mode includes: determining that the instrumentation mode of the executable file is an offline instrumentation mode under the condition that the type of the test case is a unit test; and under the condition that the type of the test case is interface test, function test, performance test, compatibility test or automation test, determining that the instrumentation mode of the executable file is a runtime instrumentation mode.

According to an embodiment of the present disclosure, acquiring the difference code includes: receiving identification information of target version software and historical version software input by a user; positioning to the target version software and the historical version software according to the identification information; analyzing the target version software and the historical version software respectively to obtain a first analysis tree related to the target version software and a second analysis tree related to the historical version software; and comparing and analyzing the first analysis tree and the second analysis tree to obtain an updated source code of the target version software compared with the updated source code which is newly added or modified compared with the historical version software, wherein the updated source code is the difference code.

According to an embodiment of the present disclosure, the software testing method further includes: updating the preset test case according to the test coverage rate result of the difference code, so that the updated test case combination meets the preset requirement on the test coverage rate of the difference code; wherein, the updating mode comprises: one or more test cases are newly added on the basis of the preset test case, or the preset test case is modified to obtain a modified test case, and the updated test case combination correspondingly comprises: the test method comprises the steps of presetting test cases, adding new test cases or modifying the test cases.

In a second aspect, embodiments of the present disclosure provide a method of statistical test coverage. The method for counting the test coverage rate comprises the following steps: receiving selection information of a user on a full test or an incremental test; according to the selection information, carrying out corresponding code testing based on a testing coverage rate tool, and outputting a corresponding full code testing coverage rate report or a difference code testing coverage rate report; wherein, in the case that the selection information is an incremental test, the test coverage tool performs a code test by executing the software test method, so as to obtain a test coverage result about the difference code, and generates a difference code test coverage report according to the test coverage result of the difference code.

According to the embodiment of the disclosure, the report output component of the test coverage rate tool is provided with an unnecessary filling parameter; in the case that the user selects the incremental test, the difference code is transmitted to the report output component as the unnecessary filling parameter, so that the report output component generates a difference code test coverage report according to the test coverage result of the difference code; in the case where the user selects the full test, the above-mentioned unnecessary padding parameter is empty.

In a third aspect, embodiments of the present disclosure provide an apparatus for software testing. The device for testing software comprises: the device comprises a difference code acquisition module, a compiling and instrumentation module, a test module and a test coverage analysis module. The difference code obtaining module is used for obtaining a difference code, and the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software. And the compiling and instrumentation module is used for compiling and instrumentation the difference codes to obtain an executable file with the probe. The probe is used for recording track information experienced when the byte codes in the executable file execute the program. The test module is used for testing the executable file based on a preset test case and obtaining track information experienced by the bytecode in the test process based on the probe record. The test coverage analysis module is used for comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result related to the difference code.

The above apparatus for software testing further comprises: and a test case updating module. The test case updating module is used for updating the preset test case according to the test coverage rate result of the difference code, so that the updated test case combination can make the test coverage rate of the difference code meet the preset requirement. Wherein, the updating mode comprises: one or more test cases are newly added on the basis of the preset test case, or the preset test case is modified to obtain a modified test case, and the updated test case combination correspondingly comprises: the test method comprises the steps of presetting test cases, adding new test cases or modifying the test cases.

In a fourth aspect, embodiments of the present disclosure provide an apparatus for statistical test coverage. The device for counting the test coverage rate comprises the following steps: an information receiving module and a test coverage tool. The information receiving module is used for receiving selection information of a user on a full test or an incremental test. And the test coverage rate tool is used for carrying out corresponding code test according to the selection information and outputting a corresponding full code test coverage rate report or a difference code test coverage rate report. Wherein, in the case that the selection information is an incremental test, the test coverage tool is configured to perform the software test method to perform a code test, so as to obtain a test coverage result about the difference code, and is configured to generate a difference code test coverage report according to the test coverage result of the difference code.

In a fifth aspect, embodiments of the present disclosure provide an electronic device. The electronic equipment comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus; a memory for storing a computer program; and the processor is used for realizing the software testing method or the statistical testing coverage rate method when executing the program stored in the memory.

In a sixth aspect, embodiments of the present disclosure provide a computer-readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the method for testing software or the method for counting test coverage as described above.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure at least has part or all of the following advantages:

the method comprises the steps of compiling and instrumentation the difference code to obtain an executable file containing the probe, testing the executable file based on a preset test case, and obtaining a test coverage rate result related to the difference code by comparing and analyzing track information experienced by the byte code in the probe executable file with a structure of the updated source code in the testing process based on track information experienced by the byte code in the probe executable file, so that the coverage condition of the preset test case on a part of the newly added and/or modified updated source code can be determined, and corresponding test cases can be supplemented aiming at logic which is not completely covered in the updated source code subsequently, thereby improving the testing efficiency of software and improving the reliability of evaluating the test coverage degree.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings needed to be used in the description of the embodiments or the related art will be briefly described below, and it is obvious for those skilled in the art to obtain other drawings without inventive exercise.

FIG. 1A schematically illustrates an exemplary system architecture for the method and apparatus for software testing suitable for use with embodiments of the present disclosure;

FIG. 1B schematically illustrates another exemplary system architecture for the method and apparatus for software testing suitable for use with embodiments of the present disclosure;

FIG. 2 schematically illustrates a flow diagram of a method of software testing according to an embodiment of the present disclosure;

fig. 3 schematically shows a detailed implementation flowchart of operation S201 according to an embodiment of the present disclosure;

fig. 4 schematically shows a detailed implementation flowchart of operation S202 according to an embodiment of the present disclosure;

fig. 5 schematically shows a detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure;

FIG. 6 schematically shows a flow chart of a method of software testing according to another embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of a method of statistical test coverage in accordance with an embodiment of the present disclosure;

FIG. 8 schematically illustrates a block diagram of an apparatus for software testing according to an embodiment of the present disclosure;

FIG. 9 schematically illustrates a block diagram of an apparatus for statistical test coverage in accordance with an embodiment of the present disclosure; and

fig. 10 schematically shows a block diagram of an electronic device provided in an embodiment of the present disclosure.

Detailed Description

Common open source tools within the industry today (e.g., jacoco, an open source Java code coverage tool) support test coverage capture of only the full amount of code. On one hand, the test efficiency is low in the full code test mode, and on the other hand, for various types of test cases, the reliability of the test is difficult to evaluate, and whether all code logics are covered by the test cases cannot be determined.

In view of this, embodiments of the present disclosure provide a method, an apparatus, a device, and a medium for software testing and statistical testing coverage. The software testing method comprises the following steps: acquiring a difference code, wherein the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software; compiling and instrumentation the difference codes to obtain an executable file with a probe, wherein the probe is used for recording track information experienced when the byte codes in the executable file execute a program; testing the executable file based on a preset test case, and obtaining track information experienced by the bytecode in the testing process based on the probe record; and comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result about the difference code.

To make the objects, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

Fig. 1A schematically illustrates an exemplary system architecture suitable for use with the software testing methods and apparatus of embodiments of the present disclosure. FIG. 1B schematically illustrates another exemplary system architecture suitable for use with the methods and apparatus for software testing of embodiments of the present disclosure.

Referring to fig. 1A, an exemplary system architecture 100 suitable for use with the methods and apparatus for software testing of embodiments of the present disclosure includes:

terminal devices

101, 102, 103, a network 104 and a server 105. The network 104 serves as a medium for providing communication links between the

terminal devices

101, 102, 103 and the server 105. Network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, to name a few.

A user (e.g., a software tester) may use the

terminal devices

101, 102, 103 to interact with the server 105 over the network 104 to send software test requests, receive corresponding software test results, and so on. The

terminal devices

101, 102, 103 may have a software development system/tool installed thereon.

The

terminal devices

101, 102, 103 may be various electronic devices having a display screen and capable of supporting web browsing, such as electronic devices including but not limited to tablet computers, notebook computers, desktop computers, and the like.

The server 105 may be a server that provides various services, such as a server that provides test services to software developed by users using the

terminal devices

101, 102, 103 (for example only). The background management server can analyze and process the recorded data of the software in the testing process, and feed back the processing result (such as the testing result of the software) to the terminal device.

Referring to fig. 1B, another exemplary distributed system architecture 110 suitable for use with the method and apparatus for software testing of embodiments of the present disclosure includes: a web page end 111, a client 112, and a software development system (or may also be referred to as a software development platform) 113 deployed on the web page end 111.

The web page end 111 may be various electronic devices having a display screen and capable of supporting web page browsing.

The client 112 is a device capable of performing data communication with the web page 111 through a network, and is capable of providing distributed services, such as data storage and data processing, for the web page 111.

The software development system 113 may be a DevOps platform (a one-stop software development management platform), which distributes traffic to different web servers 111 and one or more corresponding clients 112 providing services through load balancing.

It should be noted that the method for testing software provided by the embodiment of the present disclosure may be generally executed by the server 105, or may also be executed by the client 112 with a certain computing capability, and the

terminal device

101, 102, 103 may also perform testing of software locally. Accordingly, the software testing apparatus provided by the embodiment of the present disclosure may be generally disposed in the server 105, or on the client 112 with certain computing capability, or on the

terminal device

101, 102, 103. The method for testing software provided by the embodiments of the present disclosure may also be performed by a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105. Accordingly, the software testing apparatus provided by the embodiment of the present disclosure may also be disposed in a server or a server cluster different from the server 105 and capable of communicating with the

terminal devices

101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks, and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

A first exemplary embodiment of the present disclosure provides a method of software testing.

FIG. 2 schematically shows a flow diagram of a method of software testing according to an embodiment of the present disclosure.

Referring to fig. 2, a method for testing software provided by an embodiment of the present disclosure includes the following operations: s201, S202, S203 and S204. The above operations S201 to S204 may be performed by the server 105 or the client 112 for providing a distributed service (cloud service), and may also be performed locally in the terminal device.

In operation S201, a difference code is obtained, where the difference code is an updated source code of a new and/or modified target version software compared to a historical version software.

In operation S202, the difference code is compiled and instrumented to obtain an executable file with a probe, where the probe is used to record track information that the bytecode in the executable file experiences when executing the program.

In operation S203, the executable file is tested based on a preset test case, and track information experienced by the bytecode during the test process is obtained based on the probe record.

In operation S204, the trace information experienced by the bytecode is compared with the structure of the updated source code, and a test coverage result about the difference code is obtained.

Source code is human-readable text written in a particular programming language, the goal of which is to set accurate rules and specifications for computers that can be converted to machine language. Thus, source code is the basis for programs and web sites. Source code may exist in each piece of software that executes according to programming in the source code, often in the form of a text file.

Each time a software development or upgrade is updated on a historical version of the software, there are differences between different versions or branches of software code.

In operation S201, source code of a target version of software updated on the basis of a historical version of software is obtained, and the updated source code includes: the types of the added source code, the deleted source code, the modified source code and the like use the added and/or modified source code (the source code with the change of the former version and the latter version is described as the updated source code) as the difference code. The formats of the two versions of software are, for example, files with suffixes.

In operation S202, the difference code is in a source code form, and an executable file with a probe may be obtained by compiling and instrumentation the difference code, for example, java source code is compiled to obtain a file with suffix class, which is composed of bytecodes, and a probe with a monitoring function is added to the bytecodes corresponding to the source code. The Java class file is a binary file that can run on any hardware platform and operating system that supports a Java virtual machine.

In operation S203, the preset test case is a test case for testing the historical version software and the target version software. And presetting a corresponding test case according to the requirement of the actual software test. The test case may be one or more of a combination of unit test, interface test, functional test, performance test, compatibility test, or automation test.

Test coverage is a measure of the degree of test completion, and it usually measures the execution of test cases according to some coverage criteria to determine whether the test is executed sufficiently. In particular, test coverage may characterize whether test cases really completely cover various possibilities in the application code and how much code was executed when the test cases were run.

For example, by analyzing the track information of the bytecode experience, the information of the candidate source code covered in the process of running the test case is obtained, and the information actually characterizes which source codes of which lines are executed, that is, the position information of the executed source code and the content of the corresponding executed source code are characterized. These candidate source codes cover code in the difference code and other code in the target version software except the difference code, and only the part of information covering the difference code (i.e. the above-mentioned update source code) is concerned in the test coverage result of the difference code. Therefore, target information covering the difference code is screened out from the information of the candidate source code subsequently; determining the number of covered difference codes according to the target information of the covered difference codes; and comparing the number of the covered difference codes with the structure of the updated source code to obtain the test coverage rate.

Based on the operations S201 to S204, the difference code is compiled and instrumented to obtain an executable file including a probe, the executable file is tested based on a preset test case, and during the test process, based on track information experienced by the bytecode in the probe executable file, the track information experienced by the bytecode is compared with the structure of the updated source code to obtain a test coverage result related to the difference code, so that the coverage of the preset test case on the newly added and/or modified updated source code can be determined, and subsequently, corresponding test cases can be supplemented for logic that is not completely covered in the updated source code, so that the test efficiency of software can be improved, and the reliability of evaluating the test coverage degree can be improved.

Fig. 3 schematically shows a detailed implementation flowchart of operation S201 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, referring to fig. 3, operation S201 of acquiring a difference code includes the following sub-operations: s2011, S2012, S2013 and S2014.

At sub-operation S2011, identification information of the target version software and the historical version software input by the user is received.

The user here may be a person having a software testing requirement, for example a software tester. The identification information includes: the repository address and code branch version number, or transaction commit id (commit id). For example, the target version software and the historical version software may be stored in a database, and the target version software and the historical version software in the database may be located and obtained according to the identification information.

In sub-operation S2012, the target version software and the historical version software are located according to the identification information.

In sub-operation S2013, the target version software and the historical version software are respectively parsed, and a first parse tree related to the target version software and a second parse tree related to the historical version software are obtained.

In sub-operation S2014, the first parse tree and the second parse tree are compared and analyzed to obtain an updated source code of the target version software, which is the difference code, compared with the updated source code of the historical version software after being added or modified.

For example, the sub-operations S2011 to S2014 may be implemented based on a version control system, which may be a Git (distributed version control system) or an SVN (Subversion, which is an open source version control system).

JGit is a Java library for operation git that supports the use of code operation git to obtain the differences between a specified branch and the master. Illustratively, in the case where the encoding language is java, the sub-operations S2011 to S2014 may be implemented by JGit. Jgit is mainly used to obtain code from git and to obtain files where changes exist. The java parser-core is a java parsing class, and can parse the class file into a tree shape, so that the difference class can be conveniently obtained, and the difference code can be obtained.

Fig. 4 schematically shows a detailed implementation flowchart of operation S202 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, as shown in fig. 4, the operation S202 of compiling and instrumentation the difference code to obtain the executable file with the probe includes the following sub-operations: s2021, S2022a, S2022b and S2023 b.

In sub-operation S2021, based on the type of the test case, it is determined that the instrumentation mode of the executable file is an offline instrumentation mode or a runtime instrumentation mode.

In sub-operation S2022a, in a case where the instrumentation mode is an offline instrumentation mode (offline), a probe is inserted in the differential code compiling process, and the compiled bytecode structure with the probe constitutes the executable file with the probe.

In sub-operation S2022b, when the instrumentation mode is the runtime instrumentation mode (on-the-fly), the difference code is compiled to obtain an executable file.

In sub-operation S2023b, during the process of deploying and loading the executable file in the test environment, a probe is inserted into the bytecode of the executable file, so as to obtain the executable file with the probe.

Interface testing is a type of testing of interfaces between components of a test system. The interface test is mainly used for detecting interaction points between external systems and between internal subsystems. The main emphasis of the test is to check the exchange of data, transfer and control management processes, and mutual logic dependency relationship between systems.

Functional testing, also known as behavioral testing, tests the characteristics and operational behavior of a product to determine that they meet design requirements based on product characteristics, operational descriptions, and user solutions.

The performance test is to simulate various normal, peak and abnormal load conditions through an automatic test tool to test various performance indexes of the system. Both load tests and pressure tests belong to the performance tests, and both can be performed in combination. The performance of the system under various working loads is determined through load tests, and the aim is to test the change of various performance indexes of the system when the load is gradually increased. Stress testing is a test that achieves the maximum level of service that a system can provide by determining the bottleneck or unacceptable performance point of a system. The purpose of the application performance test is to examine the performance of the client application, and the entrance of the test is the client.

The software compatibility test refers to checking whether the software can be interacted and shared correctly. Testing whether software can cooperate with each other becomes increasingly important as users demand from the ability to share data among various types of software and the ability to utilize space to execute multiple programs simultaneously. The goal of the software compatibility testing effort is to ensure that the software interacts in the way the user desires. There are generally four types of compatibility: the forward compatibility and backward compatibility, compatibility among different versions, standards and specifications, and data sharing compatibility.

The automated testing refers to designing an automated testing case (requirement specification in the software process) by analyzing testing requirements (requirement analysis in the software process) by using an automated testing tool (equivalent to a software development tool), so as to build a framework of the automated testing (summary design in the software process), design and write an automated script (detailed design and coding), test the correctness of the script, and complete the set of testing script (namely application software with main functions of testing).

The test types of the automated test may include: white box test, black box test, etc. The automated testing methods used by different test types are different, the white box test mainly aims at the unit test at the code level, and the black box test mainly aims at the verification test at the functional level and the system level.

And determining corresponding pile inserting modes according to different test case types, and further performing injection operation of the probe at different times. Take the test coverage tool Jacoco as an example, which supports two instrumentation modes of on-the-fly and offline simultaneously. In the On-the-fly mode, a Java Virtual Machine (JVM) specifies a specific jar file through a Java agent parameter to start Instrumentation (Instrumentation is an interface provided by Java from the JVM, and provides a series of methods for viewing and operating Java Class definitions, such as modifying Class bytecodes, adding jar files to the classpath of the classLoader, and the like.

In the offline mode, a file is firstly inserted before testing, then a class or jar packet inserted with the file is generated, after the class and jar packet inserted with the file is tested, dynamic coverage information is generated to the file, and finally the coverage information is processed in a unified mode, and a report is generated.

Fig. 5 schematically shows a detailed implementation flowchart of operation S204 according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, referring to fig. 5, the operation S204 of comparing the trace information experienced by the bytecode with the structure of the updated source code to obtain a test coverage result about the difference code includes the following sub-operations: s2041, S2042, S2043, and S2044.

In sub-operation S2041, the track information experienced by the byte code is analyzed to obtain information of the candidate source code covered in the process of running the test case, where the information represents the position information of the executed source code and the content of the source code executed correspondingly.

In sub-operation S2042, target information covering the updated source code is screened from the information of the candidate source code.

In suboperation S2043, the number of updated source codes to be overwritten is determined according to the target information.

In sub-operation S2044, a test coverage is obtained by comparing the number of the covered update source codes with the structure of the update source codes.

Based on the sub-operations S2041 to S2044, the target information of the updated source code can be quickly located to be covered, and the test coverage degree of the preset test case for the difference code is obtained.

FIG. 6 schematically shows a flow chart of a method of software testing according to another embodiment of the present disclosure.

Referring to fig. 6, the method for testing software according to the embodiment of the present disclosure includes, in addition to the above operations S201 to S204, an operation S601: and updating the preset test case according to the test coverage rate result of the difference code, so that the updated test case combination meets the preset requirement on the test coverage rate of the difference code. The operation S601 and the operations S201 to S204 may form a plurality of iterative loop processes, and in one iterative process, the operation of updating the test case is executed once after the operations S201 to S204 are executed; and then, performing next iteration based on the updated test case combination as a preset test case.

Wherein, the updating mode comprises: one or more test cases are newly added on the basis of the preset test case, or the preset test case is modified to obtain a modified test case, and the updated test case combination correspondingly comprises: the test method comprises the steps of presetting test cases, adding new test cases or modifying the test cases.

In one embodiment, the updating is performed by adding a new test case. The above operation S601 includes: determining the uncovered update source code in the update source code (difference code) according to the test coverage rate result of the difference code; constructing a new test case about the uncovered update source code; verifying whether the test coverage rate of the newly added test case on the uncovered updated source code meets the preset requirement; and under the condition that the test case does not meet the preset requirement, continuously constructing a newly added test case until the verification result meets the preset requirement.

In another embodiment, the original test case is modified to update. The above operation S601 includes: determining the uncovered update source code in the update source code (difference code) according to the test coverage rate result of the difference code; constructing a modified test case; verifying whether the test coverage rate of the change test case on the uncovered update source code and the covered update source code meets the preset requirement; and under the condition that the test case does not meet the preset requirement, continuously constructing and changing the test case until the verification result meets the preset requirement. In the scene of modifying the original test case, the original coverage condition may change, so the actual coverage condition of the modified test case for all the updated source codes needs to be verified at the same time.

A second exemplary embodiment of the present disclosure provides a method of statistical test coverage.

Fig. 7 schematically illustrates a flow chart of a method of statistical test coverage in accordance with an embodiment of the present disclosure.

Referring to fig. 7, a method for counting test coverage provided by an embodiment of the present disclosure includes the following operations: s701 and S702.

In operation S701, selection information of a full test or an incremental test by a user is received.

In operation S702, a corresponding code test is performed based on the test coverage tool according to the selection information, and a corresponding full code test coverage report or a differential code test coverage report is output.

The above operation S702 includes two parallel branches: operations S702a and S702 b.

In operation S702a, in the case that the selection information is an incremental test, the test coverage tool performs a code test by performing the software test as described above, so as to obtain a test coverage result about the difference code, and generates a difference code test coverage report according to the test coverage result of the difference code.

In operation S702b, in the case that the selection information is a full test, the test coverage tool performs a full code test to obtain a test coverage result for the full code, and generates a difference code test coverage report according to the test coverage result for the full code.

The report is in a visual form, and the file format is as follows: html, xml, and the like.

Generating a difference code test coverage report by the Jacoco of the secondary development, and regarding the Jacoco secondary development transformation logic as follows: adding a non-essential filling parameter (a non-essential filling parameter is set in a report output component corresponding to a test coverage rate tool) in a report (report output) stage, receiving the content of the difference code, calculating the coverage rate of the full code by default when the parameter is not transmitted, and outputting a test coverage rate report of the full code; when the parameters are transmitted, the difference code coverage rate is calculated, a difference code test coverage rate report is output, and the flexibility of the test and the output result is higher.

In the embodiment of the present disclosure, taking an on-the-fly mode as an example, when an application service is deployed, a-java parameter is added to a start script, and a parameter form of a pointer added to the start script is, for example:

-javaagent:/tmp/jacoco/lib/jacocoagent.jar＝include＝*,output＝tcpserver,port＝6600,add ress＝*,append＝true。

starting test, running/executing unit test, interface test, function test, performance test, compatibility test and other corresponding test cases. And pre-configuring the instrumentation mode and the type of the corresponding test case, and performing pointer insertion/injection in a targeted manner.

After the test is completed, a export operation is performed. For example, a file in a specific protocol format, for example an exec file, is generated by executing a dump command, and the derived command line is exemplified as follows: java-jar jacococli. jar dump-address localhost-port 6600-destfile./jacoco-demo. exec. In an embodiment, the test result may be exported to the local after the server performs the test, and the export may be performed by generating an exec file by executing a dump command, taking jacoco (an open source Java code coverage tool) as an example of the code coverage tool. Alternatively, in another embodiment, the test is performed directly locally.

And for the test coverage rate of the full code, directly generating a test report by calling the test coverage rate result of the full code of the native jacoco.

For the test coverage of the difference code, the difference code needs to be calculated first, and the difference code can be acquired by implementing the operation S201.

Subsequently, the unexecuted or incompletely executed code logic can be analyzed based on the color marks in the report of the differential code test coverage, and corresponding test cases are supplemented (corresponding to operation S601 in the first embodiment).

A third exemplary embodiment of the present disclosure provides an apparatus for software testing.

Fig. 8 schematically shows a block diagram of an apparatus for software testing according to an embodiment of the present disclosure.

Referring to fig. 8, an apparatus 800 for software testing provided by an embodiment of the present disclosure includes: a difference code acquisition module 801, a compilation and instrumentation module 802, a test module 803, and a test coverage analysis module 804.

The difference code obtaining module 801 is configured to obtain a difference code, where the difference code is an update source code of a target version of software added and/or modified compared to a historical version of software. The difference code acquiring module 801 includes functional modules or sub-modules for implementing the sub-operations S2011 to S2014.

The compiling and instrumentation module 802 is configured to compile and instrumentation the difference code to obtain an executable file with a probe. The probe is used for recording track information experienced when the byte codes in the executable file execute the program. The compile and peg module 802 includes functional modules or sub-modules for implementing the sub-operations S2021, S2022a, S2022b, and S2023b described above.

The test module 803 is configured to test the executable file based on a preset test case, and obtain track information experienced by the bytecode during a test process based on the probe record.

The test coverage analysis module 804 is configured to compare and analyze the track information experienced by the bytecode with the structure of the updated source code to obtain a test coverage result about the difference code. The test coverage analysis module 804 includes functional modules or sub-modules for implementing the sub-operations S2041 to S2044.

According to an embodiment of the present disclosure, the apparatus 800 for testing software further includes: and a test case updating module. The test case updating module is used for updating the preset test case according to the test coverage rate result of the difference code, so that the updated test case combination can make the test coverage rate of the difference code meet the preset requirement. Wherein, the updating mode comprises: one or more test cases are newly added on the basis of the preset test case, or the preset test case is modified to obtain a modified test case, and the updated test case combination correspondingly comprises: the test method comprises the steps of presetting test cases, adding new test cases or modifying the test cases. How the procedure is updated can be referred to the description of the first embodiment.

A fourth exemplary embodiment of the present disclosure provides an apparatus for statistical test coverage.

Fig. 9 schematically shows a block diagram of an apparatus for statistical test coverage according to an embodiment of the present disclosure.

Referring to fig. 9, an apparatus 900 for counting test coverage provided by an embodiment of the present disclosure includes: an information receiving module 901 and a test coverage tool 902.

The information receiving module 901 is configured to receive selection information of a user on a full test or an incremental test.

The test coverage tool 902 is configured to perform a corresponding code test according to the selection information, and output a corresponding full code test coverage report or a differential code test coverage report. Wherein, in the case that the selection information is an incremental test, the test coverage tool is configured to perform the software test method to perform a code test, so as to obtain a test coverage result about the difference code, and is configured to generate a difference code test coverage report according to the test coverage result of the difference code. Specifically, referring to fig. 9, the test coverage tool 902 includes functional modules or sub-modules capable of implementing the operations S201 to S204: a difference code acquisition module 9021, a compiling and instrumentation module 9022, a test module 9023, and a test coverage analysis module 9024. In addition, a functional module or sub-module for implementing operation S601 may be further included: and a test case updating module.

The test coverage tool 902 is configured to perform a full code test to obtain a test coverage result for the full code if the selection information is the full test, and generate a differential code test coverage report according to the test coverage result for the full code.

In this embodiment, the apparatus 900 may correspond to the distributed system architecture 110 shown in fig. 1B. Illustratively, the web page side 111 (front end) employs an Vue framework for package deployment via an npm run build command. The client (server) 112 is a Springboot framework, the database uses a Mysql library, the project is constructed and compiled through a maven install command, and a jar (Java ARchive, a cross-platform software package file format) package is started during deployment.

The test coverage tool 902 may be installed on the web page 111, and may perform test data processing according to traffic distribution to the client 112 of the corresponding service, so as to obtain a test coverage result about the difference code.

In the third embodiment, any multiple of the difference code acquiring module 801, the compiling and instrumentation module 802, the testing module 803 and the testing coverage analyzing module 804 may be combined and implemented in one module, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. At least one of the difference code acquisition module 801, the compiling and instrumentation module 802, the test module 803, and the test coverage analysis module 804 may be implemented at least in part as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented in hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or in any one of or a suitable combination of software, hardware, and firmware. Alternatively, at least one of the difference code acquisition module 801, the compiling and instrumentation module 802, the testing module 803 and the test coverage analysis module 804 may be at least partially implemented as a computer program module which, when executed, may perform corresponding functions.

In the fourth embodiment, any multiple of the information receiving module 901, the difference code obtaining module 9021, the compiling and instrumentation module 9022, the testing module 9023, and the testing coverage analysis module 9024 may be combined in one module to be implemented, or any one of the modules may be split into multiple modules. Alternatively, at least part of the functionality of one or more of these modules may be combined with at least part of the functionality of the other modules and implemented in one module. At least one of the information receiving module 901, the difference code acquiring module 9021, the compiling and instrumentation module 9022, the testing module 9023, and the testing coverage analyzing module 9024 may be at least partially implemented as a hardware circuit, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or may be implemented by hardware or firmware in any other reasonable manner of integrating or packaging a circuit, or implemented by any one of three implementation manners of software, hardware, and firmware, or an appropriate combination of any several of them. Alternatively, at least one of the information receiving module 901, the difference code acquiring module 9021, the compiling and instrumentation module 9022, the testing module 9023, and the test coverage analysis module 9024 may be at least partially implemented as a computer program module that, when executed, may perform a corresponding function.

A fifth exemplary embodiment of the present disclosure provides an electronic apparatus.

Referring to fig. 10, an electronic device 1000 provided in the embodiment of the present disclosure includes a processor 1001, a communication interface 1002, a memory 1003 and a communication bus 1004, where the processor 1001, the communication interface 1002 and the memory 1003 complete communication with each other through the communication bus 1004; a memory 1003 for storing a computer program; the processor 1001 is configured to implement the software testing method or the statistical test coverage method described above when executing the program stored in the memory.

A sixth exemplary embodiment of the present disclosure also provides a computer-readable storage medium. The computer readable storage medium has a computer program stored thereon, and the computer program, when executed by a processor, implements the method for testing software or the method for counting test coverage as described above.

The computer-readable storage medium may be contained in the apparatus/device described in the above embodiments; or may be present alone without being assembled into the device/apparatus. The computer-readable storage medium carries one or more programs which, when executed, implement the method according to an embodiment of the disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the present disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It is noted that, in this document, relational terms such as "first" and "second," and the like, may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other identical elements in a process, method, article, or apparatus that comprises the element.

The foregoing are merely exemplary embodiments of the present disclosure, which enable those skilled in the art to understand or practice the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the disclosure. Thus, the present disclosure is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of software testing, comprising:

acquiring a difference code, wherein the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software;

compiling and instrumentation the difference code to obtain an executable file with a probe, wherein the probe is used for recording track information experienced when the byte code in the executable file executes a program;

testing the executable file based on a preset test case, and obtaining track information experienced by the bytecode in the testing process based on the probe record; and

and comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage rate result about the difference code.

2. The method of claim 1, wherein comparing the trace information experienced by the bytecode with the structure of the updated source code to obtain a test coverage result about the difference code, comprises:

analyzing track information of bytecode experience to obtain information of covered candidate source codes in the process of running the test case, wherein the information represents position information of executed source codes and content of the corresponding executed source codes;

screening target information covering the updated source code from the information of the candidate source code;

determining the number of the covered updated source codes according to the target information; and

and comparing the number of the covered updating source codes with the structure of the updating source codes to obtain the test coverage rate.

3. The method of claim 1, wherein compiling and instrumentation the difference code to obtain an executable file with probes comprises:

determining the instrumentation mode of the executable file to be an offline instrumentation mode or an runtime instrumentation mode based on the type of the test case;

under the condition that the instrumentation mode is an offline instrumentation mode, inserting a probe in the differential code compiling process, wherein the compiled byte code structure with the probe forms the executable file with the probe;

under the condition that the instrumentation mode is the instrumentation mode in operation, compiling the difference code to obtain an executable file; and in the process of deploying and loading the executable file in the test environment, inserting a probe into the byte code of the executable file so as to obtain the executable file with the probe.

4. The method of claim 3, wherein determining that the instrumentation mode of the executable file is an offline instrumentation mode or a runtime instrumentation mode based on the type of the test case comprises:

determining that the instrumentation mode of the executable file is an offline instrumentation mode under the condition that the type of the test case is a unit test;

and under the condition that the type of the test case is interface test, functional test, performance test, compatibility test or automation test, determining that the instrumentation mode of the executable file is an operation instrumentation mode.

5. The method of claim 1, wherein obtaining the difference code comprises:

receiving identification information of target version software and historical version software input by a user;

positioning to the target version software and the historical version software according to the identification information;

analyzing the target version software and the historical version software respectively to obtain a first analysis tree related to the target version software and a second analysis tree related to the historical version software; and

and comparing and analyzing the first analytical tree and the second analytical tree to obtain an updated source code of the target version software compared with the updated source code which is newly added or modified compared with the historical version software, wherein the updated source code is the difference code.

6. The method according to any one of claims 1-5, further comprising:

updating the preset test case according to the test coverage rate result of the difference code, so that the updated test case combination meets the preset requirement on the test coverage rate of the difference code;

wherein, the updating mode comprises: one or more test cases are newly added on the basis of the preset test cases, or the preset test cases are modified to obtain modified test cases, wherein the updated test case combination correspondingly comprises: the test method comprises the steps of presetting test cases, adding new test cases or modifying the test cases.

7. A method for statistical test coverage, comprising:

receiving selection information of a user on a full test or an incremental test;

according to the selection information, carrying out corresponding code testing based on a testing coverage rate tool, and outputting a corresponding full code testing coverage rate report or a difference code testing coverage rate report;

wherein, in case the selection information is an incremental test, the test coverage tool performs a code test by performing the method of any one of claims 1-5, thereby obtaining a test coverage result for a difference code, and generates a difference code test coverage report based on the test coverage result for the difference code.

8. The method of claim 7, wherein an unnecessary fill parameter is set in a report output component of the test coverage tool;

in the event that a user selects incremental testing, the discrepancy code is passed into the report output component as the non-essential fill parameter, such that the report output component generates a discrepancy code test coverage report as a function of the test coverage results of the discrepancy code;

in the case where the user selects the full test, the unnecessary padding parameter is empty.

9. An apparatus for software testing, comprising:

the difference code acquisition module is used for acquiring a difference code, wherein the difference code is an updated source code of the target version software which is newly added and/or modified compared with the historical version software;

the compiling and instrumentation module is used for compiling and instrumentation the difference codes to obtain an executable file with a probe, and the probe is used for recording track information experienced when the byte codes in the executable file execute the program;

the test module is used for testing the executable file based on a preset test case and obtaining track information experienced by the byte code in the test process based on the probe record; and

and the test coverage analysis module is used for comparing and analyzing the track information experienced by the bytecode and the structure of the updated source code to obtain a test coverage result related to the difference code.

10. An apparatus for statistical test coverage, comprising:

the information receiving module is used for receiving selection information of a user on the full test or the incremental test;

the test coverage rate tool is used for carrying out corresponding code test according to the selection information and outputting a corresponding full code test coverage rate report or a difference code test coverage rate report;

wherein, in the case that the selection information is an incremental test, the test coverage tool is configured to perform the method of any one of claims 1-5 to perform a code test, thereby obtaining a test coverage result for a difference code, and to generate a difference code test coverage report based on the test coverage result for the difference code.

11. An electronic device is characterized by comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory are communicated with each other through the communication bus;

a memory for storing a computer program;

a processor for implementing the method of any one of claims 1 to 8 when executing a program stored on a memory.

12. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the method of any one of claims 1-8.