CN114116468A

CN114116468A - Application testing method and device, electronic equipment and storage medium

Info

Publication number: CN114116468A
Application number: CN202111358821.9A
Authority: CN
Inventors: 邹珍珍; 谢蕾; 胡涛
Original assignee: Hangzhou Netease Cloud Music Technology Co Ltd
Current assignee: Hangzhou Netease Cloud Music Technology Co Ltd
Priority date: 2021-11-17
Filing date: 2021-11-17
Publication date: 2022-03-01

Abstract

The present disclosure relates to the field of computer technologies, and in particular, to an application testing method, an application testing apparatus, an electronic device, and a storage medium, where test scenario data of an application to be tested is obtained, and each test scenario data is executed in response to request information for preset background traffic data, so as to obtain corresponding response information, where each response information includes a scenario execution result and verification information, and the verification information is used to verify correctness of the execution result of the test scenario data; and determining the test result of the application to be tested based on the corresponding request time of the request information, the scene execution result and the verification information respectively. Thus, the efficiency and accuracy of the test can be improved.

Description

Application testing method and device, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to an application testing method and apparatus, an electronic device, and a storage medium.

Background

As the architectural complexity of software applications increases, a variety of failures may occur during the application process. When the high availability of the system is insufficient and the system stability is poor, the failure may cause the user to experience a reduced experience of the software application. Therefore, how to test the high availability capability of the application in dealing with the failure becomes a problem to be solved urgently.

In the related art, when a fault test is performed, a test flow is generally performed manually, and therefore, the test efficiency is low. In addition, in the related art, observation errors are generated by manually observing the test results, thereby reducing the test accuracy.

Disclosure of Invention

The embodiment of the disclosure provides an application testing method and device, electronic equipment and a storage medium, so as to improve the testing efficiency and accuracy.

The specific technical scheme provided by the embodiment of the disclosure is as follows:

acquiring test scene data of an application to be tested, wherein each test scene data is generated based on operation content data and an operation sequence label;

responding to request information aiming at preset background flow data, executing each test scene data respectively, and obtaining corresponding response information, wherein each response information comprises a scene execution result and verification information, and the verification information is used for verifying the correctness of the execution result of the test scene data;

and determining the test result of the application to be tested based on the corresponding request time of the request information, the scene execution result and the verification information respectively.

Optionally, the obtaining of the test scenario data specifically includes:

monitoring a code modification event aiming at an application to be tested;

and responding to the monitored code modification event, and acquiring test scene data.

Optionally, the obtaining of the test scenario data specifically includes:

and acquiring test scene data according to a preset test period.

Optionally, the method further includes:

obtaining operation sequence labels, wherein each operation sequence label indicates an operation sequence between operation content data contained in the test scene data; and the number of the first and second groups,

obtaining base scene tags, wherein each base scene tag indicates an execution sequence of test scenes of which the types are base scenes;

and generating test scene data based on the corresponding operation sequence label, the basic scene label and the corresponding operation content data.

Optionally, each of the test scenario data is executed to obtain corresponding response information, and the method specifically includes:

determining first test scene data corresponding to a basic scene label with a label type of being preferentially executed, and executing the first test scene data;

executing each second test scene data respectively to obtain corresponding response information, wherein each second test scene data is test scene data which does not correspond to a basic scene label;

and determining third test scene data corresponding to the basic scene label of which the label type is non-preferentially executed, and executing the third test scene data.

Optionally, each second test scenario data is executed respectively to obtain corresponding response information, and the method specifically includes:

for each piece of second test scenario data, the following operations are respectively performed:

and respectively executing the corresponding operation content data according to the operation sequence labels corresponding to the second test scene data to obtain corresponding response information.

Optionally, the obtaining of the corresponding response information specifically includes:

aiming at the test scene data, respectively executing the following operations:

responding to the application to be tested not to depend on other services, and adding verification information corresponding to the test scene data to response information containing a scene execution result;

and responding to the dependence of the application to be tested on other services, adding the verification information corresponding to the test scene data to the request information of the request dependence service, and obtaining response information returned aiming at the request information.

Optionally, adding the verification information corresponding to the test scenario data to the response information including the scenario execution result includes:

for the response information, the following operations are respectively executed:

in response to the response object corresponding to the acquired response information, adding verification information corresponding to the test scene data to response information containing a scene execution result;

and in response to further acquiring the response object, reading the verification information from the information database and adding the verification information to the response information.

Optionally, adding the verification information corresponding to the test scenario data to the request information requesting for the dependent service, and obtaining response information returned for the request information, specifically including:

aiming at the request information, the following operations are respectively executed:

in response to the request object corresponding to the request information requesting the dependent service, adding the verification information to the request information, and obtaining response information returned for the request information;

and in response to the fact that the request object is not acquired, adding the verification information into an information database, in response to the fact that the request object is further acquired, reading the verification information from the information database and adding the verification information into the request information, and acquiring response information returned aiming at the request information.

An application testing device comprising:

the acquisition module is used for acquiring test scene data, wherein each test scene data is generated based on the operation content data and the operation sequence label;

the processing module is used for responding to request information aiming at preset background flow data, executing each test scene data respectively and obtaining corresponding response information, wherein each response information comprises a scene execution result and verification information, and the verification information is used for verifying the correctness of the execution result of the test scene data;

and the checking module is used for determining the test result of the application to be tested based on the corresponding request time of the request information, the scene execution result and the checking information.

Optionally, the obtaining module is specifically configured to:

monitoring a code modification event aiming at an application to be tested;

Optionally, the obtaining module is specifically configured to:

and acquiring test scene data according to a preset test period.

Optionally, the system further includes a generating module, where the generating module is specifically configured to:

Optionally, when each of the test scenario data is executed respectively and corresponding response information is obtained, the processing module is specifically configured to:

Optionally, when each second test scenario data is executed respectively and corresponding response information is obtained, the processing module is specifically configured to:

Optionally, when the corresponding response information is obtained, the processing module is specifically configured to:

aiming at the test scene data, respectively executing the following operations:

Optionally, when the verification information corresponding to the test scenario data is added to the response information including the scenario execution result, the processing module is specifically configured to:

Optionally, when the verification information corresponding to the test scenario data is added to the request information requesting for the dependent service and response information returned for the request information is obtained, the processing module is specifically configured to:

An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the steps of the application testing method described above when executing the program.

A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the above-mentioned application testing method.

The beneficial effects of the disclosed embodiment are as follows:

the method comprises the steps of obtaining test scene data of an application to be tested, wherein each test scene data is generated based on operation content data and an operation sequence label, responding to request information aiming at preset background flow data, executing each test scene data respectively, obtaining corresponding response information, each response information comprises a scene execution result and verification information, the verification information is used for verifying the correctness of the execution result of the test scene data, and the test result of the application to be tested is determined based on the request time of the corresponding request information, the scene execution result and the verification information. Therefore, each piece of test scene data can be automatically executed in response to the request information for the preset background flow data, so that the automatic test of the application is realized, and the efficiency of the application test is improved.

Drawings

The above and other objects, features and advantages of exemplary embodiments of the present disclosure will become readily apparent from the following detailed description read in conjunction with the accompanying drawings. Several embodiments of the present disclosure are illustrated by way of example, and not by way of limitation, in the figures of the accompanying drawings and in which:

FIG. 1 is a schematic flow chart illustrating the generation of test scenario data according to an embodiment of the present disclosure;

FIG. 2 is a schematic diagram of a test scenario data sample in an embodiment of the present disclosure;

FIG. 3 is a schematic flow chart diagram illustrating a method for creating test scenario data according to an embodiment of the present disclosure;

FIG. 4 is a flow chart of an application testing method in an embodiment of the present disclosure;

FIG. 5 is a flowchart of a first manner of obtaining test scenario data in an embodiment of the present disclosure;

FIG. 6 is a schematic flow chart illustrating a method for obtaining response information according to an embodiment of the present disclosure;

FIG. 7 is a schematic flow chart illustrating a method for obtaining response information according to an embodiment of the present disclosure;

FIG. 8 is a flowchart illustrating a method for adding verification information according to an embodiment of the disclosure;

FIG. 9 is a flow chart of another method of adding verification information in an embodiment of the present disclosure;

FIG. 10 is a schematic illustration of a mail notification in an embodiment of the disclosure;

FIG. 11 is a schematic flow chart illustrating an automated verification method according to an embodiment of the present disclosure;

FIG. 12 is a flowchart illustrating a method for fault scenario testing according to an embodiment of the present disclosure;

FIG. 13 is a schematic diagram of an example implementation in an embodiment of the present disclosure;

FIG. 14 is a schematic structural diagram of an application test apparatus according to an embodiment of the present disclosure;

fig. 15 is a schematic structural diagram of an electronic device in an embodiment of the disclosure.

Detailed Description

The technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are only some embodiments of the present disclosure, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments disclosed herein without making any creative effort, shall fall within the protection scope of the present disclosure.

embodiments of the present disclosure may be implemented as a system, apparatus, device, method, or computer program product. Accordingly, the present disclosure may be embodied in the form of: entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software.

In this document, it is to be understood that any number of elements in the figures are provided by way of illustration and not limitation, and any nomenclature is used for differentiation only and not in any limiting sense.

For convenience of understanding, technical terms involved in the embodiments of the present disclosure are explained:

performing fault drilling: the method is used for detecting whether the software application has high available capacity in certain fault scenes, wherein the fault scenes comprise RPC-dependent service exception, RPC-dependent service timeout, middleware exception, middleware timeout, infrastructure fault and the like.

It should be noted that, in the embodiment of the present disclosure, the fault drilling is also referred to as an application test.

Background flow data: in fault drilling, test traffic is applied to an application in order to simulate the actual running state of the application.

Jenkins: an open source, persistent integration tool may perform specified tasks according to a setting. Jenkins Pipeline is a workflow framework running on Jenkins, and a plurality of tasks which originally run independently are connected to realize complex flow arrangement and visualization which are difficult to complete by a single task.

And (5) TestNG: the open-source automatic test framework has rich case arrangement annotation and a perfect test data management module.

Cucumber: the open-source automatic testing framework can describe a testing scene by using a natural language.

The JavaAgent: the Java code injection mode modifies the byte codes under the condition of not influencing the normal compilation of a Java project, can intercept the modification of the byte codes before loading the class file, and can also change the byte codes of the loaded classes in the running period.

Testing scene data: i.e. fault scenarios, test scenarios for verifying a high availability capability of the application, the test scenario data being generated based on the operation content data and the operation sequence tags.

Manipulating the content data: the specific operation of implementing the test step may be characterized, for example, fault injection, and also, for example, result verification.

Operation sequence tag: and the operation sequence is used for indicating the operation sequence among the operation content data contained in the test scene data. In the embodiments of the present disclosure, the operation sequence labels may be "if", "when", "and", and the like, which are not limited in the embodiments of the present disclosure.

Basic scene label: for indicating the execution order of the test scenarios based on the scenario type. In the embodiment of the present disclosure, the basic scene tag may be before and after the scene is executed, which is not limited in the embodiment of the present disclosure.

The principles and spirit of the present disclosure are explained in detail below with reference to several representative embodiments of the present disclosure.

The inventor finds that with the development of computer technology and the increase of the user population, the architecture complexity of software applications is higher and higher, and different types of faults can occur every time online hot activities in the applications. If the high availability of the application is insufficient, the system stability is poor, and the accident handling plan is missing, then, when a fault occurs, the self-messy feet can be caused in various system alarms, the repair time is delayed, so that the immeasurable economic loss is caused, and the experience of the user on the software application can be reduced. Therefore, how to test the high availability capability of the application in dealing with the failure becomes a problem to be solved urgently.

In the related art, when an application is tested, a test flow is manually executed, so that the test method in the related art is low in efficiency, when an emergency demand or an activity occurs and time is short, a fault test needs to be executed according to priority, and all fault scenes may not be sufficiently tested.

Moreover, since many high available capacities need to determine whether to start the protection policy according to the number or the ratio of the exceptions, in this way in the related art, it is not possible to accurately determine whether the number or the ratio of the exceptions counted when triggering the protection policy strictly meets expectations only according to the monitoring data. Through the manual execution test flow, when observing the test result, can produce observation error to reduced the precision of test, in addition, in order to avoid system's contingency, artificial maloperation, control deviation, need many times rehearsal to observe and can draw the conclusion.

In order to solve the above problems, the present disclosure provides an application testing method, an application testing device, an electronic device, and a storage medium, where test scenario data of an application to be tested is obtained, each test scenario data is generated based on operation content data and an operation sequence tag, each test scenario data is executed in response to request information for preset background traffic data, corresponding response information is obtained, each response information includes a scenario execution result and verification information, the verification information is used to verify correctness of the execution result of the test scenario data, and the test result of the application to be tested is determined based on request time of the corresponding request information, the scenario execution result, and the verification information. Therefore, the test scene data of the application to be tested is obtained, the request information aiming at the preset background flow data is responded, the automatic test of the application to be tested can be realized, the test efficiency is improved, and the labor cost is reduced. In addition, in the embodiment of the disclosure, the verification information is displayed in the response information, so that the scene execution result can be verified by adopting the verification information in the response information, and the verification accuracy is improved.

Having described the general principles of the present disclosure, various non-limiting embodiments of the present disclosure are described in detail below.

Based on the above embodiments, firstly, a process of generating test scenario data in the embodiments of the present disclosure is explained in detail, and referring to fig. 1, a schematic flow diagram of generating test scenario data in the embodiments of the present disclosure specifically includes:

s10: an operation sequence label is obtained.

Wherein each operation sequence label indicates an operation sequence between the operation content data included in the test scenario data.

In the embodiment of the disclosure, a test project is newly created, an operation sequence tag file is created in a resource directory in the test project, and an operation sequence tag used for indicating an operation sequence between different operation content data is defined in the operation sequence tag file, so as to obtain at least one operation sequence tag.

The test project is java project, and the type of the test project is not limited in the embodiment of the present disclosure.

The resource directory may be, for example, an src directory in java engineering, which is not limited in this embodiment of the disclosure.

The operation sequence tag file may be, for example, a step packet, which is not limited in the embodiment of the present disclosure.

S10 in the embodiment of the present disclosure is explained below using a specific example. For example, in the implementation, a step packet is created under an src directory in a java project established in advance, then, a java class for defining an operation sequence between operation content data is newly established in the step packet, and an operation sequence tag of the operation content data is defined by using a Cucumber test framework, and a connection is established between the operation sequence tag and the operation content data, so that during an application test, corresponding operation content data can be executed based on the operation sequence tag.

The newly created java class for defining the operation sequence between the operation content data may be, for example, faultinjectsetp.

The operation sequence tag may be a related annotation defined using the cucumber test framework, a value in the annotation is a regular expression, And a tag name used for defining the operation sequence tag, And the operation sequence tag may be, for example, an assumption (@ Given), Then (@ Then), When (@ When), But (@ But), And (@ ad), And so on, which is not limited in the embodiment of the present disclosure.

The operation content data is Java Method, and the Java Method is used to implement specific operations of the application test, for example, background traffic start, fault injection, fault cancellation, result verification, and the like, which is not limited in the embodiment of the present disclosure.

S11: a base scene label is obtained.

Wherein each base scenario tag indicates an execution order of test scenarios of which the type is a base scenario.

In the embodiment of the disclosure, a basic scene label file is created under a resource directory in a test project, and an execution sequence label for indicating a test scene with a type as a basic scene is newly created in the basic scene label file, so as to obtain at least one basic scene label.

The basic scene tag file may be a testCase package, which is not limited in the embodiment of the present disclosure.

S11 in the embodiment of the present disclosure is explained below using a specific example. For example, in specific implementation, a testCase package is created under an src directory in a java project which is established in advance, and then a basic java class for indicating the execution sequence of a test scene of which the type is a basic scene is newly established in the testCase package, so that the Cucumber framework case orchestration capability can be improved in the basic java class, and the Cucumber framework case orchestration capability can be provided with the capability of running certain operations before and after the execution of some test scenes.

Firstly, using a TestNG @ BeforeState annotation to annotate scenes with @ before and after function Tag in a scene set of the Feature file before test scene execution, and respectively defining operations before and after execution of all fault scenes of the Feature file by using the @ before and after tags, such as before fault injection tool installation and after fault tool uninstallation. And then combining the TestNG @ BeforeClass and @ AfterClass annotations, and acquiring the scenes marked with the @ before and after the scene set is executed.

The base java class may be, for example, basetest.

Among them, the operation sequence tags can be classified into at least the following two tag types: the first type of tag is a priority execution type, and in specific implementation, a priority execution tag can be expressed as @ before; the second type of tag is a non-priority execution type, and in particular implementation, the non-priority execution tag may be denoted as @ after, which is not limited in the embodiment of the present disclosure.

In addition, it should be noted that S10 may be executed first and then S11 may be executed, S11 may be executed first and then S10 may be executed, and of course, S10 and S11 may also be executed at the same time, and the execution order of S10 and S11 is not limited in the embodiment of the present disclosure.

S12: and generating test scene data based on the corresponding operation sequence label, the basic scene label and the corresponding operation content data.

In the embodiment of the disclosure, test scenario data is generated based on the corresponding operation sequence label, the basic scenario label, and the corresponding operation content label.

In specific implementation, a test Scenario is designed in a test Scenario data file, a keyword function is used to define a test Scenario data set, a Scenario (Scenario) defines a test Scenario, And execution steps of operation content data in each test Scenario data are composed of operation sequence tags, if (Given), if (When), Then (Then), But (But), And (And), wherein the execution steps need to uniquely match a regular expression of the defined operation sequence tags.

Also, Background (Background) is used for pre-execution environment initialization work for each test scenario data, e.g. a faulty plug-in Attach to the sanctioned system JVM.

For example, as shown in fig. 2, a schematic diagram of a test scenario data sample in the embodiment of the present disclosure is shown, where the test scenario data includes basic scenario labels "@ before" and "@ after", "@ before" is used for the installation work of the fault tool before the scenario set is executed, and "@ after" is used for the uninstallation work of the fault tool after the scenario set is executed.

The operation sequence tags "if", "when", "and", "then", and the operation content data, such as "set background traffic concurrency number: 3".

In the embodiment of the disclosure, the test scenario data is generated based on the operation content data, the operation sequence label and the basic scenario label, so that the test scenario data can be directly called and executed in the test process, and automation of application test can be realized.

Based on the above embodiment, a detailed description is given below to a process of creating test scenario data in the embodiment of the present disclosure by using a specific example, and referring to fig. 3, a schematic flow diagram of a method for creating test scenario data in the embodiment of the present disclosure is specifically included:

s301: and creating a java project based on the project management tool Maven.

In the embodiment of the disclosure, a Java project is newly created based on a project management tool Maven, and a test frame TestNG and a test frame Cucumber are added in configuration information pom.

It should be noted that the test frameworks in the embodiments of the present disclosure are dependent on each other.

S302: in the java project, a test scenario file is created.

In the embodiment of the present disclosure, a feature directory is newly created under the resource directory in the java project created in S201, and a test scenario data file is newly created under the feature directory.

At least one piece of test scene data in each test scene data file forms a scene set, namely, the test scene data file comprises at least one piece of test scene data; the test scenario data file may be, for example, test scenario data for a computer room, which is not limited in the embodiment of the present disclosure.

It should be noted that, in the embodiment of the present disclosure, each test scenario data in the test scenario data file may be classified based on a scenario type, and each test scenario data file in the test scenario data file may also be freely combined and arranged according to an actual test requirement, which is not limited in the embodiment of the present disclosure.

S303: creating a step package under a resource src catalog of java engineering, newly creating an operation sequence java class in the created step package, and defining an operation sequence label under the operation sequence java class.

In the embodiment of the disclosure, a step package is created under an src directory of java engineering, an operation sequence java class is newly created in the created step package, And operation sequence labels @ Given, @ Then, @ When, @ But And @ And are defined by a test framework Cucumber.

S304: creating a testCase package under a resource src catalog of java engineering, newly creating a basic java class in the created testCase package, and defining a basic scene label under the basic java class.

In the embodiment of the disclosure, a testCase package is created under a resource src directory of java engineering, a basic java class is newly created in the testCase package, and the case arrangement capability of a test framework Cucumber is perfected in the basic java class, so that the initialization environment configuration can be realized, and the installation of a fault injection tool before the execution of each fault scene and the unloading work of the fault tool after the execution are realized.

S305: and generating test scene data based on the operation content data, the operation sequence label and the basic scene label.

In the embodiment of the present disclosure, the operation content data, the operation sequence tag, and the basic scene tag are written in the test scene data file, so as to generate the test scene data.

It should be noted that, in the embodiment of the present disclosure, each operation content data needs to uniquely match a regular expression in an operation sequence label or a base scene label.

S306: creating a java class for executing the test scene data file in the testCase package of the java project, and inheriting the basic java class.

In the embodiment of the disclosure, a java class for executing the test scenario data file is created in a testCase package of java engineering, and inherits a basic java class, for example, a switchroomtestrunner.

S307: and defining a configuration file and specifying the java class of the test scene data file to be operated.

In the embodiment of the disclosure, an xml directory is newly created under a resource directory in a java project, a testNG xml configuration file is created under the xml directory, and a java class of a test scene data file to be operated is specified.

In the embodiment of the disclosure, the test scenario data is generated by adopting the operation content data, the operation sequence tag and the basic scenario tag, so that the automatic fault injection and destruction can be realized in the application test process.

Based on the above embodiment, referring to fig. 4, a flowchart of an application testing method in the embodiment of the present disclosure specifically includes:

s40: and acquiring test scene data of the application to be tested.

Wherein each test scenario data is generated based on the operation content data and the operation sequence label.

In the embodiment of the disclosure, test scene data of an application to be tested is acquired.

It should be noted that the test scenario data in the embodiment of the present disclosure is generated based on the operation content data and the operation sequence label.

Optionally, in the embodiment of the present disclosure, three possible implementation manners are provided for obtaining the test scenario data of the application to be tested, and a detailed description is provided below on a manner of obtaining the test scenario data of the application to be tested in the embodiment of the present disclosure.

The first mode is as follows: and triggering an event.

In the embodiment of the present disclosure, the test scenario data may be obtained by monitoring a code modification event, and referring to fig. 5, a flowchart of a first manner of obtaining the test scenario data in the embodiment of the present disclosure specifically includes:

s401: code modification events for the application to be tested are intercepted.

In the embodiment of the disclosure, whether a code modification event aiming at an application to be tested occurs is monitored.

Optionally, in the embodiment of the present disclosure, a callback link may be configured for the application to be tested, and after the callback link is configured, a code modification event for the application to be tested is monitored.

It should be noted that, in the embodiment of the present disclosure, the code modification event for the application to be tested may be monitored continuously, or the code modification event for the application to be tested may be monitored according to a preset period, which is not limited in the embodiment of the present disclosure.

The code modification event may, for example, submit and push a code to the Gitlab for a developer, which is not limited in the embodiment of the present disclosure.

S402: and acquiring test scene data in response to the monitored code modification event.

In the embodiment of the disclosure, after the code modification event is monitored, the test scenario data is acquired in response to the monitored code modification event.

Optionally, in the embodiment of the present disclosure, when obtaining the test scenario data, the callback link of the application to be tested may be automatically recalled in response to the monitored code modification event, and the test scenario data associated with the callback link is obtained based on the obtained callback link.

By the automatic triggering mode in the embodiment of the disclosure, it can be ensured that the application test flow is immediately started when the project code is modified, and the influence of code change on the high availability of the application can be timely discovered.

The second mode is as follows: triggering in a fixed period.

In the embodiment of the disclosure, the test scene data can be automatically acquired according to a fixed period.

The method specifically comprises the following steps: and acquiring test scene data according to a preset test period.

In the embodiment of the disclosure, a fixed test period is preset, and test scene data is acquired according to the fixed test period, so that an application test process is automatically executed.

When a fixed test period is preset, a cron expression configuration mode can be adopted to trigger the test flow at regular time.

It should be noted that the manner of configuring the cron expression to acquire the test scenario has periodicity.

For example, the configured cron expression is 01 x, and the feature that the test scenario data is automatically acquired at 1 point every morning.

The third mode is as follows: and (5) manual triggering.

In the embodiment of the disclosure, after logging in the jenkins console, a user can manually trigger to acquire test scene data.

In addition, it should be noted that, in the embodiment of the present disclosure, the method is not limited to the above three ways of acquiring the test scenario data.

S41: and responding to the request information aiming at the preset background flow data, executing each test scene data respectively, and obtaining corresponding response information.

Each response message comprises a scene execution result and verification information, and the verification information is used for verifying the correctness of the execution result of the test scene data.

In the embodiment of the present disclosure, each piece of test scenario data is executed in response to request information for preset background traffic data, so as to obtain response information generated based on the corresponding test scenario data.

Optionally, in the embodiment of the present disclosure, a possible implementation manner is provided for obtaining corresponding response information, and referring to fig. 6, a flowchart of a method for obtaining response information in the embodiment of the present disclosure is shown, which specifically includes:

s411: and determining first test scene data corresponding to the base scene label with the label type being preferentially executed, and executing the first test scene data.

In the embodiment of the present disclosure, the basic scenario tag may be divided into two types, where the first type is a priority execution type, and the second type is a non-priority execution type. When the tag type of the base scenario tag is execution priority, the first test scenario data needs to be executed prior to the second test scenario data. When the tag type of the basic scenario tag is the non-priority execution, the third test scenario data needs to be executed after all the second test scenario data is executed.

Therefore, in the embodiment of the present disclosure, first, based on the tag type corresponding to each of the at least one basic scene tag, the tag type is determined to be the basic scene tag that is executed preferentially from the at least one basic scene tag, and corresponding first test scene data is determined, and finally, the determined first test scene data is executed.

The @ before represents that the type of the tag is a basic scene tag which is executed preferentially, and the @ after represents that the type of the tag is a basic scene tag which is not executed preferentially, first test scene data corresponding to the @ before is executed first.

S412: and respectively executing each second test scene data to obtain corresponding response information.

And each second test scene data is the test scene data which does not correspond to the basic scene label.

In the embodiment of the present disclosure, when the test scenario data does not correspond to the basic scenario tag, it is determined that the test scenario data is second test scenario data capable of determining corresponding response information based on the request information, and therefore, each second test scenario data is sequentially executed according to the sequence of each test scenario data in the test scenario data file, so as to obtain corresponding response information.

For example, when the test scenario data is "a machine room service injection network delay fault, and a machine room switching high availability capability" is verified, it is determined that the test scenario data does not correspond to the basic scenario label, and when the test scenario data is the second test scenario data, the second test scenario data is executed, and corresponding response information is obtained.

Optionally, in the embodiment of the present disclosure, a possible implementation manner is provided for executing each piece of second test scenario data, where any one piece of second test scenario data is taken as an example, and a process of obtaining response information is described as follows:

In the embodiment of the present disclosure, since the operation sequence tags are used to indicate the operation sequence among the operation content data, the operation content data corresponding to each operation sequence tag is respectively executed according to the operation sequence tag corresponding to the second test scenario data, so as to obtain corresponding response information.

For example, when the operation sequence tags are "if" and "if", the operation content data corresponding to the operation sequence tag "if" is executed first, and then the operation content data corresponding to the operation sequence tag "if" is executed.

Optionally, in the embodiment of the present disclosure, a possible implementation manner is provided for obtaining corresponding response information, and a process of obtaining response information in the embodiment of the present disclosure is described in detail below, and as shown in fig. 7, a schematic flow diagram of a method for obtaining response information in the embodiment of the present disclosure specifically includes:

s4121: and responding to the condition that the application to be tested does not depend on other services, and adding the verification information corresponding to the test scene data into the response information containing the scene execution result.

In the embodiment of the disclosure, the test scene can be divided into the following two types, the first type is independent of other services, that is, the application to be tested can normally run without depending on other services in the running process; the second type is that the application to be tested depends on other services, that is, the application to be tested needs to depend on other services in the running process to be able to realize normal running.

When the type of the application to be tested is not dependent on other services, the test result of the application to be tested can be verified and obtained through the scene execution result of the application to be tested, and the verification information corresponding to the test scene data is added to the response information containing the scene execution result in response to the fact that the application to be tested is not dependent on other services.

The scene execution result of the application to be tested is an execution result obtained after the application to be tested is operated in the test scene.

In this way, after the response information including the scene execution result and the verification information is obtained, the scene execution result can be verified based on the verification information in the response information, so as to obtain the corresponding test result.

For example, when the test scenario is machine room switching, the test scenario is executed, the obtained scenario execution result is network delay 10ms, and the verification information included in the response information is network delay 10ms, it is determined that the corresponding test result is that no fault exists.

Optionally, in the embodiment of the present disclosure, a possible implementation manner is provided for adding check information, and referring to fig. 8, a flowchart of a method for adding check information in the embodiment of the present disclosure is shown, which specifically includes:

s4121-1: and in response to the response object corresponding to the acquired response information, adding the verification information corresponding to the test scene data to the response information containing the scene execution result.

In the embodiment of the present disclosure, in response to acquiring the response object corresponding to the response information, that is, when the application shows the code position of the key information in the test process, and the response object corresponding to the response information can be just acquired, the verification information corresponding to the test scene data is directly added to the corresponding response information.

When the method is specifically implemented, the code corresponding to the response information can be directly modified.

S4121-2: and in response to further acquiring the response object, reading the verification information from the information database and adding the verification information to the response information.

In the embodiment of the present disclosure, in response to that a response object is not acquired, that is, a response object is not acquired, in the above case, check information may be first added to the information database, and whether a response object is acquired is monitored continuously, and when it is determined that a response object is acquired, check information is read from the information database in response to a response object further acquired, and the read check information is added to response information.

The information database may be a Memcache or a Redis cache service, which is not limited in the embodiment of the present disclosure.

S4122: and responding to the dependence of the application to be tested on other services, adding the verification information corresponding to the test scene data to the request information of the request dependence service, and obtaining response information returned aiming at the request information.

In the embodiment of the disclosure, when the application to be tested depends on other services, that is, the normal operation of the application to be tested needs to depend on other services, the response information corresponding to the test scenario data is added to the request information requesting for the dependent services, so that the response information returned for the request information is obtained after the test scenario data is executed.

Optionally, in the embodiment of the present disclosure, a possible implementation manner is provided for obtaining the response information, and referring to fig. 9, a flowchart of another method for adding the verification information in the embodiment of the present disclosure specifically includes:

s4122-1: and in response to acquiring a request object corresponding to the request information requesting the dependent service, adding the verification information to the request information, and acquiring response information returned for the request information.

In the embodiment of the disclosure, when the application to be tested shows a code position capable of adding check information in a testing process, and can acquire a request object corresponding to request information requesting for a dependent service, the request information is directly modified in response to the acquisition of the request object corresponding to the request information requesting for the dependent service, and the check information is added to the request information, so that response information returned for the request information is acquired.

When the method is specifically implemented, the code corresponding to the request information can be directly modified.

S4122-2: and in response to further acquiring the request object, reading the verification information from the information database and adding the verification information to the request information, and acquiring response information returned for the request information.

In the embodiment of the present disclosure, when the code position showing the key information in the test process is applied and a request object corresponding to request information requesting a dependent service cannot be acquired, in response to that the request object is not acquired, in the above case, check information may be first added to the information database, and whether the request object is acquired is continuously monitored, and when the request object is determined to be acquired, the check information is read from the information database in response to a further acquired request object, and the read check information is added to the request information, so as to acquire response information returned for the request information.

S413: and determining third test scene data corresponding to the basic scene label of which the label type is non-preferentially executed, and executing the third test scene data.

In the embodiment of the disclosure, first, based on a tag type corresponding to each of at least one basic scene tag, a corresponding basic scene tag is determined to be executed non-preferentially from the at least one basic scene tag, corresponding third test scene data is determined, and finally, the determined third test scene data is executed.

The @ before represents that the type of the tag is a base scene tag which is executed preferentially, and the @ after represents that the type of the tag is a base scene tag which is not executed preferentially, third test scene data corresponding to the @ after is executed.

S42: and determining the test result of the application to be tested based on the corresponding request time of the request information, the scene execution result and the verification information respectively.

In the embodiment of the disclosure, the test result of the application to be tested is determined based on the request time, the scene execution result and the verification information of the corresponding request information.

Therefore, the scene execution result is verified through the verification information in the response information, the problem that the verification granularity is not enough due to monitoring data acquisition errors and manual observation errors can be avoided, and the precision of result verification is improved.

Optionally, in this embodiment of the present disclosure, under the/target/subset-reports directory, TestNG may automatically generate an Email-report html test report based on a test result, and send a test report mail notification in combination with a Jenkins Email plug-in, as shown in fig. 10, which is a schematic diagram of the mail notification in the embodiment of the present disclosure, a test scenario in which a machine room is switched can be obtained through the test report, the test scenario is successful 12 times, the test scenario is used for 1011496ms in common, and the test scenario is used for 355 times, and the test scenario is used for 859345ms in common.

In the embodiment of the disclosure, the application test can be executed by timing triggering or code change automatic triggering, so that the automation of the application test is realized, and the verification efficiency is improved. Moreover, the verification information of the performance condition during the system failure and during the failure recovery is returned in the response information of the background flow data, so that the verification accuracy can be improved.

Based on the above embodiment, referring to fig. 11, a schematic flow chart of an automatic verification method in the embodiment of the present disclosure is shown, which specifically includes:

1. and (5) testing engineering.

The testing project concurrently requests background flow data for the application to be tested through java multithreading, and simultaneously records the request time of each request message and corresponding response messages.

2. An application to be tested.

The application to be tested is based on the JavaAgent technology, before loading a class file generated after compiling test scene data, the byte code is intercepted and modified, and verification information required during application test verification is written into response information and returned to the test engineering.

Based on the difference of test scenarios, the embodiment of the present disclosure can be divided into the following four code instrumentation manners.

The first mode is as follows:

when the code position of the application to be tested showing the verification information in the test process just can obtain the response object of the response information, the JavaAgent firstly obtains the verification information, and then the JavaAgent directly adds the verification information into the response information.

The second mode is as follows:

when the application to be tested shows the code position of the check information in the test process and cannot obtain the response object of the response information, the JavaAgent firstly obtains the check information and writes the check information into the information database, and when the response object corresponding to the response information is determined to be obtained, the JavaAgent obtains the check information from the information database and adds the check information into the response information.

The information database may be, for example, a Memcache or a Redis cache service.

The third mode is as follows:

when the code position of the application to be tested showing the check information in the test process can just obtain the request object corresponding to the request information requesting the dependent service, the JavaAgent firstly obtains the check information and directly adds the check information to the request information.

The fourth mode is that:

when the code position of the application to be tested shows the check information in the test process and the request object corresponding to the request information requesting the dependent service cannot be obtained, the JavaAgent firstly obtains the check information and writes the check information into the information database, and when the request object corresponding to the request information is determined to be obtained, the JavaAgent obtains the check information from the information database and adds the check information into the request information.

In the embodiment of the disclosure, based on the JavaAgent technology, code instrumentation is performed on the application to be tested, and the verification information is added to the response information containing the scene execution result, so that the response information of the background flow data contains the verification information of the performance of the application to be tested during the fault period and during the fault recovery, and the accuracy of judging whether the high availability of the system meets the expectation or not can be improved.

Based on the above embodiment, the following describes in detail the application test method in the embodiment of the present disclosure by taking test scenario data as an example of a fault scenario, and referring to fig. 12, the flowchart of a method for testing a fault scenario in the embodiment of the present disclosure specifically includes:

1. and triggering a fault test.

In the embodiment of the present disclosure, the fault test triggering may be specifically implemented by the following three ways:

the first mode is as follows: and monitoring a code modification event through webhook, so that fault scene data are automatically acquired, and fault drilling is triggered.

The second mode is as follows: and automatically acquiring fault scene data according to a preset period through a cron expression to trigger fault drilling.

The third mode is as follows: manually acquiring fault scene data and triggering fault drilling.

2. And deploying tasks by the test environment.

In the embodiment of the disclosure, the test environment deployment task calls the deployment platform OpenAPI by writing a script or realizes remote automatic deployment based on an Angle tool.

3. The fault scenario data performs a task.

In the embodiment of the disclosure, fault scene data is built based on the test frame TestNG and the test frame cuumber, a project code is pulled in a Jenkins Job for executing a fault scene, and mvn clean test-dmaven test-fault-error command can be executed to automatically execute the fault scene data for the automatic test of the fault scene, so as to obtain corresponding response information.

The fault scene data execution tasks comprise background flow starting, fault injection and destruction and test result verification.

4. And informing the task of the test result.

In the embodiment of the present disclosure, the test result notification task is an Email notification sent based on an HTML test report output by the TestNG framework after the test is completed and the Jenkins Email plug-in, which is shown in fig. 13 and is a schematic diagram of an execution sample in the embodiment of the present disclosure.

It should be noted that, in the embodiment of the present disclosure, a continuous integration tool Jenkins is used to run a fault test full process, and Jenkins Pipeline is used to run execution steps in series, where the execution steps include a test environment deployment task, a fault scenario data execution task, and a test result notification task.

In the embodiment of the disclosure, through automation of a trigger mechanism, automation of deployment of a test environment, automation of execution of fault scene data and automation of notification of test results, the frequency and efficiency of fault tests can be improved, the available guarantee frequency is improved by more than 10 times compared with customized activities, the timeliness of tests is increased, the labor input cost is reduced, and the input manpower is basically reduced to 0. Moreover, by the method in the embodiment of the disclosure, automatic and normalized high-availability evaluation of application can be realized, fault regression can be performed once or multiple times every day, and regression can be automatically triggered immediately when codes are modified, so that the problem of on-line stability caused by improper manual evaluation can be avoided.

Based on the same inventive concept, the embodiment of the present disclosure further provides an application testing apparatus, where the application testing apparatus may be a hardware structure, a software module, or a hardware structure plus a software module, and the embodiment of the application testing apparatus may inherit the content described in the foregoing method embodiment. Based on the above embodiments, referring to fig. 14, a schematic structural diagram of an application testing apparatus in the embodiment of the present disclosure is shown, which specifically includes:

an obtaining module 1400, configured to obtain test scenario data, where each test scenario data is generated based on operation content data and an operation sequence tag;

a processing module 1401, configured to respond to request information for preset background traffic data, and execute each piece of test scene data respectively to obtain corresponding response information, where each piece of response information includes a scene execution result and verification information, and the verification information is used to verify correctness of the execution result of the test scene data;

the checking module 1402 is configured to determine a test result of the application to be tested based on the request time, the scene execution result, and the checking information of the corresponding request information, respectively.

Optionally, the obtaining module 1400 is specifically configured to:

monitoring a code modification event aiming at an application to be tested;

Optionally, the obtaining module 1400 is specifically configured to:

and acquiring test scene data according to a preset test period.

Optionally, the method further includes a generating module 1403, where the generating module 1403 is specifically configured to:

Optionally, when each piece of test scenario data is executed respectively to obtain corresponding response information, the processing module 1401 is specifically configured to:

Optionally, when each second test scenario data is executed respectively to obtain corresponding response information, the processing module 1401 is specifically configured to:

Optionally, when obtaining corresponding response information, the processing module 1401 is specifically configured to:

aiming at the test scene data, respectively executing the following operations:

Optionally, when the verification information corresponding to the test scenario data is added to the response information including the scenario execution result, the processing module 1401 is specifically configured to:

Optionally, when the verification information corresponding to the test scenario data is added to the request information requesting the dependent service and response information returned in response to the request information is obtained, the processing module 1401 is specifically configured to:

Based on the above embodiments, fig. 15 is a schematic structural diagram of an electronic device in an embodiment of the disclosure.

The disclosed embodiments provide an electronic device that may include a processor 1510 (CPU), a memory 1520, an input device 1530, an output device 1540, and the like, wherein the input device 1530 may include a keyboard, a mouse, a touch screen, and the like, and the output device 1540 may include a Display device such as a Liquid Crystal Display (LCD), a Cathode Ray Tube (CRT), and the like.

The memory 1520 may include read-only memory (ROM) and Random Access Memory (RAM), and provides the processor 1510 with program instructions and data stored in the memory 1520. In the disclosed embodiment, the memory 1520 may be used to store a program for applying the testing method in any of the disclosed embodiments.

The processor 1510 is configured to execute any of the application testing methods according to the embodiments of the present disclosure by calling the program instructions stored in the memory 1520.

Based on the above embodiments, in the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and the computer program, when executed by a processor, implements the application testing method in any of the above method embodiments.

As will be appreciated by one skilled in the art, embodiments of the present disclosure may be provided as a method, system, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present disclosure may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and so forth) having computer-usable program code embodied therein.

The present disclosure is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the present disclosure. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

It will be apparent to those skilled in the art that various changes and modifications can be made in the present disclosure without departing from the spirit and scope of the disclosure. Thus, if such modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalents, the present disclosure is intended to include such modifications and variations as well.

Claims

1. An application testing method, comprising:

2. The method of claim 1, wherein obtaining test scenario data specifically comprises:

monitoring a code modification event aiming at an application to be tested;

3. The method of claim 1, wherein obtaining test scenario data specifically comprises:

and acquiring test scene data according to a preset test period.

4. The method of claim 1, further comprising:

5. The method of claim 4, wherein the executing each of the test scenario data respectively to obtain corresponding response information specifically comprises:

6. The method of claim 5, wherein the step of executing each second test scenario data to obtain corresponding response information comprises:

7. The method according to any one of claims 1 to 6, wherein obtaining corresponding response information specifically comprises:

aiming at the test scene data, respectively executing the following operations:

8. An application testing apparatus, comprising:

9. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the steps of the method of any of claims 1-7 are implemented when the program is executed by the processor.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program when executed by a processor implements the steps of the method of any one of claims 1 to 7.