CN112732550A

CN112732550A - Model-based automatic testing method and system

Info

Publication number: CN112732550A
Application number: CN202011131507.2A
Authority: CN
Inventors: 朱少凡; 汪澔; 王文杰; 唐剑; 王世元; 谭文韬
Original assignee: Commercial Aircraft Corp of China Ltd; Beijing Aeronautic Science and Technology Research Institute of COMAC
Current assignee: Commercial Aircraft Corp of China Ltd; Beijing Aeronautic Science and Technology Research Institute of COMAC
Priority date: 2020-10-21
Filing date: 2020-10-21
Publication date: 2021-04-30

Abstract

The invention discloses a model-based automatic testing method and system. Wherein, the method comprises the following steps: acquiring external environment definition information; constructing a scene model according to the external environment definition information; determining a test target through the scene model; and analyzing the test target to obtain a test result. The invention solves the technical problems of incomplete test, low efficiency and difficult test data unification in the prior art.

Description

Model-based automatic testing method and system

Technical Field

The invention relates to the field of automatic testing, in particular to an automatic testing method and system based on a model.

Background

The invention relates to an automatic test method based on a model, which realizes formalized expression of functional requirements and automatic generation of test cases by utilizing a modeling mode, can adapt to the access of a tested system of software and hardware and the automatic analysis of test results, and finally establishes a data accurate transmission process of requirement-test-analysis. Belongs to the field of avionics systems.

Generally for a test, we will have the following parts:

a) and (3) testing environment: the test environment is an external environment for a test, in which conditions under which a System Under Test (SUT) should be tested are defined, and in this deceleration scenario, factors such as runway type and weather conditions are part of the test environment

b) System Under Test (SUT): the system is an entity of the system to be tested, which is often provided by an external third party, can be a model, can be software or hardware (real part). Wherein the test model expresses the performance state and behavior of the tested system body,

c) test case: the test case describes the whole testing framework and is responsible for guiding the tester to perform testing and result analysis, and the test case is divided into several parts: 1) and (3) test excitation: test stimuli are conditions that trigger tests, which are often diverse, for example, in an aircraft ground deceleration scenario, where a drive deceleration begins, a brake is opened, a spoiler is opened, engine back-thrust, etc., is the test drive, and the system under test responds according to parameters and behaviors defined under the trigger conditions, etc. 2) The system comprises a test model, a data processing module and a data processing module, wherein the test model is different from a tested system and describes expected state transition and behavior definition of the system under a certain test scene; 3) and (3) testing environment: the item describes a real test environment by using a modeled or literal expression; 4) testing the expected output: the test expected output expresses how the system should respond under this test case, and what the system should output given the test stimulus and the input.

The existing test flow generally starts with the compiling of a test outline, after a tester takes corresponding test requirements, the test case corresponding to each requirement is artificially defined and compiled by combining the prior test experience and requirement content through discussion, then test excitation of a tested system (a model, software, hardware and the like) is input according to the content of each test case, and the test case under a certain requirement is analyzed according to the comparison between the response value of the tested system and an expected result. After the test of all test cases is completed, all test results are integrated, and finally the test work of the tested system based on the requirement is completed.

The prior technical scheme has the following defects:

a) the civil aircraft comprehensive avionics system is complex in structure and various in use scenes, and comprehensive and rigorous test design and analysis are needed for simulation experiments of the comprehensive avionics system.

b) In the face of a complex avionics system, the traditional method for manually constructing the test case is low in working efficiency, the constructed test case possibly cannot ensure the integrity and comprehensiveness of the test, and a manual mode is easy to have a high error possibility.

c) In the traditional method, a document is used as a data exchange mode between a design department and a test department, the data source is difficult to unify and test, the comprehensive program generated based on text description has ambiguity, and the test result is difficult to repeatedly reproduce and trace.

d) After the test requirements or the design of the tested avionics system are changed, the maintainability and the reusability of the test case constructed by the traditional method are poor.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the invention provides a model-based automatic testing method and system, which at least solve the technical problems of incomplete testing, low efficiency and difficulty in test data unification in the prior art.

According to an aspect of an embodiment of the present invention, there is provided a model-based automated testing method, including: acquiring external environment definition information; constructing a scene model according to the external environment definition information; determining a test target through the scene model; and analyzing the test target to obtain a test result.

Optionally, the acquiring the external environment definition information includes: determining test scene data; and analyzing the functional requirements according to the test scene data to obtain the external environment definition information.

Optionally, the determining, by the scene model, a test target includes: determining the test target by a lightweight configuration, or targeting the functional requirement as the test target.

Optionally, before analyzing the test target to obtain a test result, the method further includes: judging whether the test target is the functional requirement or not; when the test target is the functional requirement, executing a preset analysis mode.

Optionally, the preset analysis manner includes: if the functional requirements are state combinations, when all state tests pass, the test result of the functional requirements is that the tests pass; if the functional requirement is a temporal logic, the test result of the functional requirement is a test pass when the result of the logic expression is true.

According to an aspect of the embodiments of the present invention, there is also provided a model-based automated testing system, including: the acquisition module is used for acquiring external environment definition information; the construction module is used for constructing a scene model according to the external environment definition information; the determining module is used for determining a test target through the scene model; and the analysis module is used for analyzing the test target to obtain a test result.

Optionally, the obtaining module includes: the determining unit is used for determining test scene data; and the analysis unit is used for analyzing the functional requirements according to the test scene data to obtain the external environment definition information.

Optionally, the determining module includes: a first determination unit configured to determine the test target by a lightweight configuration, or a second determination unit configured to set the functional requirement as the test target.

Optionally, the system further includes: the judging module is used for judging whether the test target is the functional requirement or not; and the analysis module is also used for executing a preset analysis mode when the test target is the functional requirement.

There is also provided, in accordance with an aspect of an embodiment of the present invention, a computer program product including instructions which, when executed on a computer, cause the computer to perform a model-based automated testing method.

According to an aspect of the embodiment of the present invention, there is also provided a non-volatile storage medium, which includes a stored program, wherein the program controls a device in which the non-volatile storage medium is located to execute a model-based automated testing method when running.

According to an aspect of the embodiments of the present invention, there is also provided an electronic apparatus, including a processor and a memory; the memory has stored therein computer readable instructions for execution by the processor, wherein the computer readable instructions when executed perform a model-based automated testing method.

In the embodiment of the invention, an acquisition module is adopted for acquiring the external environment definition information; the construction module is used for constructing a scene model according to the external environment definition information; the determining module is used for determining a test target through the scene model; the analysis module is used for analyzing the test target to obtain a test result and the like, and solves the technical problems that the test is incomplete, the efficiency is low and the test data is difficult to unify in the prior art.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

FIG. 1 is a flow diagram of a method for model-based automated testing in accordance with an embodiment of the present invention;

FIG. 2 is a block diagram of a model-based automated test system according to an embodiment of the present invention;

FIG. 3 is a schematic diagram of a model-based automated test architecture according to an embodiment of the present invention;

FIG. 4 is a schematic diagram of a model-based automated test tool architecture according to an embodiment of the present invention;

FIG. 5 is a schematic diagram of a model-based automated test procedure according to an embodiment of the present invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

In accordance with an embodiment of the present invention, there is provided a method embodiment of a model-based automated testing method, it being noted that the steps illustrated in the flowchart of the figure may be performed in a computer system, such as a set of computer-executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than here.

Example one

Fig. 1 is a flow chart of a method for model-based automated testing according to an embodiment of the present invention, as shown in fig. 1, the method comprising the steps of:

step S102, obtaining external environment definition information.

Specifically, the model-based test (MBT) according to the embodiment of the present invention is intended to describe, generate, execute, and analyze a test by using a modeling means, generate a test case by using an automated means, access an actual System Under Test (SUT) in a defined test environment, perform a test under the guidance of the test case, and compare a final generated result with a desired result in the test case, thereby completing verification and confirmation related to a requirement and the like.

Specifically, in the functional requirement capturing stage of the embodiment of the present invention, a test scenario is determined, corresponding functional requirements in the scenario are extracted, and a tester analyzes the content of the proposed functional requirements to determine the external environment definition required by the functional requirements, thereby providing input for modeling the test scenario.

And step S104, constructing a scene model according to the external environment definition information.

Specifically, in the scene modeling stage, a definition for modeling the tested system and the external environment interacting with the tested system based on the SysML language is adopted. And defining a scene model by using an architecture diagram of SysML language and a state machine diagram, expressing all normal states of the tested system and the external environment under a selected scene, and classifying the states according to the subsystems in the tested system. Based on the state and model of the system to be tested, according to the content description of the functional requirement, the formal expression of the requirement is completed by utilizing the temporal logic (LTL). The functional requirements will transform the textual description into a specific logical expression by temporal logic in a state-combinatorial manner. Thereby creating a state to demand mapping and tracing.

Where SysML defines semantics for the structural, behavioral, demand, and parametric models of the system. The structural model emphasizes the hierarchy of the system and the interconnection relationships between objects, including classes and assemblies. The behavior model emphasizes the behavior of objects in the system, including their activity, interaction, and state history. The demand model emphasizes the retrospective relationship between demands and the satisfaction relationship of the design to the demands. Parametric models emphasize the constraining relationships between the properties of the system or component. SysML provides the full semantics for model representation, and like UML, the structure of SysML language is also based on a four-layer meta-model structure: meta-meta models, and user objects. The meta-meta model layer has the highest abstraction level, is a model for defining meta model description language, and provides the most basic concept and mechanism for defining elements and various mechanisms of the meta model. A meta-model is an instance of a meta-meta model that defines a model of a model description language. The meta-model provides various packages of the expression system, defined types of model elements, tag values and constraints, and the like. A model is an instance of a meta-model that defines a domain-specific description language model. The user object is an instance of the model. Any complex system appears to the user as a specific object of mutual communication, with the purpose of achieving the functionality and performance of the complex system.

And S106, determining a test target through the scene model.

Specifically, the test engineer may import the built target scene model into the model-based automated testing tool, and define the test target. Where the test engineer may target a state machine, such as: the test target is "the system should reach a certain state"; because the requirements are formally defined, a test engineer can directly test the requirements as a test target; the test target based on the scene takes direct compiling of the test scene as a means, and the problem that the scene model cannot well describe dynamic change is solved; LTL temporal logic gives engineers more possibilities and flexibility to define test targets. Subsequently, an important task for the test engineer is to define how to input the test stimulus into the actual system under test in the correct form by implementing the test adapter after defining the test target. Because the model-based automatic testing tool provides a corresponding framework, only a test engineer is required to develop lightweight codes; meanwhile, the test engineer needs to configure some interfaces in the non-test excitation in the actual system under test, because the interface information is not used as the excitation input in the model test process, but changes with the input and output changes of the system under test, thereby affecting the test state of the system under test. A corresponding light weight configuration is therefore required for the test engineer.

And step S108, analyzing the test target to obtain a test result.

Specifically, with the help of a model-based automated testing tool, defined test cases are automatically generated, a testing process is compiled and run in the form of codes, and an actual system under test is connected and tested in the presence of an interface adapter. The model-based automatic test tool will complete the automatic generation of test results according to the response condition of the actual system under test.

Specifically, if the requirement is directly selected as the test target in the test definition, the test tool will determine that: if the requirements are state combinations, the test tool judges whether each state passes the test, only when all the state tests pass, the requirement is considered to pass the test, otherwise, the requirement does not pass the test; if the requirement is temporal logic, the test tool judges whether the result of the logic expression is true or not at the end of the test, if so, the test is passed, otherwise, the test is not passed; the test tool generates a corresponding test result report document according to the test result and the analysis.

It should be noted that the test result may be generated by displaying the user in a test report, or by displaying the entire test result and process in a combined form of a graphic and a text according to the test result, and displaying the analysis result to the user in a form of a graphic and a text, so that the test result is displayed more abundantly.

Through the steps, the technical problems that in the prior art, the test is incomplete, the efficiency is low, and the test data is difficult to unify can be solved.

Implement two

Fig. 2 is a block diagram of a model-based automated testing method and system according to an embodiment of the present invention, and as shown in fig. 2, the system includes:

an obtaining module 20, configured to obtain the external environment definition information.

And the building module 22 is used for building a scene model according to the external environment definition information.

And the determining module 24 is used for determining the test target through the scene model.

And the analysis module 26 is configured to analyze the test target to obtain a test result.

According to another aspect of embodiments of the present invention, there is also provided a computer program product comprising instructions which, when run on a computer, cause the computer to perform a model-based automated testing method.

Specifically, the method comprises the following steps: acquiring external environment definition information; constructing a scene model according to the external environment definition information; determining a test target through the scene model; and analyzing the test target to obtain a test result.

According to another aspect of the embodiment of the present invention, a nonvolatile storage medium is further provided, and the nonvolatile storage medium includes a stored program, wherein the program controls a device in which the nonvolatile storage medium is located to execute a model-based automated testing method when running.

According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a processor and a memory; the memory has stored therein computer readable instructions for execution by the processor, wherein the computer readable instructions when executed perform a model-based automated testing method.

EXAMPLE III

According to an application scenario of an embodiment of the present invention, fig. 2 is a schematic diagram of an automated testing architecture based on a model according to an embodiment of the present invention, as shown in fig. 2, in a functional requirement capturing stage, a testing scenario is first determined, corresponding functional requirements in the scenario are extracted, and a tester analyzes the content of the proposed functional requirements to determine an external environment definition required by the functional requirements, thereby providing an input for modeling the testing scenario. In the scene modeling stage, the definition of modeling the tested system and the external environment interacting with the tested system based on SysML language is adopted. And defining a scene model by using an architecture diagram of SysML language and a state machine diagram, expressing all normal states of the tested system and the external environment under a selected scene, and classifying the states according to the subsystems in the tested system. Based on the state and model of the system to be tested, according to the content description of the functional requirement, the formal expression of the requirement is completed by utilizing the temporal logic (LTL). The functional requirements will transform the textual description into a specific logical expression by temporal logic in a state-combinatorial manner. Thereby creating a state to demand mapping and tracing. The test engineer will import the built target scene model into the model-based automated test tool and define the test targets. Where the test engineer may target a state machine, such as: the test target is "the system should reach a certain state"; because the requirements are formally defined, a test engineer can directly test the requirements as a test target; the test target based on the scene takes direct compiling of the test scene as a means, and the problem that the scene model cannot well describe dynamic change is solved; LTL temporal logic gives engineers more possibilities and flexibility to define test targets. Subsequently, an important task for the test engineer is to define how to input the test stimulus into the actual system under test in the correct form by implementing the test adapter after defining the test target. Because the model-based automatic testing tool provides a corresponding framework, only a test engineer is required to develop lightweight codes; meanwhile, the test engineer needs to configure some interfaces in the non-test excitation in the actual system under test, because the interface information is not used as the excitation input in the model test process, but changes with the input and output changes of the system under test, thereby affecting the test state of the system under test. A corresponding light weight configuration is therefore required for the test engineer. With the help of the model-based automatic test tool, the defined test case is automatically generated, the test process is compiled and executed in the form of code, and the actual tested system is connected and the test is executed in the participation of the interface adapter. The model-based automatic test tool will complete the automatic generation of test results according to the response condition of the actual system under test.

If the requirement is directly selected as a test target in the test definition, the test tool will judge:

1. if the requirements are state combinations, the test tool judges whether each state passes the test, only when all the state tests pass, the requirement is considered to pass the test, otherwise, the requirement does not pass the test;

2. if the requirement is temporal logic, the test tool judges whether the result of the logic expression is true or not at the end of the test, if so, the test is passed, otherwise, the test is not passed;

the method realizes that whether the tested system passes the test or not is judged by utilizing the formal expression of the requirements under the test scene, and realizes the automatic tracing of the requirements and the error-free transmission of data, thereby ensuring the validity of the test case and the test integrity aiming at the requirements.

According to the application scenario of the embodiment of the present invention, fig. 3 is a schematic diagram of a model-based automated testing tool architecture according to the embodiment of the present invention, and as shown in fig. 3, the model-based automated testing method can implement tracing of requirements and automatic generation of test cases, thereby implementing automated verification of functional requirements based on the scenario. Wherein the automatic test tool based on the model is used as an effective support for the method, and the related functions are realized semi-automatically or automatically.

Wherein, the test management front end accesses the component through a Web-based interface; the test specification subsystem is responsible for developing and verifying various formalized test specifications; the test visual subsystem is responsible for online visualizing the execution condition of the test in various test scenes; the real-time test subsystem is responsible for dynamic generation test, real-time execution test and dynamic test evaluation.

The model-based automated testing tool has the following functions:

1. tested software supporting development by C/C + + language

2. Can cover all test levels, including unit test, software integration test, software and hardware integration test and system test

3. The comprehensive management can be carried out on the test items, including the information, the test cases and the test states of the test items

4. Providing a unified test description language, providing test related instructions and underlying libraries

5. Support UML, SysML modeling language

6. Automatically identifying test cases based on UML or SysML test models, automatically calculating test data according to test strategies to generate test scripts, and creating test check components and model components

7. The method supports the loop test of the model and can support the verification of an LTL verification formula on the tested model

According to the application scenario of the embodiment of the present invention, fig. 4 is a schematic diagram of an automated test operation process based on a model according to the embodiment of the present invention, and as shown in fig. 4, after a third-party modeling tool supporting the SysML language is selected to perform scenario model development and demand formalization expression, the model is imported into an automated test tool based on the model, and a test target is formulated by selecting a most appropriate test strategy according to various strategies provided in the test tool, so as to automatically generate a test case and a test stimulus. The interface adapter is completed and the test configuration is selected for generation under the code framework provided by the model-based automated test tool. And finally, running the test, opening a corresponding interface to check the test result after the test running is finished, and generating a document of the test result through the automatic test based on the model.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the embodiments provided in the present application, it should be understood that the disclosed technology can be implemented in other ways. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units may be a logical division, and in actual implementation, there may be another division, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a removable hard disk, a magnetic or optical disk, and other various media capable of storing program codes.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. A model-based automated testing method, comprising:

acquiring external environment definition information;

constructing a scene model according to the external environment definition information;

determining a test target through the scene model;

and analyzing the test target to obtain a test result.

2. The method of claim 1, wherein the obtaining external environment definition information comprises:

determining test scene data;

and analyzing the functional requirements according to the test scene data to obtain the external environment definition information.

3. The method of claim 2, wherein determining, by the context model, a test target comprises:

determination of the test target by means of a lightweight configuration, or

Taking the functional requirement as the test target.

4. The method of claim 3, wherein prior to said analyzing said test object for a test result, said method further comprises:

judging whether the test target is the functional requirement or not;

when the test target is the functional requirement, executing a preset analysis mode.

5. The method of claim 4, wherein the predetermined analysis manner comprises:

if the functional requirements are state combinations, when all state tests pass, the test result of the functional requirements is that the tests pass;

if the functional requirement is a temporal logic, the test result of the functional requirement is a test pass when the result of the logic expression is true.

6. A model-based automated test system, comprising:

the acquisition module is used for acquiring external environment definition information;

the construction module is used for constructing a scene model according to the external environment definition information;

the determining module is used for determining a test target through the scene model;

and the analysis module is used for analyzing the test target to obtain a test result.

7. The system of claim 6, wherein the acquisition module comprises:

the determining unit is used for determining test scene data;

and the analysis unit is used for analyzing the functional requirements according to the test scene data to obtain the external environment definition information.

8. The system of claim 7, wherein the determining module comprises:

a first judgment unit for determination of the test target by a lightweight configuration, or

The second judging unit is used for taking the functional requirement as the test target.

9. The system of claim 8, further comprising:

the judging module is used for judging whether the test target is the functional requirement or not;

and the analysis module is also used for executing a preset analysis mode when the test target is the functional requirement.

10. The system of claim 9, wherein the predetermined analysis manner comprises: