Background technology
In the R﹠D process of product, after product enters the system testing stage,, need comprehensively test its function and performance under the actual environment for use of simulation in order to guarantee the quality of product.According to the defective of the product of being found in test process, the developer revises product design.For guaranteeing can not introduce new mistake, need repeatedly product to be carried out regression test to the modification of inherited error.Each test of product all need manually be built environment by the tester and be simulated actual service condition, and we are referred to as to construct test environment.In order to guarantee the accurate of test result, also need finish the measurement of the every index of system by use test instrument/instrument.Use for convenience, most of instrument/instrument provide the graphical operation interface, and some adopts embedded OS, some in addition the Control Software Package that can install on computers is provided.
And the carrying out of script (as TCL, PYTHON etc.) such as explaining, can embed, extendible etc. characteristic lays a good foundation for the realization automatic test.Being write the product (as telecommunication apparatus or software) that forms by these scripts not only provides graphic user interface (Graphical User Interfaces GUI), simultaneously also provides command line interface or man-machine interface, so can carry out automatic test.And in existing automatic test script design, have all multi-modes, drive the test script Design Mode as linear test script Design Mode, structuring test script Design Mode, shared test script Design Mode or key word.
For example, based on the Robot tool platform of Rational company, just can realize that the key word in the automatic test of a kind of GUI of being applied to drives method of testing.The function that whole process comprises of this method of testing all is to be driven by key word, and whole test process controlled in key word.Please, can more clearly understand the realization mechanism and the process of this method of testing in conjunction with Figure 1 and Figure 2.In this method of testing, the test person tool can at first use the form of similar Excel worksheet, sets up test case in the mode of input key word (Key-Word).In Fig. 1, be exactly the example of a kind of test case (as KeyWords_Web.xls), it is made up of key word and parameter thereof, wherein, first row are key words, the action that explanation will be carried out, and it comprises such as input (ENTER), action (ACTION) and verification (VERIFY) or the like; Second to the 4th row are necessary parameters of action of carrying out first row; And whether the 5th row have indicated each testing procedure and have passed through.
Test case is in operational process, drive script (Drive TSL script) operation initialization script (LogonTSL script), for initial work is carried out in whole test, and be written into control documents (Run.txt), obtain the current test case that will move (* .xls).
Drive script (Drive TSL script) after initial work is finished, start-up control script (ControllerTSL script), come responsible key word of resolving whole test case file by control script (Controller TSL script), call corresponding with it realization script (User Defined TSLscript) according to these key words.
Realize actual each test action of execution of script, and use other row, after executing execution result is returned to the control script as parameter.When controlling script executing delegation to the end, whole test case just finishes to carry out; All use-case files (* .xls) operation finishes in control documents (Run.txt), and this test process finishes.
But the automated testing method that existing this key word drives has following weak point:
Existing technology is mainly used in the GUI automatic test, and based on the Robot tool platform of Rational company, and its cross-platform (refering in particular to operating system) and stride a little less than the transplantability of tool platform is not suitable for being applied to the automatic test of communication facilities series products; Test data is not separated with test logic, and its machine-processed immature to the foundation of test environment and recovery, and in a single day test data changes, and will cause the maintenance cost height; The test logic redundancy, sharing is relatively poor; Can not load required product function library according to actual needs, transplant the cost height, dirigibility is relatively poor; Loading too much product function library simultaneously can cause whole automatic test course to become slow.
Summary of the invention
The present invention is directed to the prior art weak point a kind of automatization test system and method for data-driven are provided, its test case can be cross-platform/strides tool platform to transplant; Its test environment can be shared and reuse; The loading efficiency of function library is high and possess the ability that configuration/control the 3rd testing tool is tested.
In order to address the above problem, the technical solution used in the present invention is: the automatization test system that a kind of data-driven is provided, it includes testing tool and testing tool and equipment under test, it is characterized in that, further include the test platform that is used to drive described testing tool and testing tool, and the function library that links to each other with this test platform; Test environment generation module, test case generation module, test logic generation module and test suite generation module that described test platform includes a runtime engine at least and links to each other with this runtime engine;
Runtime engine can from test environment generation module, test logic generation module, and the test case generation module read test environment, test logic, test case data, and loading function storehouse drives test logic to tested testing equipment;
The test suite generation module is used for generating according to the test result of test case the test suite of this test, described test suite is used for test case is classified, and comprises the operation information that this carries out unsanctioned test environment, test logic, test case data information and test script at least.
Wherein, described function library includes product function library, testing tool function library and testing tool function library; The test environment of this test can be set up or recover to the test environment generation module, and definition has needs the function library that loads and the condition of setting up and the recovery condition of test environment under global data required in the test, the test environment; The test case generation module, can generate test case according to test case data, this test case data define function library that the required test environment of this test logic, this test logic move required loading, and test process in the required test data of test logic; The test logic generation module is used to generate the test logic of this test, and this test logic is used to define testing process, used testing tool of synchronism detection and testing tool, and according to the test case data that runtime engine provides, equipment under test is tested;
Wherein, described test platform further includes the operational factor generation module, be used to generate the operational factor that the test log output condition is provided and the test report output condition is provided, after end of test (EOT), described runtime engine is according to defined output condition output test log of operational factor generation module and test report.
Wherein, described test report is used for representing the situation of the passing through statistics of these all test cases of test, it can comprise test data statistics, the test case inventory of each test result and the detecting information of each test case, and this test data statistics comprises: the test logic script sum of test time, operation, the test case sum of operation, execution result are respectively the test case sum of OK, POK, NG, NT; This test case inventory and detecting information comprise: test logic numbering, execution number of times, automatic test cases numbering, system testing use-case numbering, execution time, test environment, automatic test cases title.
Wherein, described test log is used for writing down the implementation status of the test case of this test process, and it records each step of this test, the start time and the time spent information of each step, and the error message when carrying out failure.
Wherein, described testing tool and testing tool provide testing tool function library and testing tool function library respectively, and described testing tool and testing tool can be integrated in the test platform.
Wherein, described product function library, testing tool function library or testing tool function library are configured according to test environment and test case data, are loaded in test process by runtime engine, or load by test platform is static.
The present invention also provides a kind of automated testing method of data-driven, and it is used at test platform equipment under test being carried out automatic test, and it comprises the steps: the test logic that the test platform selection will move, and starts test; Runtime engine carries out initialization according to the operational factor that the operational factor generation module is generated; Runtime engine is set up the operation tree of test environment and test logic according to the test logic of this operation; Runtime engine begins to carry out this test according to the operation tree of test environment and test logic; The described environment of each layer test set is set up the described environment of each layer test set, is reached the privately owned environment of orlop test logic; Carry out undermost test logic successively; Recover the privately owned environment and the described environment of each layer test set of orlop test logic.
Wherein, the step of carrying out undermost test logic also comprises: load the required product function library of test logic; Load the required testing tool function library of test logic; Take out each bar test case data, carry out test logic; The daily record of test logic is carried out in enlightenment; Generate the test case reports on the implementation of test logic.
Beneficial effect of the present invention is: in the present invention, runtime engine is by realizing possessing cross-platform, as to stride tool platform transplanting ability such as pure TCL script; In automatic test course, test environment and test logic can be shared and reuse, and can drive identical test logic in different test environments; By the test logic configuration file, can specify the function library of the testing tool/instrument/product that will load, the loading efficiency of function library is improved; Test logic and test data are separated, and make that the maintenance cost of test case is lower; Test data is represented with form, makes that the expansion of test case is more quick; After a robotization executes, if the test case that has not execution to pass through, then generate a test suite by runtime engine, comprise this and carry out information such as unsanctioned test environment, test logic, test case data, only need carry out test suite when carry out next time like this, just can carry out unsanctioned test case, improve the efficient of test.
Embodiment
At first technical term involved in the present invention is described as follows:
Test set (Test Cluster), it is the same test environment configurations and the networking configuration of some test logics, is test environment division logically, is the minimum material of forming test environment.
Test logic (Test Logic), it is the identical testing process of some test cases, the script describing of one group of test case that promptly testing process is identical, test data is different, test logic can move under different test environments.
Test case data (TestCaseData), its value by different test data items is formed, and each of the value of test data item makes up, and has constituted the test case data of different test cases.
Test case (TestCase), each combination of the value of the test data item of composition test case data, the test logic corresponding with test case data is called a test case together.
Test suite (TestSuite), by product performance, module or other (basic function, systemic-function) differentiation, the set of some test environments, test logic, test case data operation order, number of run.
Tool Command Language (tool command language, TCL), a kind of explanation execution script language.
As shown in Figure 3, be the structural representation of the automatization test system of data-driven among the present invention.Therefrom as can be seen, this test macro mainly includes test platform 1, testing tool and testing tool 3, equipment under test 4.And the function library 2 that includes runtime engine 10 and be connected with this runtime engine 10.
Wherein, test platform 1 can be any one test platform that is integrated with script interpreter (as TCL), the also Shell (as TCL Shell) of script, and it can manage, edit test script, and starts test script and begin test.Test script comprises test environment, test logic, test case data.
Testing tool and testing tool 3 are third party's program, and it is used to test equipment under test 4, and both can also be integrated in the test platform 1 for this, accept the centralized management of test platform 1.
Runtime engine 10 logically belongs to test platform 1, finishes adaptive between test platform 1 and the test script, according to the configuration in the test script, loads required function library 2, and the control whole test process;
In test process, test script is set up test environment, and tests equipment under test 4 by script control third party's testing tool and testing tool 3; In test process, between the testing tool 3, between the testing tool 3 and the coordination between testing tool 3 and the testing tool 3 come synchronously by test script.
As shown in Figure 4, be the more detailed functional block diagram of the automatization test system of data-driven among the present invention.It includes test platform 1, reaches the function library 2, testing tool/instrument 3 and the equipment under test 4 that link to each other with test platform 1 respectively.Wherein test platform 1 further includes runtime engine 10, test environment generation module 11, test case generation module 12, operational factor generation module 13, test suite generation module 14 and test logic generation module 15 again, wherein, test suite generation module 14 is optional module.Respectively each functional module is described below:
Function library 2 can include multiple function library, as product function library, testing tool function library and testing tool function library etc.Wherein, the product function library is used for configuration and control product; The testing tool function library is used for configuration and control testing tool; The testing tool function library is used for configuration and control testing tool.These function libraries are configured according to test environment and test case data, are loaded in test process by runtime engine, also can be configured to load by test platform is static.
And testing tool and testing tool 3, it can provide above-mentioned testing tool function library or testing tool function library, accepts the configuration and the control of test environment and test logic, and finishes the test to equipment under test 4.
Equipment under test 4 is an equipment tested in the test process, and it accepts the configuration and the control of test environment, accepts configuration, control and the test of test logic.
Test environment generation module 11 is mainly used in the foundation and the recovery of the test environment of finishing this test.This test environment comprises equipment under test and peripheral equipment 4, testing tool/testing tool 3; Required global data (variable) in its definition test; Definition needs the function library that loads under the test environment; The condition of setting up of definition test environment (as the initialization of equipment under test and peripheral equipment, testing tool/testing tool etc.); The recovery condition (as the release of equipment under test and peripheral equipment 4 resources, the release of testing tool/testing tool 3 public resources such as grade etc.) of definition test environment.
Test case generation module 12 is mainly used in according to test case data and generates test case, and wherein, this test case data definition test logic need in which test environment operation; The definition test logic moves necessary peculiar test environment; Definition test logic operation institute must loading specific function library; Be defined in the test process test data that test logic is required; Be used for runtime engine 10 driving test logics and finish test.
Operational factor generation module 13 is mainly used in the generating run parameter, and this operational factor provides the test log output condition and the test report output condition is provided.
Test logic generation module 15 is used to generate the test logic of this test, and this test logic is used to define testing process; In test process, used testing tool of synchronism detection and testing tool 3; According to the test case data that runtime engine 10 provides, finish test to equipment under test 4.
Test suite generation module 14 is used to generate the test suite of this test.Described test suite comprises this and carries out information such as unsanctioned test environment, test logic, test case data.Test suite is test script division logically, can reduce basic function test suite, characteristic test cover or the like.Content only in test suite: move which test script (test environment, test logic, test case data), and the operation information (as operation order, number of run) of these test scripts (test environment, test logic, test case data).
Runtime engine 10, it finishes the editor of test logic, test environment, test case data at test platform 1, and starts after the test of current selected test logic, the execution of control test script.Runtime engine can from test environment generation module 11, test logic generation module 15, and test case generation module 12 read test environment, test logic, test case data (and before the test beginning, set up test environment; After the end of test (EOT), recover test environment); The loading function storehouse, the version number in automatic recognition function storehouse, renewal function index file; In the test process,, drive test logic equipment under test 4 is tested according to test case data; In test process, according to operational factor, output test report and test log; According to the test result of test case, drive test suite generation module 14 and generate the test suite that this tests unsanctioned test case composition, convenient follow-up test once more unsanctioned test case.
Test report is after end of test (EOT), is provided by runtime engine 10.After this test report is used to represent this end of test (EOT), the statistics of the situation of passing through of all test cases in this test.As shown in Figure 5, be the example of a kind of test report among the present invention.Therefrom as can be seen, this test report has been listed the test data statistics, this test data statistics comprise such as: the test logic script sum of test time, operation, the test case sum of operation, execution result are respectively OK, part of O K (partial OK, POK), bad (NotGood, NG), do not test (Not Test, NT) the test case sum of Denging.And list the details of above-mentioned different execution results.
In the expression test process, write down the implementation status of the test case of current operation by runtime engine 10; After the end of test (EOT), by the implementation status (being test log) of the complete test case of runtime engine 10 outputs; As shown in Figure 6, be the format sample of a kind of test log among the present invention, it has at length write down each step of this test, and information such as the start time of each step and time spent.
As shown in Figure 7, be the process flow diagram of the method for data driven automation test of the present invention.In step S70, the test logic that the test platform selection will move, and start test.In step S71, runtime engine is finished initial work according to the operational factor that the operational factor generation module is generated.In step S72, runtime engine is set up the operation tree of test environment and test logic according to the test logic of this operation, and the operation tree and the operation explanation of this test environment and test logic can be referring to shown in Figure 8.In step S74, runtime engine begins to carry out this test according to the operation tree of test environment and test logic.At first in step S76, set up the described environment of test set of each layer, with reference to operation tree shown in Figure 8, promptly successively: carry out function 1 and set up environment TestCluster1; Carry out function 2, set up environment TestCluster11; Carry out function 3: set up environment TestCluster111; Carry out function 4: the privately owned environment of setting up test logic TestLogic1.And in step S78, carry out undermost test logic successively, promptly carry out TestLogic1; Carry out function D: the privately owned environment that recovers test logic TestLogic1; In like manner, continue to carry out function 5: the privately owned environment of setting up test logic TestLogic2; Carry out TestLogic2; Carry out function E: the privately owned environment that recovers test logic TestLogic2; Carry out function 6: set up the privately owned environment of test logic TestLogic3 and carry out TestLogic3; Carry out function F: the privately owned environment that recovers test logic TestLogic3.Then, in step S709, recover the described environment of each layer test set in the other direction.Promptly carry out function C: recover environment TestCluster111; Carry out function 7: set up environment TestCluster112, and carry out the test logic that under test set TestCluster112, moves; Carry out function G: recover environment TestCluster112; Carry out function 8: set up environment TestCluster113, and carry out the test logic that under test set TestCluster113, moves; Carry out function H: recover environment TestCluster113; Carry out function B: recover environment TestCluster11; Carry out function 9: set up environment TestCluster12, and carry out the test logic of TestCluster12: carry out function I: recover environment TestCluster12.Carry out function A: recover environment TestCluster1.
As shown in Figure 9, be the process flow diagram of carrying out the orlop test logic among Fig. 8 of the present invention.In this Fig. 9, be to be that example describes to carry out TestLogic1.In step S780, load the required product function library of test logic; In step S782, load the required testing tool function library of test logic; In step S784, take out each bar test case data, carry out test logic; In step S786, the daily record of test logic is carried out in enlightenment; In step S788, generate the test case reports on the implementation of test logic.The process of undermost other test logics (such as TestLogic2, TestLogic2....TestLogic5) all similarly.
Wherein, from operation tree as can be seen, test logic TestLogic2 carries out under two environment, reaches the purpose that test logic is shared in test process, and corresponding data item is represented in the different test results use-case data under varying environment.
As shown in figure 10, be the physical storage structure synoptic diagram of data driven automation test among the present invention.The storage organization of the data driven automation test of Huawei Company is made up of five parts: function library (Lib), test script library (Scripts), test suite (TestSuites), test report (Report), test log (Log);
Wherein, function library can be divided into public function storehouse of dividing by testing tool, testing tool and the product function library of dividing by product again, the corresponding function library of each product.
Test script library also is to divide by product; Deposit test environment, test logic and the test case data of product in this test script library; And deposit the environmental parameter configuration file of data driven automation test system.
The test suite storehouse also is to divide by product; Deposit the test suite of product.
The test report catalogue also is to divide by product; Test process of every startup produces the test report of this test process, writes down the situation of passing through of the test case of moving in this test process.
The test log catalogue also is to divide by product; Test process of every startup produces the test log of this test process, writes down the operational process of test case in this test process.
In the present invention, the realization of runtime engine also can pass through other script (as Python etc.) or compiler language (as C/C++, Pascal etc.) realization except TCL realizes; The physical storage structure of data driven automation test system also can be realized by database (as Oracle, SQL Server etc.).
In the present invention, runtime engine is by realizing possessing cross-platform, as to stride tool platform transplanting ability such as pure TCL script; In the automatic test course, test environment and test logic can be shared and reuse, and can drive identical test logic in different test environments; By the test logic configuration file, can specify the function library of the testing tool/instrument/product that will load, the loading efficiency of function library is improved; Test logic and test data are separated, and make that the maintenance cost of test case is lower; Test data is represented with form, makes that the expansion of test case is more quick; After a robotization executes, if unsanctioned test case is arranged, then generate a test suite by runtime engine, comprise this and carry out information such as unsanctioned test environment, test logic, test case data, only need carry out this test suite when carry out next time like this, just can carry out unsanctioned test case, improve the efficient of test.