CN117687909A - Operator test case generation method, system, equipment and medium - Google Patents
Operator test case generation method, system, equipment and medium Download PDFInfo
- Publication number
- CN117687909A CN117687909A CN202311668240.4A CN202311668240A CN117687909A CN 117687909 A CN117687909 A CN 117687909A CN 202311668240 A CN202311668240 A CN 202311668240A CN 117687909 A CN117687909 A CN 117687909A
- Authority
- CN
- China
- Prior art keywords
- operator
- test
- information
- test case
- parameter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012360 testing method Methods 0.000 title claims abstract description 419
- 238000000034 method Methods 0.000 title claims abstract description 59
- 238000011990 functional testing Methods 0.000 claims abstract description 13
- 238000012795 verification Methods 0.000 claims description 92
- 238000004590 computer program Methods 0.000 claims description 10
- 238000005516 engineering process Methods 0.000 claims description 10
- 238000004422 calculation algorithm Methods 0.000 claims description 6
- 230000003247 decreasing effect Effects 0.000 claims description 4
- 238000012545 processing Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 7
- 238000010276 construction Methods 0.000 description 5
- 238000010586 diagram Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000004913 activation Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000009411 base construction Methods 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000013522 software testing Methods 0.000 description 1
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The disclosure provides a method, a system, a device and a medium for generating test cases of operators, wherein the method for generating the test cases comprises the following steps: acquiring frame information of an open source frame in an AI software stack and operator information corresponding to each operator in the open source frame; generating a test case template corresponding to the operator information based on the frame information and the operator information; obtaining a plurality of initial operator parameter information based on operator reference test cases of different versions; acquiring target version information of a test case matched with an open source framework; obtaining a plurality of pieces of target parameter operator information corresponding to the preset test requirements based on the target version information and the preset test requirements; generating a plurality of first test cases corresponding to preset test requirements based on the target parameter operator information and the test case template; and obtaining a plurality of target test cases for performing functional test on each operator in the open source framework according to a plurality of first test cases under different preset test requirements.
Description
Technical Field
The disclosure relates to the field of software testing, and in particular relates to a method, a system, equipment and a medium for generating test cases of operators.
Background
Operators: the AI (Artificial Intelligence ) model consists of individual computational units, commonly referred to as Operators (OP), corresponding to some computational logic, such as: relu is an operator used as an activation function.
AI software stack: refers to a series of software components and tool sets that make up an artificial intelligence system for supporting the development, deployment and operation of various AI applications.
The operator test cases of the existing AI software stack are mainly concentrated and applicable to operators in official operator models or some open source models, the operator range is relatively limited, and meanwhile, the operator model is collected or constructed manually for a long time.
In addition, in the AI-oriented deployment software stack, the existing supporting frames (onnx, pyrach, tf and the like) have the problems that the operator test cases are not comprehensive enough, the cost for collecting and constructing operators is too high and the like,
the test case is incomplete, so that a plurality of hidden dangers exist in an AI software stack, operator supportability problems exist in later use, and the conditions of high maintenance and update cost and the like occur after the test case is put into use.
Disclosure of Invention
The technical problem to be solved by the present disclosure is to overcome the defect of high test case cost in the prior art, and provide a method, a system, a device and a medium for generating a test case of an operator.
The technical problems are solved by the following technical scheme:
according to a first aspect of the present disclosure, there is provided a test case generation method of an operator, the generation method including:
acquiring frame information of an open source frame in an AI software stack and operator information corresponding to each operator in the open source frame;
generating a test case template corresponding to the operator information based on the frame information and the operator information;
obtaining a plurality of initial operator parameter information based on operator reference test cases of different versions;
acquiring target version information of a test case matched with the open source framework;
obtaining a plurality of pieces of target parameter operator information corresponding to the preset test requirements based on the target version information and the preset test requirements;
generating a plurality of first test cases corresponding to the preset test requirements based on the target parameter operator information and the test case template;
and obtaining a plurality of target test cases for performing functional test on each operator in the open source framework according to a plurality of first test cases corresponding to different preset test requirements.
Preferably, the step of obtaining the plurality of initial operator parameter information based on the operator reference test cases of different versions includes:
Processing the operator reference test cases by adopting a crawler technology and/or a recommendation algorithm technology to obtain a plurality of initial operator parameter information;
wherein the initial operator parameter information includes: initial version information and initial parameter information.
Preferably, the test case template comprises a plurality of pieces of first operator parameter information;
the step of generating a plurality of first test cases corresponding to the preset test requirement based on the target parameter operator information and the test case template comprises the following steps:
when the preset test requirement is a full-random test requirement, selecting at least two pieces of first operator parameter information in the test case template to be randomly combined to obtain first parameter combination information, and assigning values to the first parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the full-random test requirement;
when the preset test requirement is a part of the selected test requirement, selecting two pieces of first operator parameter information in the test case template and taking the two pieces of first operator parameter information as first fixed parameter information, randomly combining the first fixed parameter information with other pieces of first operator parameter information in the test case template to obtain second parameter combination information, and assigning the second parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the part of the selected test requirement;
When the preset test requirement is a single-choice test requirement, selecting one piece of first operator parameter information in the test case template and taking the first operator parameter information as second fixed parameter information, randomly combining the second fixed parameter information with other pieces of first operator parameter information in the test case template to obtain third parameter combination information, and assigning values to the third parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the single-choice test requirement.
Preferably, the step of obtaining a plurality of target test cases for performing functional tests on each operator in the open source framework further includes:
verifying whether each first test case is suitable for the operator in the open source framework by adopting a preset verification rule, if so, reserving the corresponding first test case; if not, deleting the corresponding first test case;
the preset verification rule comprises verification party modes with different verification levels, and the first test cases are verified in sequence by adopting the verification modes with different verification levels decreasing in sequence.
Preferably, the preset verification rule includes: a structure verification mode and a numerical verification mode;
the step of adopting the structure verification mode for verification comprises the following steps:
determining a verification model corresponding to the operator according to the open source framework;
inputting the first test case into the verification model, and judging whether the verification model can run the first test case or not;
if yes, entering the numerical verification mode;
if not, deleting the first test case;
the step of verifying by adopting the numerical verification mode comprises the following steps:
determining verification input information and verification output information according to the operator information;
inputting the verification input information into the first test case to obtain a first test output;
judging whether the first test output is consistent with the verification output information or not;
if the first test cases are consistent, storing and outputting the corresponding first test cases;
and if the first test cases are inconsistent, deleting the corresponding first test cases.
Preferably, the generating method further comprises:
constructing a parameter information base according to the plurality of initial operator parameter information;
the generating method further comprises the following steps:
after deleting the first test case, searching the parameter information base according to the first test case, and determining the target parameter operator information corresponding to the first test case;
Deleting the target parameter operator information corresponding to the first test case in the parameter information base.
According to a second aspect of the present disclosure, there is provided a test case generating system of an operator, the generating system including:
the operator information acquisition module is used for acquiring frame information of an open source frame in the AI software stack and operator information corresponding to each operator in the open source frame;
the test case template generation module is used for generating a test case template corresponding to the operator information based on the frame information and the operator information;
the initial parameter information acquisition module is used for acquiring a plurality of initial operator parameter information based on operator reference test cases of different versions;
the target version acquisition module is used for acquiring target version information of the test case matched with the open source framework;
the target parameter information acquisition module is used for acquiring a plurality of pieces of target parameter operator information corresponding to the test requirements based on the target version information and the preset test requirements;
the first test case generation module is used for generating a plurality of first test cases corresponding to the preset test requirements based on the target parameter operator information and the test case template;
The target test case generation module is used for obtaining a plurality of target test cases for performing functional tests on each operator in the open source framework according to a plurality of first test cases under different preset test requirements.
According to a third aspect of the present disclosure, there is provided an electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, the processor implementing the test case generation method of the operator according to the first aspect of the present disclosure when executing the computer program.
According to a fourth aspect of the present disclosure, there is provided a chip applied to an electronic device, where the chip is configured to execute the test case generating method of the operator according to the first aspect of the present disclosure.
According to a fifth aspect of the present disclosure, a chip module is provided, and the chip module is applied to an electronic device, and includes a transceiver component and a chip, where the chip is configured to execute the test case generation method of the operator according to the first aspect of the present disclosure. According to a sixth aspect of the present disclosure, there is provided a computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the test case generation method of the operator of the first aspect of the present disclosure.
On the basis of conforming to the common knowledge in the art, the preferred conditions can be arbitrarily combined to obtain the preferred embodiments of the present disclosure.
The positive progress effect of the present disclosure is:
according to the test case generation method of the operator, the test case template of the operator in the open source framework is generated, so that corresponding first operator parameter information can be selected in the test case template based on preset test requirements, corresponding initial parameter information is generated through operator reference test cases under different versions acquired in advance, target parameter information is selected in the initial parameter information according to target version information to be assigned to the first operator parameter information, a plurality of first test cases corresponding to the preset test requirements are generated, and the test cases corresponding to the operator are generated rapidly, so that the test efficiency of the operator is improved; the test case generation method of the operator provided by the disclosure can also realize the whole coverage of the test case for generating the corresponding operator information, and reduce the construction cost of the test case.
Further, according to the plurality of first test cases under different preset test requirements, a plurality of target test cases for performing functional test on each operator in the open source framework are generated, and the completeness test on the operator functions is realized through the target test cases.
Drawings
FIG. 1 is a flow chart of a method for generating test cases of operators in an embodiment of the disclosure;
FIG. 2 is a flowchart illustrating a first test case generation in an embodiment of the present disclosure;
FIG. 3 is a flow chart illustrating the structure verification in an embodiment of the present disclosure;
FIG. 4 is a flow chart of performing numerical verification in an embodiment of the disclosure;
FIG. 5 is a schematic flow chart of a test case generation method for implementing operators in an embodiment of the disclosure;
FIG. 6 is a schematic diagram of a test case generation system of an operator according to an embodiment of the disclosure;
fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the disclosure.
Detailed Description
The present disclosure is further illustrated by way of examples below, but is not thereby limited to the scope of the examples described.
Example 1
The operator test cases in the existing AI software stack are mainly aimed at operators in some official operator models or some open source models, the range of the operator test cases is relatively limited, meanwhile, the time for collecting or manually constructing the operator test cases is long, and a complete method for automatically constructing the operator test cases is lacked.
In order to solve the problem of incomplete operator test cases in the support of each open source framework operator in an AI software stack, a method for automatically generating the operator function completeness test cases for the AI software stack is designed based on each AI framework and Python script, and comprises the steps of generating an operator template, establishing a parameter information base, and simultaneously combining and generating the operator cases. Specific:
As shown in fig. 1, in a specific embodiment of the present disclosure, a test case generating method of an operator is provided, where the generating method includes:
s11: and acquiring frame information of an open source frame in the AI software stack and operator information corresponding to each operator in the open source frame.
S12: based on the frame information and the operator information, generating a test case template corresponding to the operator information.
S13: and obtaining a plurality of initial operator parameter information based on operator reference test cases of different versions.
S14: and acquiring target version information of the test case matched with the open source framework.
S15: and obtaining a plurality of pieces of target parameter operator information corresponding to the preset test requirements based on the target version information and the preset test requirements.
S16: and generating a plurality of first test cases corresponding to the preset test requirements based on the target parameter operator information and the test case template.
S17: and obtaining a plurality of target test cases for performing functional test on each operator in the open source framework according to a plurality of first test cases corresponding to different preset test requirements.
The operator test case generation method provided by the disclosure can be used for supporting tests of AI software stacks on different frame operators, is wide in coverage range of construction cases, can relate to dimensions such as different versions, attribute combinations, input and output types and the like, is rapid and convenient, and saves cost of collecting operator cases by manpower.
According to the test case generation method of the operator, the test case template of the operator in the open source framework is generated, so that corresponding first operator parameter information can be selected in the test case template based on preset test requirements, corresponding initial parameter information is generated through operator reference test cases under different versions acquired in advance, target parameter information is selected in the initial parameter information according to target version information to be assigned to the first operator parameter information, a plurality of first test cases corresponding to the preset test requirements are generated, and the test cases corresponding to the operator are generated rapidly, so that the test efficiency of the operator is improved; the test case generation method of the operator provided by the disclosure can also realize the whole coverage of the test case for generating the corresponding operator information, and reduce the construction cost of the test case.
The test template in the embodiment of the disclosure comprises: the type of the test case, and information such as the name, the data type, the data arrangement format, the attribute and the like of the input and output. Test case templates in embodiments of the present disclosure are saved using json.
In addition, in a specific implementation manner of the embodiment of the disclosure, a test case template can be automatically generated according to a support supporting the template by designing a definition manner of the template, and different test cases of the corresponding operators are generated through the test case template.
In this embodiment of the present disclosure, the generating the test case template corresponding to the operator information is a mature technology based on the frame information and the operator information, and therefore will not be described herein in detail.
In the embodiment of the disclosure, by adopting a crawler technology and/or a recommendation algorithm technology, operator reference test cases of different versions are processed to obtain a plurality of initial operator parameter information; wherein the initial operator parameter information includes: initial version information and initial parameter information.
The generating method in the embodiment of the disclosure further comprises the following steps: and constructing a parameter information base according to the obtained plurality of initial operator parameter information, so that the assignment of the test case template can be conveniently and rapidly realized.
In one particular embodiment of the present disclosure, different versions of operator reference tests are collected from various sources (e.g., official documents, forums, blogs, etc.) using crawler technology, and initial operator parameter information corresponding to the different versions is obtained. The crawler can grab operator use cases according to specific webpage structures and rules, extract information such as parameter names, types, default values, value ranges and the like of operators, and store acquired data into a parameter database or a file.
In another specific embodiment of the present disclosure, a recommendation algorithm may be designed to generate initial operator parameter information corresponding to different versions, depending on the function and implementation of the operator. By using a content-based recommendation algorithm, such as based on attributes of interest to the user, application scenarios, etc., to recommend relevant initial operator parameter information.
The test case template in the embodiment of the disclosure comprises a plurality of pieces of first operator parameter information; therefore, the operator templates can be automatically generated according to the supported framework, and test case combinations of different corresponding operators can be generated.
The preset test requirements in the embodiments of the present disclosure include: full random test requirements, partial selection test requirements, single selection test requirements.
When the preset test requirement is a full-random test requirement, at least two pieces of first operator parameter information are selected from the test case template to be randomly combined to obtain first parameter combination information, and the first parameter combination information is assigned according to the target parameter operator information to generate a plurality of first test cases corresponding to the full-random test requirement.
In one specific embodiment of the present disclosure, the full random test requirements are: the test needs to cover all the structure, attribute values and types of the operator, and arrange and combine the operator and the operator to construct a large number of test cases with different operators.
Specific applicable scenarios, such as when the AI software stack adds a new operator support, e.g. opset7averagepool under the onnx framework (an open source framework), there are 5 attributes and one input and output, and the full test would combine between 5 attributes to produce 2 5 The 32 structural combinations are then filled with parameter values of each attribute to form test cases of several full random combinations of corresponding opset7averagepool operators under the onnx framework.
When the preset test requirement is a part of the selected test requirement, selecting two pieces of first operator parameter information in the test case template and taking the two pieces of first operator parameter information as first fixed parameter information, randomly combining the first fixed parameter information with other pieces of first operator parameter information in the test case template to obtain second parameter combination information, and assigning values to the second parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the part of the selected test requirement.
In one specific embodiment of the present disclosure, the partial selection test requirements are: and designating a test range, combining attributes in the range to generate a use case, controlling the constructed use case range, and adopting the test requirement when version updating only aims at part of the attributes.
Specific application scenarios include: the operator structure of key attributes, for example, an onnx operator opset7 averagepool operator, has 5 attributes, wherein kernel_shape (one attribute of the operator) and pads (one attribute of the operator) are key attributes, different test cases of different structures containing the two attributes are generated, and meanwhile, multiple groups of parameters are selected for testing, and kernel_shape and pads support and boundary values thereof are tested.
When the preset test requirement is a single-choice test requirement, selecting one piece of first operator parameter information in the test case template and taking the first operator parameter information as second fixed parameter information, randomly combining the second fixed parameter information with other pieces of first operator parameter information in the test case template to obtain third parameter combination information, and assigning values to the third parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the single-choice test requirement.
In one specific embodiment of the present disclosure, the single choice test requirement is: and selecting a designated attribute of the test case template, and generating all combinations containing the attribute so as to realize the support condition of the key test on the attribute.
Specific application scenarios include: when the operator performs version update, a certain attribute is newly added, and the AI software stack adds the support of the newly added attribute, because other attributes are tested before, only the structure of the combination of the newly added attribute and other attributes needs to be emphasized to be tested in order to save test time, for example, the onnx averagepool operator opset10 version is one more ceil_mode (representing that the calculated output tshape is rounded up and down), and only the averagepool operator containing the cel_model attribute needs to be constructed 1*2 5 =32 structures, if retested is required, test 2 is required 6 =64 structures, saving a lot of test time.
In the embodiment of the disclosure, through designing different preset test requirements, the operator cases can be generated in full, part and single mode, the support of the test cases on the operators is flexibly realized, and the test cases corresponding to the operators are generated in a targeted mode.
As shown in fig. 2, in a specific implementation of the embodiment of the disclosure, step S16 includes:
s21: and selecting a plurality of first operator parameter information in the test case template according to a preset test requirement.
S22: and inquiring target parameter operator information corresponding to the target version information and the first operator parameter information in the parameter information base.
S23: and assigning a value to the first operator parameter information according to the target parameter operator information so as to generate a first test case corresponding to the preset test requirement.
And generating a plurality of target test cases for carrying out functional test on each operator in the open source framework according to a plurality of first test cases under different preset test requirements, and realizing the completeness test on the operator functions through the target test cases.
Furthermore, in an embodiment of the present disclosure, the step of obtaining a plurality of target test cases for performing a functional test on each operator in the open source framework further includes: verifying whether each first test case is suitable for operators in the open source framework by adopting a preset verification rule, if so, reserving the corresponding first test case; if not, deleting the corresponding first test case.
The preset verification rule comprises verification modes with different verification levels, and the first test cases are verified in sequence by adopting the verification modes with the different verification levels decreasing in sequence.
The verification method in the embodiment of the disclosure comprises the following steps: structure verification method and numerical verification method.
As shown in fig. 3, the step of performing verification by adopting a structure verification manner in the embodiment of the disclosure includes:
S31: and determining a verification model of the corresponding operator according to the open source framework.
S32: the first test case is input into the verification model.
S33: and judging whether the verification model can run the first test case or not.
If yes, entering a numerical verification mode; if not, deleting the first test case.
As shown in fig. 4, the steps of performing verification by using a numerical verification method in the embodiment of the disclosure include:
s41: and determining verification input information and verification output information according to the operator information.
S42: and inputting the verification input information into the first test case to obtain a first test output.
S43: and judging whether the first test output and the verification output information are consistent.
If the first test cases are consistent, the first test cases are stored and output; if not, deleting the first test case.
Furthermore, in an embodiment of the present disclosure, the generating method further includes: and constructing a parameter information base according to the plurality of initial operator parameter information. When the first test output is inconsistent with the verification output information, further comprising: searching a parameter information base according to the first test case, and determining target parameter operator information corresponding to the first test case; and deleting the target parameter operator information corresponding to the first test case in the parameter information base.
According to the test case generation method of the operator, the test case template of the operator in the open source framework is generated, so that corresponding first operator parameter information can be selected in the test case template based on preset test requirements, corresponding initial parameter information is generated through operator reference test cases under different versions acquired in advance, target parameter information is selected in the initial parameter information according to target version information to be assigned to the first operator parameter information, a plurality of first test cases corresponding to the preset test requirements are generated, and the test cases corresponding to the operator are generated rapidly, so that the test efficiency of the operator is improved; the test case generation method of the operator provided by the disclosure can also realize the whole coverage of the test case for generating the corresponding operator information, and reduce the construction cost of the test case.
As shown in fig. 5, the implementation principle of the test case generation method of the operator of the present disclosure is specifically described below with reference to an example:
and determining operator information of the corresponding AI open source frame according to the corresponding AI open source frame, and determining test case information of the corresponding operator information according to the operator information.
Determining a test case template corresponding to the operator information according to the test case information; and simultaneously, generating a plurality of parameter information corresponding to the test case templates through parameter statistics (such as web crawler searching) or by a recommendation system.
Establishing a parameter information base corresponding to operator parameters according to the plurality of parameter information;
generating an operator test case of a corresponding operator under a certain test requirement according to the test case template and the parameter information base;
verifying whether the structure of the operator meets the structure of the operator under the corresponding framework or not through a verification model; if not, updating the test case template; if yes, continuing to verify whether the assignment of the operator accords with the assignment condition of the operator under the corresponding framework; if yes, the corresponding operator test cases are saved, and if not, the parameter information base is updated.
The test case generation method of the operators provided by the disclosure can be applied to the AI software stack development process, and can automatically combine and generate the test cases of the operators under the full range and corresponding appointed frames for different AI frame operator supportability tests, and complete the completeness test of the operators to the AI software stack.
According to the test case generation method of the operator, a set of full-range and automatic method for testing the support of the AI operator can be customized according to the test requirement of the AI deployment software stack, so that a user can automatically construct a test case template and a parameter information base corresponding to the AI framework and the operator type according to the appointed AI framework and the operator type, complete test operator cases are generated through combination, and meanwhile, the test case template and the parameter information base can be updated through setting a verification method.
Example 2
As shown in fig. 6, in an embodiment of the present disclosure, there is further provided a test case generating system for an operator, where the generating system includes:
the operator information obtaining module 100 is configured to obtain frame information of an open source frame in the AI software stack, and operator information corresponding to each operator in the open source frame.
The test case template generation module 200 is configured to generate a test case template corresponding to the operator information based on the frame information and the operator information.
The initial parameter information obtaining module 300 is configured to obtain a plurality of initial operator parameter information based on operator reference test cases of different versions.
The target version obtaining module 400 is configured to obtain target version information of the test case matched with the open source framework.
The target parameter information obtaining module 500 is configured to obtain a plurality of target parameter operator information corresponding to a preset test requirement based on the target version information and the preset test requirement.
The first test case generating module 600 is configured to generate a plurality of first test cases corresponding to a preset test requirement based on the target parameter operator information and the test case template.
The target test case generating module 700 is configured to obtain, according to a plurality of first test cases under different preset test requirements, a plurality of target test cases for performing functional tests on each operator in the open source framework.
The initial parameter information obtaining module 300 in the embodiment of the present disclosure is specifically configured to: processing the operator reference test cases by adopting a crawler technology and/or a recommendation algorithm technology to obtain a plurality of initial operator parameter information; wherein the initial operator parameter information includes: initial version information and initial parameter information.
The test case template in the embodiment of the disclosure comprises a plurality of pieces of first operator parameter information; the preset test requirements include: full random test requirements, partial selection test requirements, single selection test requirements.
The first test case generation module 600 is specifically configured to:
when the preset test requirement is a full-random test requirement, selecting at least two pieces of first operator parameter information in the test case template to be randomly combined to obtain first parameter combination information, and assigning values to the first parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the full-random test requirement;
when the preset test requirement is a part of the selected test requirement, selecting two pieces of first operator parameter information in the test case template and taking the two pieces of first operator parameter information as first fixed parameter information, randomly combining the first fixed parameter information with other pieces of first operator parameter information in the test case template to obtain second parameter combination information, and assigning the second parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the part of the selected test requirement;
When the preset test requirement is a single-choice test requirement, selecting one piece of first operator parameter information in the test case template and taking the first operator parameter information as second fixed parameter information, randomly combining the second fixed parameter information with other pieces of first operator parameter information in the test case template to obtain third parameter combination information, and assigning values to the third parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the single-choice test requirement.
As shown in fig. 6, the generating system in the embodiment of the present disclosure includes: the information base construction module 800 is configured to construct a parameter information base according to the plurality of initial operator parameter information.
The first test case generation module 600 in the embodiment of the present disclosure further includes: the device comprises a first selection unit, a first query unit and a generation case unit.
The first selecting unit is used for selecting a plurality of first operator parameter information in the test case template according to a preset test requirement;
the first query unit is used for querying target parameter operator information corresponding to the target version information and the first operator parameter information in the parameter information base;
The case generation unit is used for assigning a value to the first operator parameter information according to the target parameter operator information so as to generate a first test case corresponding to a preset test requirement.
As shown in fig. 6, the generating system in the embodiment of the present disclosure includes: the verification module 900 is configured to verify whether the first test case is adapted to an operator in the open source framework by using a preset verification rule, and if so, reserve the first test case; if not, deleting the first test case; the preset verification rule comprises verification modes with different verification levels, and the first test case is verified by adopting the verification modes with different verification levels gradually decreasing.
The verification module in the embodiment of the disclosure comprises a structure verification unit and a numerical verification unit.
The structure verification unit is used for realizing a structure verification mode, and particularly, the structure verification unit is used for determining a verification model of a corresponding operator according to the open source framework; inputting the first test case into a verification model, and judging whether the verification model can run the first test case or not; if yes, entering a numerical verification mode; if not, deleting the corresponding first test case.
The numerical verification unit is used for realizing a numerical verification mode, and particularly, the numerical verification unit is used for determining verification input information and verification output information according to operator information; inputting verification input information into a first test case to obtain a first test output; judging whether the first test output is consistent with the verification output information; if the first test cases are consistent, storing and outputting the corresponding first test cases; if not, deleting the corresponding first test case.
The numerical verification unit is further used for searching a parameter information base according to the first test case when the first test output is inconsistent with the verification output information, and determining target parameter operator information corresponding to the first test case; and deleting the target parameter operator information corresponding to the first test case in the parameter information base.
It should be noted that, the working principle corresponding to the test case generating system of the operator in this embodiment is the same as the working principle of the test case generating method of the operator in embodiment 1, so that a detailed description is omitted here.
In the test case generation system of the operator provided by the disclosure, by generating the test case template of the operator in the open source framework, corresponding first operator parameter information can be selected in the test case template based on a preset test requirement, corresponding initial parameter information is generated by referring to test cases of operators under different versions acquired in advance, and target parameter information is selected in the initial parameter information according to target version information to assign value to the first operator parameter information, so that a plurality of first test cases corresponding to the preset test requirement are generated, the test cases corresponding to the operator are quickly generated, and the test efficiency of the operator is improved; the test case generation method of the operator provided by the disclosure can also realize the whole coverage of the test case for generating the corresponding operator information, and reduce the construction cost of the test case.
Further, according to the plurality of first test cases under different preset test requirements, a plurality of target test cases for performing functional test on each operator in the open source framework are generated, and the completeness test on the operator functions is realized through the target test cases.
Example 3
Fig. 7 is a schematic structural diagram of an electronic device according to embodiment 3 of the present disclosure. The electronic device comprises a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the methods of the above embodiments when executing the program. The electronic device 30 shown in fig. 7 is merely an example and should not be construed to limit the functionality and scope of use of the disclosed embodiments.
As shown in fig. 7, the electronic device 30 may be in the form of a general purpose computing device, which may be a server device, for example. Components of electronic device 30 may include, but are not limited to: the at least one processor 31, the at least one memory 32, a bus 33 connecting the different system components, including the memory 32 and the processor 31.
The bus 33 includes a data bus, an address bus, and a control bus.
Memory 32 may include volatile memory such as Random Access Memory (RAM) 321 and/or cache memory 322, and may further include Read Only Memory (ROM) 323.
Memory 32 may also include a program/utility 325 having a set (at least one) of program modules 324, such program modules 324 including, but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of which may include an implementation of a network environment.
The processor 31 executes various functional applications and data processing, such as the methods of the above-described embodiments of the present disclosure, by running a computer program stored in the memory 32.
The electronic device 30 may also communicate with one or more external devices 34 (e.g., keyboard, pointing device, etc.). Such communication may be through an input/output (I/O) interface 35. Also, model-generating device 30 may also communicate with one or more networks, such as a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the internet, via network adapter 36. As shown in fig. 7, network adapter 36 communicates with the other modules of model-generating device 30 via bus 33. It should be appreciated that although not shown in the figures, other hardware and/or software modules may be used in connection with the model-generating device 30, including, but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, data backup storage systems, and the like.
It should be noted that although several units/modules or sub-units/modules of an electronic device are mentioned in the above detailed description, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more units/modules described above may be embodied in one unit/module in accordance with embodiments of the present disclosure. Conversely, the features and functions of one unit/module described above may be further divided into ones that are embodied by a plurality of units/modules.
Example 4
The disclosure also provides a chip applied to the electronic device of embodiment 3, and used for executing the test case generation method of the operator of embodiment 1.
Example 5
The disclosure further provides a chip module applied to the electronic device of embodiment 3, and configured to execute the test case generating method of the operator of embodiment 1.
Example 6
The present embodiment provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method in the above-described embodiment.
More specifically, among others, readable storage media may be employed including, but not limited to: portable disk, hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible implementation, the disclosure may also be implemented in the form of a program product comprising program code for causing a terminal device to carry out the steps of implementing the method in the embodiments described above, when the program product is run on the terminal device.
Wherein the program code for carrying out the present disclosure may be written in any combination of one or more programming languages, and the program code may be executed entirely on the user device, partially on the user device, as a stand-alone software package, partially on the user device, partially on a remote device, or entirely on the remote device.
While specific embodiments of the present disclosure have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and the scope of the disclosure is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the principles and spirit of the disclosure, but such changes and modifications fall within the scope of the disclosure.
Claims (11)
1. The test case generation method of the operator is characterized by comprising the following steps of:
Acquiring frame information of an open source frame in an AI software stack and operator information corresponding to each operator in the open source frame;
generating a test case template corresponding to the operator information based on the frame information and the operator information;
obtaining a plurality of initial operator parameter information based on operator reference test cases of different versions;
acquiring target version information of a test case matched with the open source framework;
obtaining a plurality of pieces of target parameter operator information corresponding to the preset test requirements based on the target version information and the preset test requirements;
generating a plurality of first test cases corresponding to the preset test requirements based on the target parameter operator information and the test case template;
and obtaining a plurality of target test cases for performing functional test on each operator in the open source framework according to a plurality of first test cases corresponding to different preset test requirements.
2. The method for generating test cases of operators according to claim 1, wherein the step of obtaining a plurality of initial operator parameter information based on different versions of operator reference test cases comprises:
processing the operator reference test cases by adopting a crawler technology and/or a recommendation algorithm technology to obtain a plurality of initial operator parameter information;
The initial operator parameter information comprises initial version information and initial parameter information.
3. The method for generating test cases of operators according to claim 1, wherein the test case template comprises a plurality of first operator parameter information;
the step of generating a plurality of first test cases corresponding to the preset test requirement based on the target parameter operator information and the test case template comprises the following steps:
when the preset test requirement is a full-random test requirement, selecting at least two pieces of first operator parameter information in the test case template to be randomly combined to obtain first parameter combination information, and assigning values to the first parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the full-random test requirement;
when the preset test requirement is a part of the selected test requirement, selecting two pieces of first operator parameter information in the test case template and taking the two pieces of first operator parameter information as first fixed parameter information, randomly combining the first fixed parameter information with other pieces of first operator parameter information in the test case template to obtain second parameter combination information, and assigning the second parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the part of the selected test requirement;
When the preset test requirement is a single-choice test requirement, selecting one piece of first operator parameter information in the test case template and taking the first operator parameter information as second fixed parameter information, randomly combining the second fixed parameter information with other pieces of first operator parameter information in the test case template to obtain third parameter combination information, and assigning values to the third parameter combination information according to the target parameter operator information to generate a plurality of first test cases corresponding to the single-choice test requirement.
4. The method for generating test cases for operators according to claim 3, wherein after the step of obtaining a plurality of target test cases for performing functional tests on each operator in the open source framework, the method further comprises:
verifying whether each first test case is suitable for the operator in the open source framework by adopting a preset verification rule, if so, reserving the corresponding first test case; if not, deleting the corresponding first test case;
the preset verification rule comprises verification modes with different verification levels, and the first test cases are verified in sequence by adopting the verification modes with the different verification levels decreasing in sequence.
5. The method for generating test cases for operators according to claim 4, wherein said preset verification rule comprises: a structure verification mode and a numerical verification mode;
the step of adopting the structure verification mode for verification comprises the following steps:
determining a verification model corresponding to the operator according to the open source framework;
inputting the first test case into the verification model, and judging whether the verification model can run the first test case or not;
if yes, entering the numerical verification mode;
if not, deleting the first test case;
the step of verifying by adopting the numerical verification mode comprises the following steps:
determining verification input information and verification output information according to the operator information;
inputting the verification input information into the first test case to obtain a first test output;
judging whether the first test output is consistent with the verification output information or not;
if the first test cases are consistent, storing and outputting the corresponding first test cases;
and if the first test cases are inconsistent, deleting the corresponding first test cases.
6. The method for generating test cases for operators according to claim 5, further comprising:
Constructing a parameter information base according to the plurality of initial operator parameter information;
the generating method further comprises the following steps:
after deleting the first test case, searching the parameter information base according to the first test case, and determining the target parameter operator information corresponding to the first test case;
deleting the target parameter operator information corresponding to the first test case in the parameter information base so as to update the parameter information base.
7. A test case generation system for an operator, the generation system comprising:
the operator information acquisition module is used for acquiring frame information of an open source frame in the AI software stack and operator information corresponding to each operator in the open source frame;
the test case template generation module is used for generating a test case template corresponding to the operator information based on the frame information and the operator information;
the initial parameter information acquisition module is used for acquiring a plurality of initial operator parameter information based on operator reference test cases of different versions;
the target version acquisition module is used for acquiring target version information of the test case matched with the open source framework;
the target parameter information acquisition module is used for acquiring a plurality of pieces of target parameter operator information corresponding to the preset test requirements based on the target version information and the preset test requirements;
The first test case generation module is used for generating a plurality of first test cases corresponding to the preset test requirements based on the target parameter operator information and the test case template;
the target test case generation module is used for obtaining a plurality of target test cases for performing functional tests on each operator in the open source framework according to a plurality of first test cases corresponding to different preset test requirements.
8. An electronic device comprising a memory, a processor and a computer program stored on the memory and running on the processor, wherein the processor implements the test case generation method of the operator of any one of claims 1-6 when the computer program is executed by the processor.
9. A chip for use in an electronic device, wherein the chip is configured to perform the method for generating test cases for operators according to any one of claims 1-6.
10. A chip module for use in an electronic device, comprising a transceiver component and a chip for executing the test case generation method of the operator according to any one of claims 1-6.
11. A computer readable storage medium having stored thereon a computer program, characterized in that the computer program, when executed by a processor, implements the test case generation method of an operator according to any of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311668240.4A CN117687909A (en) | 2023-12-06 | 2023-12-06 | Operator test case generation method, system, equipment and medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311668240.4A CN117687909A (en) | 2023-12-06 | 2023-12-06 | Operator test case generation method, system, equipment and medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117687909A true CN117687909A (en) | 2024-03-12 |
Family
ID=90134543
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311668240.4A Pending CN117687909A (en) | 2023-12-06 | 2023-12-06 | Operator test case generation method, system, equipment and medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117687909A (en) |
-
2023
- 2023-12-06 CN CN202311668240.4A patent/CN117687909A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210342313A1 (en) | Autobuild log anomaly detection methods and systems | |
| US8839107B2 (en) | Context based script generation | |
| CN108959059B (en) | Test method and test platform | |
| CN107239392B (en) | Test method, test device, test terminal and storage medium | |
| US9171025B2 (en) | Test data generation and scale up for database testing using unique common factor sequencing | |
| CN111382070B (en) | Compatibility testing method and device, storage medium and computer equipment | |
| CN103186457A (en) | Method and device for automatically generating test case | |
| CN109614312A (en) | Method for generating test case, device, electronic equipment and storage medium | |
| CN114328276B (en) | Test case generation method and device, and test case display method and device | |
| CN110737594A (en) | Database standard conformance testing method and device for automatically generating test cases | |
| CN112015771B (en) | Data retrieval method and device, electronic equipment and computer storage medium | |
| CN109032631A (en) | Application program service packs acquisition methods, device, computer equipment and storage medium | |
| CN112306861A (en) | Unittest and Jenkins tool-based interface automatic testing system and method | |
| CN112328499A (en) | Test data generation method, device, equipment and medium | |
| CN112016283A (en) | Form generation method and device | |
| CN111158656A (en) | Method and device for generating test codes based on fruit tree method | |
| CN111782452A (en) | Method, system, device and medium for interface contrast test | |
| CN111061733B (en) | Data processing method, device, electronic equipment and computer readable storage medium | |
| CN114297961A (en) | Chip test case processing method and related device | |
| CN114117134A (en) | Abnormal feature detection method, device, equipment and computer readable medium | |
| CN116629215A (en) | Teaching document generation method and device, electronic equipment and storage medium | |
| CN117687909A (en) | Operator test case generation method, system, equipment and medium | |
| CN116483707A (en) | Test method, test device, test apparatus, test program, and test program | |
| CN115983377A (en) | Automatic learning method, device, computing equipment and medium based on graph neural network | |
| CN115543428A (en) | Simulated data generation method and device based on strategy template |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |