CN113918472B

CN113918472B - Test method, test system and storage medium for operator library

Info

Publication number: CN113918472B
Application number: CN202111515904.4A
Authority: CN
Inventors: 不公告发明人
Original assignee: Beijing Bilin Technology Development Co ltd; Shanghai Biren Intelligent Technology Co Ltd
Current assignee: Beijing Bilin Technology Development Co ltd; Shanghai Bi Ren Technology Co ltd
Priority date: 2021-12-13
Filing date: 2021-12-13
Publication date: 2022-04-08
Anticipated expiration: 2041-12-13
Also published as: CN113918472A

Abstract

A test method, a test system and a storage medium of an operator library, wherein the test method of the operator library comprises the following steps: receiving test parameter information of a target operator to be tested by using a unified test framework; respectively calling a plurality of operator process subprograms for the target operator according to the test parameter information through a unified test framework; and operating the plurality of operator process subroutines to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result. The test method of the operator library can unify and automate the test flow and the test process of the operator library, thereby solving the problem that a large number of operators need to be tested independently and needing repeated work, and lightening the follow-up development and maintenance work of a fine-grained operator library test system.

Description

Test method, test system and storage medium for operator library

Technical Field

The embodiment of the disclosure relates to a test method and a test system of an operator library and a storage medium.

Background

Currently, with the development of artificial intelligence technology, more and more application scenarios not only require artificial intelligence algorithm models implemented by software, but also require hardware products with higher processing efficiency and stronger specificity to process a large amount of tasks and data. The implementation of the artificial intelligence algorithm model on hardware can be realized by disassembling the artificial intelligence algorithm into a plurality of processing steps, wherein each processing step corresponds to an operator, and the function or performance of the operator on a hardware product needs to be verified through testing.

Disclosure of Invention

Some embodiments of the present disclosure provide a method for testing an operator library, the method including: receiving test parameter information of a target operator to be tested by using a unified test framework; respectively calling a plurality of operator process subprograms for the target operator according to the test parameter information through the unified test framework; and operating the plurality of operator process subroutines to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the running the plurality of operator process subroutines to obtain a test instruction packet includes: generating standard comparison data for the target operator, wherein the standard comparison data comprises standard comparison input data and standard comparison output data; generating an operator kernel source program corresponding to the target operator; generating operator kernel description information corresponding to the target operator; compiling the operator kernel source program and obtaining a compiled operator kernel program of the target operator; and generating the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the plurality of operator procedure subroutines includes: a standard alignment data subroutine for generating the standard alignment data for the target operator; a source program generation subprogram for generating the operator kernel source program corresponding to the target operator; and the instruction packet generation subprogram is used for compiling the operator kernel source program and obtaining the compiled operator kernel program of the target operator, and generating the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the running the plurality of operator process subroutines to obtain a test instruction packet includes: and using a testing tool to run the test instruction packet to obtain a test result.

For example, in some embodiments of the present disclosure, a method for testing an operator library, where the using a testing tool to run the test instruction packet to obtain a test result includes: inputting the standard comparison input data into the compiled operator kernel program; executing the compiled operator kernel program based on the operator kernel description information and obtaining an execution result; and comparing the execution result with the standard comparison output data to obtain a test result of the target operator.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the test parameter information includes: the test case description information is used for describing the attribute of an operator corresponding to the test case; the standard comparison data configuration information is used for configuring parameters for generating standard comparison data; and the operator kernel executes parameter configuration information, and is used for configuring and generating parameters of a source program of the operator kernel and configuring and generating parameters of the instruction packet for testing.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the type of the target operator includes: tensor class, vector class, or scalar class.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the receiving, by using a unified testing framework, test parameter information of a target operator to be tested includes: recording the test parameter information in the same configuration file; and reading the configuration file by using the unified testing framework so as to receive the testing parameter information of the target operator to be tested.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the respectively invoking a plurality of operator procedure subroutines for the target operator according to the test parameter information includes: analyzing the test parameter information to obtain test input parameters, wherein the test input parameters comprise the analyzed test case description information, the analyzed standard comparison data configuration information and the analyzed operator kernel execution parameter configuration information; constructing a parameter set according to the test input parameters, wherein the parameter set is used for an operator process subprogram corresponding to the type of the operator; and invoking the plurality of operator process subroutines using the set of parameters.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the constructing a parameter set according to the test input parameters includes: constructing a parameter container containing the test input parameters, wherein the parameter container comprises a plurality of parameters for the plurality of operator process subroutines and a guideline for indicating the plurality of parameters, the calling the plurality of operator process subroutines using the set of parameters comprises: invoking the plurality of operator process subroutines using the direction of the plurality of parameters; accessing the parameter container through direction of the plurality of parameters.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the accessing the parameter container includes: in response to at least one of said plurality of operator process subprograms invoking a corresponding first parameter of said plurality of parameters, invoking an attribute access interception operation, in response to said attribute access interception operation confirming that said at least one of said plurality of operator process subprograms does not define said first parameter, accessing said parameter container using direction of said first parameter.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the constructing a parameter container containing the test input parameters includes: adding a parameter configuration operation to the parameter container, wherein the parameter configuration operation comprises displaying parameter content or accessing parameter content using a referencing means.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the invoking a plurality of operator process subroutines for the target operator, and operating the plurality of operator process subroutines to obtain a test instruction packet includes: during the running process of the operator process subprograms, dynamically creating classes corresponding to the operator process subprograms and objects of the classes according to source programs of the operator process subprograms, acquiring parameter lists of the objects, constructing corresponding parameter objects according to the parameter lists, and automatically and dynamically calling the objects of the classes according to the parameter objects.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, a plurality of operator process subroutines of the target operator include a plurality of operation processes, the plurality of operation processes include a first class of operation processes and a second class of operation processes corresponding to the first class of operation processes, and the executing the compiled operator kernel program and obtaining an execution result includes: setting the execution result of the first type of operation process as a specified result; and running the second type of operation process according to the specified result and acquiring the execution result of the second type of operation process.

Some embodiments of the present disclosure provide a method for testing an operator library, the method including: acquiring configuration information for generating a plurality of test cases; generating a plurality of test cases according to the configuration information in a recursion mode; and outputting the plurality of test cases, and testing by respectively using the test method of the operator library provided by the disclosure.

For example, in a method for testing an operator library provided in some embodiments of the present disclosure, the configuration information includes a plurality of attribute lists, and the recursively generating a plurality of test cases according to the configuration information includes: selecting N attribute lists needing to be traversed in the attribute lists, wherein the attribute lists comprise a plurality of parallel attribute values; setting M attribute lists needing to be associated in the attribute lists; and recursively generating a plurality of test cases according to the N attribute lists needing to be traversed and the M attribute lists needing to be associated, wherein M and N are positive integers.

Some embodiments of the present disclosure provide a system for testing an operator library, the system comprising: the receiving module is configured to receive test parameter information of a target operator to be tested; and the execution processing module is configured to respectively call a plurality of operator process subprograms used for the target operator according to the test parameter information, and operate the operator process subprograms to obtain a test instruction packet.

For example, some embodiments of the present disclosure provide a test system for an operator library, further including: and the test operation module is configured to operate the test instruction packet to obtain a test result.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the execution processing module includes: a standard comparison data generation module configured to generate standard comparison data for the target operator, wherein the standard comparison data includes standard comparison input data and standard comparison output data; an operator kernel source program generating module configured to generate an operator kernel source program corresponding to the target operator; and the instruction generation module is configured to generate operator kernel description information corresponding to the target operator, compile the operator kernel source program to obtain a compiled operator kernel program, and generate the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

For example, in a test system for an operator library provided in some embodiments of the present disclosure, the test run module includes: a test operation input module configured to input the standard comparison input data to the compiled operator kernel program; the test operation execution module is configured to execute the compiled operator kernel program and obtain an execution result based on the operator kernel description information; and the test operation comparison output module is configured to compare the execution result with the standard comparison output data and obtain a test result of the target operator.

For example, in a test system for an operator library provided in some embodiments of the present disclosure, the types of operators include: tensor class, vector class, or scalar class.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the execution processing module includes: the parameter analysis module is configured to analyze the test parameter information to obtain a test input parameter, wherein the test input parameter comprises analyzed test case description information, analyzed standard comparison data configuration information and analyzed operator kernel execution parameter configuration information, the test case description information is used for describing attributes of an operator corresponding to the test case, the standard comparison data configuration information is used for configuring parameters for generating standard comparison data, and the operator kernel execution parameter configuration information is used for configuring parameters of a source program for generating the operator kernel and parameters of a test instruction packet for generating the operator; a parameter construction module configured to construct a parameter set according to the test input parameters, wherein the parameter set is used for an operator process subroutine corresponding to the type of the operator; and the entry calling module is configured to call the operator process subprograms by using the parameter set.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the parameter construction module includes: a parameter container construction unit configured to construct a parameter container containing the test input parameters, wherein the parameter container comprises a plurality of parameters for the plurality of subprograms and guidelines for indicating the plurality of parameters.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the parameter container further includes: and the attribute access intercepting unit is configured to respond to the calling of the corresponding first parameter in the plurality of parameters by at least one of the plurality of operator process subprograms, call an attribute access intercepting operation, respond to the confirmation that the first parameter is not defined by at least one of the plurality of operator process subprograms by the attribute access intercepting operation, and use the guidance of the first parameter to access the parameter container.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the parameter container constructing unit includes: a parameter configuration operation adding unit configured to add a parameter configuration operation to the parameter container, the parameter configuration operation including displaying parameter content or accessing the parameter content using a referencing means.

For example, in a test system of an operator library provided in some embodiments of the present disclosure, the plurality of operator procedure subroutines of the target operator include a plurality of operation procedures, the plurality of operation procedures include one or more first class operation procedures and one or more second class operation procedures, and the test run execution module includes: and the piling testing unit is configured to set the execution result of the first type of operation process as a specified result, run the second type of operation process according to the specified result and acquire the execution result of the second type of operation process.

For example, in a test system of an operator library provided in some embodiments of the present disclosure, the calling a plurality of operator process subroutines for the target operator, respectively, and operating the plurality of operator process subroutines to obtain a test instruction packet includes: during the running process of the operator process subprograms, dynamically creating classes corresponding to the operator process subprograms and objects of the classes according to source programs of the operator process subprograms, acquiring parameter lists of the objects, constructing corresponding parameter objects according to the parameter lists, and automatically and dynamically calling the objects of the classes according to the parameter objects.

Some embodiments of the present disclosure provide a system for testing an operator library, the system comprising: the acquisition module is configured to acquire configuration information for generating a plurality of test cases; the test case generation module is configured to generate a plurality of test cases according to the configuration information in a recursion mode; and the test case output module is configured to output the plurality of test cases and respectively use the system provided by some embodiments of the disclosure to test.

For example, in a testing system of an operator library provided in some embodiments of the present disclosure, the configuration information includes a plurality of attribute lists, the test case generating module recursively generates a plurality of test cases according to the configuration information, and includes: the attribute list selecting unit is configured to select N attribute lists needing to be traversed in the attribute lists, wherein the attribute lists comprise a plurality of parallel attribute values; the attribute list association unit is configured to set M attribute lists needing to be associated in the attribute lists; and the test case recursion generating unit is configured to recursively generate a plurality of test cases according to the N attribute lists needing to be traversed and the M attribute lists needing to be associated, wherein M and N are positive integers.

Some embodiments of the present disclosure provide a system for testing an operator library, the system comprising: a memory for non-transitory storage of computer-executable instructions; and a processor for executing the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, perform a method of testing an operator library provided according to some embodiments of the present disclosure.

Some embodiments of the present disclosure provide a non-transitory storage medium, wherein the non-transitory storage medium non-transitory stores computer-executable instructions that, when executed by a computer, perform a method of testing an operator library provided according to some embodiments of the present disclosure.

Drawings

To more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below, and it is apparent that the drawings in the following description relate only to some embodiments of the present disclosure and are not limiting to the present disclosure.

FIG. 1 is a flow chart of a method for testing an operator library according to some embodiments of the present disclosure;

FIG. 2 is a flow chart of one implementation of step S103 of the test method of the operator library of FIG. 1;

FIG. 3 is a flowchart of another implementation of step S103 of the method for testing the operator library of FIG. 1;

FIG. 4 is a flowchart of one implementation of step S102 of the test method for the operator library of FIG. 1;

FIG. 5 is a flow chart of a parameter construction and automatic invocation method according to some embodiments of the present disclosure;

FIG. 6 is a flow chart of a method for testing an operator library according to some embodiments of the present disclosure;

FIG. 7 is a flowchart of one implementation of step S602 of the method for testing the operator library of FIG. 6;

FIG. 8 is a schematic diagram of a test system for operator libraries according to some embodiments of the present disclosure;

FIG. 9 is a schematic diagram of an exemplary structure of the execution processing module of FIG. 8;

FIG. 10 is a schematic diagram of an exemplary configuration of the test run module of FIG. 8;

FIG. 11 is a diagram illustrating a parameter construction and auto-invocation module according to some embodiments of the present disclosure;

FIG. 12 is a schematic diagram of a test system for operator libraries according to some embodiments of the present disclosure;

FIG. 13 is an exemplary block diagram of the test case generation module of FIG. 12;

FIG. 14 is a schematic diagram of an exemplary structure of a testing system for operator libraries according to some embodiments of the present disclosure;

FIG. 15 is an exemplary diagram of a test case library corresponding to a test case library in a test case library testing system according to some embodiments of the disclosure;

FIG. 16 is a schematic diagram of a sub-library tester according to some embodiments of the present disclosure; and

fig. 17 is a schematic diagram of a testing system for operator libraries according to some embodiments of the present disclosure.

Detailed Description

In order to make the objects, technical solutions and advantages of the embodiments of the present disclosure more apparent, the technical solutions of the embodiments of the present disclosure will be described clearly and completely with reference to the drawings of the embodiments of the present disclosure. It is to be understood that the described embodiments are only a few embodiments of the present disclosure, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the described embodiments of the disclosure without any inventive step, are within the scope of protection of the disclosure.

Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this disclosure belongs. The use of "first," "second," and similar terms in this disclosure is not intended to indicate any order, quantity, or importance, but rather is used to distinguish one element from another. Also, the use of the terms "a," "an," or "the" and similar referents do not denote a limitation of quantity, but rather denote the presence of at least one. The word "comprising" or "comprises", and the like, means that the element or item listed before the word covers the element or item listed after the word and its equivalents, but does not exclude other elements or items. The terms "connected" or "coupled" and the like are not restricted to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "upper", "lower", "left", "right", and the like are used merely to indicate relative positional relationships, and when the absolute position of the object being described is changed, the relative positional relationships may also be changed accordingly.

The present disclosure is illustrated by the following specific examples. A detailed description of known functions and known parts (elements) may be omitted in order to keep the following description of the embodiments of the present disclosure clear and concise. When any element of an embodiment of the present disclosure appears in more than one drawing, that element is identified in each drawing by the same or similar reference numeral.

The machine learning algorithm is composed of a plurality of calculation units, each of which is called an Operator (Op). An operator is a mapping operation from one function space to another. For example, any process that performs an operation can be considered an operator. For a machine learning model, the operators are the computational functions involved in the machine learning model. For example, the Convolution algorithm in a Convolution Layer (Convolution Layer) may be an operator; the process of summing the weights in a Fully-connected Layer (FC Layer) may also be an operator. The operator may be implemented by software (for example, by a computer program) or by hardware (for example, by a circuit). For example, an Artificial Intelligence (AI) chip is also known as an Artificial Intelligence accelerator or an Artificial Intelligence computing card, i.e., a module dedicated to handling a large number of computing tasks in Artificial Intelligence applications. For example, the artificial intelligence chip needs to support various neural networks, and various networks need to call an operator library (e.g., a fine-grained operator library) customized for the artificial intelligence chip in the calculation process to complete the calculation task. In order to guarantee the quality of the operator library while searching and mining the optimal performance of the artificial intelligence chip, thousands of test cases are constructed to verify the function of the operator library, which makes verification and testing (verification and validation) of a fine-grained operator library heavy.

For this reason, the inventors of the present disclosure have noticed that, by improving the verification test flow of the operator library and unifying the verification test methods of the operator library, and for example, further improving the automation degree of the verification test work of the operator library, it is possible to greatly reduce the manpower and material resources of the test work.

Some embodiments of the present disclosure provide a method for testing an operator library, where the method for testing the operator library includes: receiving test parameter information of a target operator to be tested by using a unified test framework; respectively calling a plurality of operator process subprograms for the target operator according to the test parameter information through a unified test framework; and operating the plurality of operator process subroutines to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result.

The test method for the operator library provided by some embodiments of the present disclosure unifies the test flow and the test process of the operator library, and the test of operators of different types is realized by using the unified test method, so that the problem that a large number of operators need to be tested separately and repeatedly works is solved, the subsequent development and maintenance work of a fine-grained operator library test system is reduced, the manpower and material resources of the test work are reduced, the efficiency is improved, and the cost is reduced.

Fig. 1 is a flowchart of a method for testing an operator library according to some embodiments of the present disclosure. As shown in FIG. 1, the method at least comprises steps S101 to S103:

step S101, receiving test parameter information of a target operator to be tested by using a unified test framework;

step S102, respectively calling a plurality of operator process subprograms for a target operator through a unified test framework according to test parameter information;

and step S103, operating a plurality of operator process subroutines to obtain a test instruction packet.

For example, the unified testing framework of step S101 may be a software testing tool, a software testing system, or a software testing platform with a unified testing function, which is not limited in this embodiment of the present disclosure. For example, the target operators may be of different types, corresponding to different types of artificial intelligence algorithm models; the target operator may further have different types corresponding to different execution processes of the artificial intelligence algorithm model, for example, the type of the target operator may include a deep learning network Resnet, a split operator type of Bert and Yolo networks, such as a tensor class, a vector class, or a scalar class. And the like. Tensor-type operators can be used to perform convolution algorithms, Batch Normalization (BN) algorithms, Pooling (Pooling) algorithms; a vector type operator may be used to perform a matrix multiplication operation; scalar type operators may be used to perform algorithms such as activation functions, cross entropy, loss functions, and the like, and embodiments of the present disclosure are not limited in this respect. For example, the test parameter information may include test case description information, standard alignment data configuration information, and operator kernel execution parameter configuration information. The test case description information may be used to describe attributes of an operator corresponding to the test case, where the attributes of the operator may be, for example, the structure and input/output parameters of the operator, and the driving and loading modes of the operator. The standard comparison data configuration information may be used to configure parameters for generating standard comparison data, where the standard comparison data may be, for example, input data prepared in advance before the test procedure is executed and correct output data corresponding to the input data, for example, output data obtained by correctly executing an algorithm (or other verified software) corresponding to an operator. The operator kernel execution parameter configuration information may be used to configure a parameter of a source program that generates an operator kernel and a parameter of an instruction packet for generating a test, where the operator kernel execution parameter may be information used to describe a structure of the kernel source program, for example, the number of layers of a convolution operator, a shape parameter of each layer of the convolution operator, and the like, and the embodiment of the present disclosure is not limited thereto.

Fig. 4 is a flowchart of an implementation of step S101 of the method for testing the operator library of fig. 1. As shown in FIG. 4, one example of step S101 includes at least steps S401 to S402:

step S401, recording the test parameter information in the same configuration file;

step S402, reading the configuration file by using the unified testing framework to receive the testing parameter information of the target operator to be tested.

For example, in step S401, the test case description information, the standard comparison data configuration information, and the operator kernel execution parameter configuration information that may be included in the test parameter information may be recorded in the same configuration file, for example, the configuration file may be different types (formats) of configuration files such as YAML, YAM, JSON, INI, CONF, TXT, and the like, which is not limited in this embodiment of the disclosure. For example, it is also possible to choose to record the test parameter information in different profiles, as long as these profiles can be associated with each other.

For example, in step S402, the configuration file describing the test parameter information may be read or received by using a test framework such as a software test tool, a software test system, or a software test platform having a unified test function, and then the configuration file may be analyzed according to a preset rule to obtain various test parameter information described therein; the present disclosure is not limited to these pre-set rules.

For example, in step S102, the plurality of operator process subroutines may include a standard alignment data subroutine, a source program generation subroutine, and a test instruction packet generation subroutine. The standard alignment data subroutine may be used to generate standard alignment data for the target operator; the source program generation subprogram may be configured to generate an operator kernel source program corresponding to the target operator, where the operator kernel source program may be a source program used to express an operator execution process, and the source program is a main object that needs to be verified, for example, the source program may be written in a computer language, for example, a high-level language, an assembly language, or a machine language, and this is not limited in this embodiment of the disclosure; the instruction packet generation subprogram may be used to compile an operator kernel source program and obtain a compiled operator kernel program of a target operator, for example, the compiled operator kernel program may be a machine code (for example, a binary code) expressing an operator execution process, and then the instruction packet for testing may be generated according to the standard comparison data, the operator kernel description information, and the compiled operator kernel program. For example, the test instruction packet for actually executing the test job of the test case may be an executable software code set (e.g., a machine code set) for executing the test job of the target operator, and the executable software code set may run on the test platform or may be executed separately.

For example, for a certain operator to be tested, the operator procedure subroutines may be prepared in advance for calling in a situation independent of the unified test framework, and embodiments of the present disclosure do not limit the specific implementation manner of the operator procedure subroutines. In order to enable the unified test framework to call the operator process subroutines, the operator process subroutines correspondingly provide a parameter interface and a call function interface, for example, in at least one embodiment of the present disclosure, a manner in which the test framework calls the operator process subroutines may be automatically called by the test framework on one hand, and may be dynamically called on the other hand, for example, automatically and dynamically called on the other hand, so that not only can test work automation be realized, but also test scenarios can be diversified, and test workload is reduced. The unified test framework generates various parameters and calling processes corresponding to the parameter interface and the calling function interface based on the obtained test parameter information according to the parameter interface and the calling function interface provided by the operator process subprogram.

For example, some embodiments of the present disclosure further provide a parameter constructing and automatic invoking method, used in the unified testing framework (e.g., as one of the modules), including: analyzing the test parameter information to obtain test input parameters, wherein the test input parameters comprise analyzed test case description information, analyzed standard comparison data configuration information and analyzed operator kernel execution parameter configuration information; constructing a parameter set according to the test input parameters, wherein the parameter set is used for an operator process subprogram corresponding to the type of an operator; and invoking a plurality of operator process subroutines using the parameter set.

The parameter construction and automatic calling method simplifies the transmission of parameters in the testing process of the operator library, unifies the function calling process, for example, the naming of the parameters can be more unified, and the subsequent development and maintenance work of a fine-grained operator library testing system is reduced, so that the manpower and material resources of the testing work are reduced.

Fig. 5 is a flowchart of a parameter construction and automatic invocation method according to some embodiments of the present disclosure. For example, the parameter construction and automatic call method shown in fig. 5 may be used in the execution process of step S102 of the test method of the operator library in fig. 1. As shown in FIG. 5, the parameter construction and automatic calling method at least includes steps S501-S503:

step S501, analyzing test parameter information to obtain a test input parameter;

step S502, constructing a parameter set according to the test input parameters;

in step S503, a plurality of operator procedure subroutines are called using the parameter set.

For example, in step S501, the test input parameters may include parsed test case description information, parsed standard alignment data configuration information, and parsed operator kernel execution parameter configuration information.

In some examples, different parameters of each test case are generally read from the beginning running time, and then the parameters are transferred to different modules through layer-by-layer function calling, for example, one parameter is transferred to a required module layer by layer through different functions, which results in many parameters actually transferred in the whole process; in addition, different developers may have different naming habits for parameters, which often results in different names for the same parameter, for example, a parameter with the same meaning is named as a _ name when being transferred to the module a, named as B _ name when being transferred to the module B, and named as C _ name when being transferred to the module C, and at this time (i.e. in the module C), the parameter is actually used, but the name of the parameter is changed a few times, which is not beneficial to understanding and maintaining the code.

For example, in step S502, in one example, constructing a parameter set from test input parameters may include constructing a parameter container containing the test input parameters, wherein the parameter container includes a plurality of parameters for a plurality of operator process subroutines and a guideline for indicating the plurality of parameters, which may be a pointer to the parameter container or a reference to the parameter container. For example, constructing a parameter container containing test input parameters may include adding a parameter configuration operation to the parameter container, where the parameter configuration operation includes displaying parameter content or accessing parameter content using a referencing method, where displaying parameter content may be outputting the content of the parameter container through a log output or direct print output during a test; accessing the parameter content using a referencing approach may be to translate the content of the parameter container from a list indexing operation to a referencing operation of the parameter container object to the member attributes.

For example, in step S503, invoking the plurality of operator process subroutines using the parameter set may include invoking the plurality of operator process subroutines using directions of the plurality of parameters, for example, pointers or references to the plurality of parameters may be passed when the plurality of operator process subroutines are invoked, so that the plurality of parameters themselves need not be passed. The parameter container is accessed through the direction of the plurality of parameters, for example, using pointers or references to the plurality of parameters, without requiring direct access to the parameter container.

For example, accessing the parameter container may include, in response to at least one of the plurality of operator process subroutines invoking a corresponding first parameter of the plurality of parameters, for example, when an operator process subroutine invokes a parameter of the plurality of parameters, invoking an attribute access interception operation, for example, an attribute interception member method, in response to the attribute access interception operation confirming that at least one of the plurality of operator process subroutines does not define the first parameter, for example, when a parameter does not exist in the parameter container, accessing the parameter container using a direction of the first parameter, for example, accessing the parameter container according to a pointer or reference to the parameter.

By adopting the parameter container shared by a plurality of modules, multi-level transmission of each parameter under the condition that the modules share the parameter is simplified, and the understanding and the maintenance of codes are facilitated.

For example, in step S102 in fig. 1, a plurality of operator process subroutines for a target operator are called, and running the plurality of operator process subroutines to obtain a test instruction packet may include dynamically creating classes and class objects corresponding to the plurality of operator process subroutines according to a source program of the plurality of operator process subroutines during running of the plurality of operator process subroutines, for example, dynamically creating the classes and objects corresponding to the plurality of operator process subroutines through a computer programming language (for example, Python, Ruby, C + +, Java, JavaScript, and the like, which is not limited by this disclosure) during running of a test flow, obtaining a parameter list of the objects, constructing corresponding parameter objects according to the parameter list, and dynamically calling the class objects according to the parameter objects, for example, transferring the parameter objects into the classes corresponding to the dynamically called plurality of operator process subroutines. Compared with static calling (which refers to which function is written and called well by the code book and which parameters are needed), the dynamic calling method adopted by the embodiment of the disclosure has higher flexibility, is suitable for the situation of testing the diversification of the types of the operators and the diversification of the parameters, and correspondingly has higher efficiency.

For example, in step S103 in fig. 1, the test instruction packet may be used to be executed to obtain a test result. For example, the generated test instruction packet may be provided to the test platform via, for example, a network, and the like, for operation, and the test result is fed back.

Fig. 2 is a flow chart of one implementation of step S103 of the method for testing the operator library of fig. 1. As shown in FIG. 2, an example of step S103 includes at least steps S201-S205:

step S201, generating standard comparison data for a target operator;

step S202, generating an operator kernel source program corresponding to a target operator;

step S203, generating operator kernel description information corresponding to the target operator;

step S204, compiling an operator kernel source program and obtaining a compiled operator kernel program of a target operator;

and S205, generating a test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

For example, in step S201, the standard alignment data may be, for example, input data prepared in advance before the test flow is executed and correct output data corresponding to the input data, for example, output data obtained by correctly executing output data through verified software corresponding to an operator.

For example, in step S202, the operator kernel source program may be a source program for describing an operator execution process, and the source program may be written in a computer language, for example, a high-level language, an assembly language, or a machine language, which is not limited in this respect by the embodiments of the present disclosure.

For example, in step S203, the operator kernel description information may be used to describe a structure of the operator kernel, such as the number of layers of the convolution operator, a shape parameter of each layer of the convolution operator, and the like, as well as an input/output parameter of the operator and a driving and loading manner of the operator, which is not limited by the present disclosure.

For example, in step S204, the compiled operator kernel can be machine code (e.g., binary code) that expresses the operator execution process.

For example, in step S205, the standard alignment data, the operator kernel description information, and the compiled operator kernel program may be packaged as an execution combination to provide an executable test package that can be executed by the test platform.

For example, the process of running the plurality of operator process subroutines to obtain the test instruction packet may use a test tool of the test platform to run the test instruction packet to obtain the test result.

Fig. 3 is a flowchart of another implementation of step S103 of the method for testing the operator library of fig. 1. As shown in FIG. 3, step S103 at least includes steps S301 to S303:

step S301, inputting standard comparison input data into the compiled operator kernel program;

step S302, based on the operator kernel description information, executing the compiled operator kernel program and obtaining an execution result;

and step S303, comparing the execution result with the standard comparison output data, and obtaining a test result of the target operator.

For example, in step S301, standard alignment input data may be input to the compiled operator kernel program on the test tool. Embodiments of the present disclosure are not limited with respect to the specific type and configuration of the test tool.

For example, in step S302, the compiled operator kernel program may be executed on the test tool according to the instruction of the operator kernel description information, and an execution result is obtained.

For example, in step S303, the execution result is compared with the standard comparison output data by the test tool, so as to obtain a test result indicating whether the test of the target operator is correct.

For example, the plurality of operator process subroutines of the target operator comprise a plurality of operation processes, and the plurality of operation processes comprise an operation process which does not need to be tested and an operation process which corresponds to the operation process which does not need to be tested and also needs to be tested, so that piling can be carried out in the test process, and the directionality and the accuracy of the test are improved. Here, the operation procedure that has not been required to be tested corresponds to the first kind of operation procedure mentioned in the present disclosure, and the operation procedure that also requires to be tested corresponds to the second kind of operation procedure mentioned in the present disclosure.

For example, executing the compiled operator kernel program and obtaining the execution result may include setting the execution result of the operation process that does not need to be tested as a specified result, for example, as a specific output, and using the specific output as an input of the operation process that also needs to be tested; and running the operation process to be tested according to the specified result and obtaining the execution result of the operation process to be tested, for example, testing whether the output result of the operation process to be tested meets the expected set output result on the premise that the specific output is used as the input of the operation process to be tested, for example, comparing the output results with the standard, thereby realizing the respective test of each operation process of the subprogram of the operator process.

In order to realize fine-grained test on the operator library, the coverage rate of the test needs to be improved, and a large number of test cases are needed. If the test cases are manufactured by hand and one test case corresponds to one script, the scripts that may generate a large number (e.g., thousands) of test cases need maintenance. Moreover, the script maintenance workload of the handwritten test case is large, and the codes of the handwritten test case have a lot of repetition and hard codes (hardcodes), so that the test system has insufficient elasticity and is not easy to expand, modify and maintain.

In this regard, some embodiments of the present disclosure also provide a method for automatically generating batch test cases, which is helpful to improve productivity. The test method for automatically generating the batch test cases comprises the following steps: acquiring configuration information for generating a plurality of test cases; generating a plurality of test cases according to the configuration information in a recursion mode; and outputting a plurality of test cases, and testing by using the test method of the operator library respectively.

For example, the embodiment described above unifies the test behaviors of operators into, for example, a YAML configuration file that records parameter information of a typical test case, but adds configuration information corresponding to a plurality of test cases compared to the foregoing embodiment, so that batch YAML configuration information can be further automatically generated by an automatically executable configuration file generator (here, a YAML file generator) by using the parameter information of the single test case and the configuration information corresponding to the plurality of test cases, thereby generating a plurality of test cases equivalent to or parallel to the same in batch by a typical test, and thus automatically completing the test of the plurality of test cases, and improving the test coverage. The configuration file generator may be inserted as a module into the unified test framework.

Therefore, the test method for automatically generating the batch test cases of the operator library can be realized by changing the test parameter information of the target operator, and the single test case of the operator is expanded to automatically generate a plurality of test cases of the operator in batch, so that the automation degree of the test flow of the operator library is improved, the subsequent maintenance of the test cases of the operator library is easy, and the productivity is greatly increased.

Fig. 6 is a flowchart of a testing method for automatically generating a batch of test cases for an operator library according to some embodiments of the present disclosure. As shown in FIG. 6, the method at least includes steps S601 to S603:

step S601, obtaining configuration information for generating a plurality of test cases;

step S602, generating a plurality of test cases according to configuration information recursion;

step S603, a plurality of test cases are output and tested respectively.

For example, in step S601, the configuration information for generating a plurality of test cases may be the configuration information with configuration items for generating multiple test cases. The configuration information for generating the multiple test case configuration item may include configuration values corresponding to the multiple test cases, for example, the configuration values in the configuration information of each test case in the multiple test cases are combined to obtain a configuration value set, and the configuration value set is used as the configuration value corresponding to the multiple test case configuration item, so that the configuration information for generating the multiple test cases has the capability of generating the multiple test cases.

For example, in step S602, a generated multiple test case configuration item in the configuration information may be obtained and analyzed, and a plurality of configuration value combinations are generated by recursion according to the configuration item, where the recursion may be a backtracking algorithm, which is convenient for automation, but the recursion may also be a recursion algorithm, a dynamic programming algorithm, and the like, which is not limited in this embodiment of the disclosure. And correspondingly generating a plurality of configuration information according to the plurality of configuration value combinations so as to correspondingly generate a plurality of test cases.

For example, in step S603, after the plurality of test cases generated in step S602 are output, a corresponding test flow may be executed on each test case of the plurality of test cases by the method for testing the operator library shown in fig. 1, so as to obtain a corresponding test instruction packet.

Fig. 7 is a flowchart of the execution procedure of step S602 of the method for testing the operator library in fig. 6. As shown in FIG. 7, an example of the step S602 includes at least steps S701-S703:

step S701, selecting N attribute lists needing to be traversed in a plurality of attribute lists, wherein each attribute list comprises a plurality of parallel attribute values;

step S702, setting M attribute lists needing to be associated in a plurality of attribute lists;

step S703 is to recursively generate a plurality of test cases according to the N attribute lists to be traversed and the M attribute lists to be associated.

Here, M and N are positive integers. For example, in step S701, the multi-test case configuration item generated may be N attribute lists that need to be traversed in multiple attribute lists (configuration items), each attribute list includes multiple parallel attribute values (configuration values), and each attribute value may be a single element or a set (e.g., an element array) composed of multiple elements.

For example, in step S702, there is a correlation between a plurality of configuration items of the test case, and the setting of one configuration item is closely related to the setting of other related configuration items. In order to enable the generated multiple test cases to operate normally, M attribute lists associated with the N attribute lists to be traversed need to be traversed in a traversal manner of the N attribute lists to be traversed, for example, when one of the N attribute lists to be traversed is traversed to take a first value, the attribute list associated with the one of the N attribute lists to be traversed also takes the first value synchronously.

For example, in step S703, after the attribute list that needs to be traversed and the attribute list that needs to be associated are determined, combinations of elements of a plurality of attribute lists are generated by recursion, and a plurality of test cases are correspondingly generated according to the combinations of the elements of the plurality of attribute lists.

Fig. 8 is a schematic diagram of a testing system for operator libraries according to some embodiments of the present disclosure, which provides a unified testing framework according to embodiments of the present disclosure. As shown in fig. 8, the test system 10 of the operator library includes a receiving module 110 and an execution processing module 120.

For example, the receiving module 110 is configured to receive test parameter information of a target operator to be tested; the execution processing module 120 is configured to respectively invoke a plurality of operator process subroutines for the target operator according to the test parameter information, and run the operator process subroutines to obtain a test instruction packet.

The testing system of the operator library corresponds to the testing method of the operator library shown in fig. 1, and therefore the technical effect is the same as that of the testing method of the operator library shown in fig. 1, and is not repeated here.

For example, in some cases, the test system 10 of the operator library may further include a test run module 130. For example, the test execution module 130 may execute the test instruction packet to obtain the test result.

Fig. 9 is an exemplary structure diagram of the execution processing module in fig. 8. As shown in fig. 9, the execution processing module 20 may further include a standard alignment data generation module 210, an operator kernel source program generation module 220, and an instruction generation module 230.

For example, standard alignment data generation module 210 may generate standard alignment data for the target operator, where the standard alignment data includes standard alignment input data and standard alignment output data. The operator kernel source program generation module 220 may generate an operator kernel source program corresponding to the target operator. The instruction generating module 230 may generate operator kernel description information corresponding to the target operator, compile the operator kernel source program to obtain a compiled operator kernel program, and generate the test instruction packet according to the standard comparison data, the operator kernel description information, and the compiled operator kernel program.

Fig. 10 is an exemplary structural diagram of the test run module of fig. 8. As shown in fig. 10, the simulation operation module 30 may further include a test operation input module 310, a test operation execution module 320, and a test operation comparison output module 330.

For example, the test run input module 310 may input standard alignment input data to the compiled operator kernel program. The test run execution module 320 may execute the compiled operator kernel program based on the operator kernel description information and obtain an execution result. The test operation comparison output module 330 may compare the execution result with the standard comparison output data, and obtain a test result of the target operator.

Fig. 11 is a schematic diagram of a parameter construction and automatic call module according to some embodiments of the present disclosure, which is used in the unified testing framework. As shown in fig. 11, the parameter construction and automatic call module 40 includes a parameter parsing module 410, a parameter construction module 420, and an entry call module 430.

For example, the type of the operator may include a tensor type, a vector type, a scalar type, or the like, and the embodiment of the present disclosure is not limited thereto. For example, the parameter parsing module 410 is configured to parse test parameter information to obtain a test input parameter, where the test input parameter includes parsed test case description information, parsed standard comparison data configuration information, and parsed operator kernel execution parameter configuration information, where the test case description information is used to describe an attribute of an operator corresponding to the test case, the standard comparison data configuration information is used to configure a parameter used to generate standard comparison data, and the operator kernel execution parameter configuration information is used to configure a parameter of a source program that generates an operator kernel and configure a parameter that generates a test instruction packet.

The parameter construction module 420 is configured to construct a set of parameters from the test input parameters, wherein the set of parameters is for an operator procedure subroutine corresponding to the type of operator. The calling module 430 is configured to call a plurality of operator procedure subroutines using the set of parameters.

The parameter construction and automatic calling module can be used in the test system of the operator library shown in fig. 8, as part of a unified test framework. The technical effect of the parameter construction and automatic calling module is the same as that of the parameter construction and automatic calling method, and is not described herein again.

For example, in at least one example, the parameter construction module 420 may further comprise a parameter container construction unit that may construct a parameter container containing the test input parameters, wherein the parameter container comprises a plurality of parameters for the plurality of operator process subroutines and a guideline for indicating the plurality of parameters.

For example, the parameter container further comprises an attribute access intercepting unit, for example, the attribute access intercepting unit may invoke an attribute access intercepting operation in response to a corresponding first parameter in the plurality of parameters being invoked by at least one of the plurality of operator process subprograms, and access the parameter container using a direction of the first parameter in response to the attribute access intercepting operation confirming that the first parameter is not defined by the at least one of the plurality of operator process subprograms. For example, the parameter container constructing unit includes a parameter configuration operation adding unit that can add a parameter configuration operation to the parameter container, the parameter configuration operation including displaying the parameter content or accessing the parameter content using a referencing manner.

For example, the plurality of operator procedure subroutines of the target operator include a plurality of operation procedures including one or more first class operation procedures and one or more second class operation procedures, and the test run execution module includes a pile driving test unit. For example, the piling test unit may set the execution result of the first type of operation process as a specified result, run the second type of operation process according to the specified result, and acquire the execution result of the second type of operation process.

Fig. 12 is a schematic diagram of a testing system for operator libraries according to some embodiments of the present disclosure. As shown in fig. 12, the testing system 50 of the operator library includes an obtaining module 510, a test case generating module 520, and a test case outputting module 530.

For example, the obtaining module 510 is configured to obtain configuration information for generating a plurality of test cases; the test case generation module 520 is configured to recursively generate a plurality of test cases according to the configuration information; the test case output module 530 is configured to output a plurality of test cases, and perform testing using the test systems of the above-mentioned operator library, respectively.

The technical effect of the testing system of the operator library is the same as that of the testing method of the operator library shown in fig. 6, and details are not repeated here.

FIG. 13 is a diagram illustrating an exemplary structure of the test case generation module 520 in FIG. 12. As shown in fig. 13, the test case generation module 60 may further include an attribute list selection unit 610, an attribute list association unit 620, and a test case recursion generation unit 630.

For example, the attribute list selecting unit 610 may select N attribute lists that need to be traversed from a plurality of attribute lists, where an attribute list includes a plurality of parallel attribute values; the attribute list associating unit 620 may set M attribute lists that need to be associated among the plurality of attribute lists; the test case recursion generating unit 630 may recursively generate a plurality of test cases according to the N attribute lists that need to be traversed and the M attribute lists that need to be associated, where M and N are positive integers.

Fig. 14 is a schematic structural diagram of a test system for an operator library according to some embodiments of the present disclosure. The test system of the operator library can be used for executing the test method of the operator library shown in FIG. 1 or FIG. 6.

As shown in fig. 14, the test system 70 of the operator library includes a memory 710 for non-transitory storage of computer-executable instructions and a processor 720 for execution of the computer-executable instructions. For example, processor 720 may be implemented as hardware or firmware, such as a CPU, SoC, FPGA, ASIC, or any combination of hardware or firmware and software, as embodiments of the present disclosure are not limited in this respect. Processor 720, when executing the computer-executable instructions, performs the following method for testing an operator library: receiving test parameter information of a target operator to be tested by using a unified test framework; respectively calling a plurality of operator process subprograms for the target operator according to the test parameter information through a unified test framework; and operating the plurality of operator process subroutines to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result.

The technical effect of the testing system of the operator library shown in fig. 14 is the same as that of the testing method of the operator library, and is not described herein again.

In the embodiments or examples described above in conjunction with fig. 8 to 13, the various modules and units may be implemented by software, hardware, firmware, or any combination thereof, for example, and the embodiments of the present disclosure are not limited thereto.

At least one embodiment of the present disclosure also provides a non-transitory storage medium that non-transitory stores computer-executable instructions. For example, the non-transitory storage medium may be, but is not limited to, a magnetic tape, an optical disk, a magnetic disk, a flash memory, and other non-transitory storage media. When the computer-executable instructions are executed by a computer, the method for testing the operator library provided by at least one embodiment of the disclosure is executed. The technical effect of the non-transitory storage medium is the same as that of the above-described method of testing an operator library.

The following describes a method and a system for testing an operator library according to some embodiments of the present disclosure, with reference to some specific but non-limiting examples.

For example, for each operator in an operator library, an operator library (e.g., a fine-grained operator library) testing system may be used to verify and test operator kernel source programs (kernel. s) produced by an operator kernel source program generation module (code generator) that passes through that operator. For example, the operator library can be provided for a computing task of an upper computing framework of an artificial intelligence algorithm model (such as a deep learning network) to be called after fine-grained disassembly. For example, a convolutional neural network containing two convolutional layers and one fully-connected layer may be fine-grained decomposed into a first convolutional layer, a second convolutional layer, and a fully-connected layer. The convolutional neural network comprising two convolutional layers and a full-connection layer can respectively call a first convolution operator corresponding to the first convolutional layer, a second convolution operator corresponding to the second convolutional layer and a full-connection operator corresponding to the full-connection layer during operation.

For example, the unified test case library of the operator library also includes different types of test cases, corresponding to the operator library including different types of operators. Fig. 15 is a schematic diagram of a test case library corresponding to the test case library in the sub-library test system according to some embodiments of the present disclosure.

As shown in fig. 15, the test case library 80 includes a unified test case library 810, and the unified test case library 810 may include a tensor core test case library 811 and a vector core test case library 812 according to the type of operator (e.g., tensor operator or vector operator). The tensor core test sample library 811 may include a tensor convolution core use case library 8111, and the tensor convolution core use case library 8111 may include a plurality of tensor core test cases 8111_1 to a tensor core test sample 8111_ n. Similarly, the vector core test case library 812 may include vector core test cases 8121_1 through 8121_ n. Each of these specific test cases is used for testing the situation where an operator (i.e., code written for implementing a specific working process of the operator) works under different situations, thereby verifying whether the operator is written correctly.

Fig. 16 is a schematic diagram of an operator library tester according to some embodiments of the present disclosure. Generally, the test process of the test case of each operator in the operator library is similar, so the test process can be unified.

As shown in fig. 16, the operator library tester 1000 may be used to test the test case library 80 shown in fig. 15, and the operator library tester 1000 may be part of a test system to provide a unified test framework. In the operator library tester 1000, for each target operator to be tested, the test workflow 1010 has the four flows:

(1) generating standard alignment data (gold) 1011 for testing the target operator by a standard alignment data generating module (gold generator), wherein the standard alignment data may comprise standard alignment input data and standard alignment output data;

(2) generating a kernel source program 1012 of the target operator through an operator kernel source program generation module, for example, the kernel source program may be a source program describing an operation process of the target operator, and the source program may be written in a computer language;

(3) generating operator kernel description information 1013 corresponding to the target operator by an instruction generation module (command generator);

(4) the compiler is invoked by the instruction generation module to compile the kernel source program of the target operator to obtain the compiled operator kernel program of the target operator, and the instruction generation module further generates a test instruction packet 1014 that can be executed in a test execution module (e.g., an emulator or simulator, for example, which may be provided separately or as a part of the whole system) according to the input standard comparison data (e.g., a binary file of the standard comparison data) and in combination with the operator kernel description information corresponding to the target operator.

Thus, a test process having the above four steps can be unified, for example, the test process can be uniformly orchestrated using a unified test workflow 1010. The unified testing workflow 1010 can be called by the test case of each operator in the operator library, so that the four testing processes are prevented from being repeatedly configured by the test case of each operator, and the repetition of testing codes is reduced. Meanwhile, for example, when the test flow is changed, only the unified test workflow 1010 needs to be modified, so that the tester 1000 can be conveniently modified along with the change of the requirements, thereby increasing the flexibility of the test system and simplifying the maintenance work of the test system.

As shown in fig. 16, the test interface 1020 may be an interface for testing the workflow 1010 and the test flow of a specific operator, and receive a configuration file to obtain test parameter information of the operator to be tested. For example, the test interface may analyze the input test parameter information of the target operator to be tested through the call parameter configuration and automatic call unit 1060, so as to obtain a parameter for executing a test flow corresponding to the target operator. For example, the test flow of each target operator may call three test modules, standard alignment data generator 1070, source program generator 1080 and instruction generator 1090.

The standard alignment data generator 1070, the source program generator 1080, and the instruction generator 1090 may be, for example, software modules prepared in advance independently of the test workflow. For example, the test interface 1020 may automatically identify and call the standard alignment data generator 1070, the source program generator 1080 and the instruction generator 1090 corresponding to the target operator to be tested through the parameter construction and automatic calling unit 1060 when testing the target operator, and transfer the parameters obtained through parsing into the generators to execute a specific test flow.

For example, each operator in the operator library may include a tensor operator, a vector operator, a scalar operator, or the like, and since implementation manners of test interfaces corresponding to three classes of operators, namely, the tensor operator, the vector operator, and the scalar operator, are greatly different, the test interface 1020 may be implemented as a tensor operator test interface 1021, a vector operator test interface 1022, and a scalar operator test interface 1023 corresponding to the three classes of operators, respectively. For example, the test interface 1020 may be implemented as a test interface base class by a computer language, and the tensor operator test interface 1021, the vector operator test interface 1022, and the scalar operator test interface 1023 are respectively implemented as subclasses of the base class, where the same part may be implemented in the test interface base class, and different parts may be implemented in respective subclasses.

For example, different parameters of a test case of an operator in an operator library are usually read from the beginning of running the test case on a tester, and then transferred to different test modules in a test flow through a plurality of function calls, so that a lot of parameters need to be transferred in isolation. In addition, in a plurality of functions, the formal parameter names of parameters which substantially transmit the same data in each function are different due to different naming habits of different developers, so that the management of the tester is not facilitated, and the understanding of the code of the tester is also not facilitated. Thus, a parameter container object can be constructed by the parameter construction and automatic constructor 1060 in fig. 16, and the parameter container object can be used to accommodate parameters required by each module that the test case of the target operator needs to call during the execution process. Meanwhile, the parameter container object may only pass a reference (pointer) of the parameter container object when passing in each module. For example, each module can read (access) the required parameters from the parameter container object according to the operation requirement. For example, a unified parameter read exception handling may be added to the parameter container object corresponding to a parameter read of the parameter container object. For example, an attribute access interception function may be added to the parameter container object, so that each module in the test flow calls the interception function first when reading the parameter in the parameter container object to determine whether the read parameter is defined, and if the read parameter is defined, the parameter is normally read from the parameter container object; if the read parameter is undefined, an exception with undefined parameter is thrown. For example, the parameter container object may custom add various operations to the parameter container object, such as add print parameter content operations, for use by debug (debug); for example, recursive operations are added to generate a hierarchical tree of parameter objects, and the index operations that simplify the elements of the objects are point operations (e.g., reduce an index operation shaped as run _ cfg [0] [1] [1] to a point operation shaped as run _ cfg. usharp _ list. U0. ushape).

Therefore, the parameter container object constructed by the parameter construction and the automatic constructor simplifies the transmission mode of the parameters in a plurality of test modules in the test flow, unifies the operation of the parameters and improves the efficiency of parameter management.

For example, in a test flow for executing a target operator, the tester 1000 may be a call to an interface of the tensor operator test interface 1021, the vector operator test interface 1022, or the scalar operator test interface 1023 as described above, according to the type of operator to be tested. For example, the parameter construction and automatic invocation device 1060 can be used to enable testing of different operators to only configure testing parameter information and names of operators corresponding to target operators to be tested, and the parameter construction and automatic invocation device 1060 invokes, according to the testing parameter information and the names of the operators, a tensor operator testing interface 1021, a vector operator testing interface 1022 or a scalar operator testing interface 1023 corresponding to the target operators to be tested, so as to test the target operators through test cases.

For example, in at least one example, the tester 1000 may be implemented by dynamic programming using a computer programming language, and the tester 1000 may dynamically obtain the parameter container object and the parameter list of the parameter container object through the parameter constructing and automatically invoking device 1060 during the process of running the test case of the target operator. The computer programming language may be Python, Ruby, Java, C + +, JavaScript, and the like, which is not limited in this embodiment of the disclosure.

For example, the test flow of each operator in the operator library may be described by a configuration file, and the configuration file may include a plurality of configuration items, each configuration item corresponding to a configuration value, where the configuration file may be different types of configuration files such as YAML, YAM, JSON, INI, CONF, TXT, and for example, the test flow of each operator in the operator library may also be optionally recorded in different configuration files, as long as the configuration files can be associated with each other, which is not limited by the present disclosure. For example, profiles for different operators in an operator library may have the same structure (e.g., have the same configuration items) but different configuration values. For example, different parameter container objects corresponding to different operators can be constructed by setting different values of each configuration item in the configuration file, and different test interfaces corresponding to different operators are automatically and dynamically called, so that a fixed test function is called instead of a static state, and the test processes of test cases of different operators in an operator library are unified into the mode that the configuration items of the configuration file are set to different configuration values.

The YAML type profile code shown below is used as an example to illustrate, and the YAML type profile corresponds to a target operator to be tested. The following code is merely used to illustrate the form of the configuration file, and the embodiments of the present disclosure are not limited to the specific content of the code (e.g., specific name of the parameter, value, etc.).

case_name: test_vector_bwd_bn_act _in_1x128x56x28_fp32xbf16xbf16

case_module:

case_category: bwd_bn_act

golden_cfg:

ishape: 1x256x56x28

idtype: bf16

idata_pattern: linear

with_more_ch: True

run_cfg:

usharp_list:

U1:

···

U0:

···

U3:

···

U2:

l2_contorl_desc_bit: BYP

ushape: 1x256x56x28

udtype: bf16

tlr_ranges: [0, 224]

tlr_data_ready_sync_mode: BY_TLR_COUNT

tlr_buffer_mode: SINGLE

U4:

···

activation_type: RELU

tiling_strategy:

sample_gran: 1

channel_gran: 64

column_gran: 28

row_gran: 56

relative_threshold: 0.01

default_clear:

U1: [3, 4]

U3: all

U4: all

default_compare: [U1, U3, U4]

helper_ldm:

usharp_list:

U2:

l2_control_desc_bit: BYP

ushape: 1x256x56x28

udtype: BF16

tlr_ranges: [0, 224]

tlr_buffer_ready_sync_mode: BY_TLR_COUNT

tlr_buffer_mode: SINGLE

tiling_strategy:

sample_gran: 1

channel_gran: 64

column_gran: 28

row_gran: 28

For example, the YAML type configuration file may include a basic description part of a target operator test case to be tested, a parameter configuration part of a standard alignment data generation module, a parameter configuration part of an operator kernel source program generation module, a parameter configuration part of an instruction generation module, and a pile driving configuration part.

As shown in the YAML type profile code above, in the basic description section, case _ name represents the test case name; case _ module represents the name of the network to be tested, such as resnet 50; case _ category represents an operator category. The golden _ cfg part corresponds to the parameter configuration part of the standard alignment data generation module; the run _ cfg part corresponds to a parameter configuration part of an operator kernel source program generation module and a parameter configuration part of an instruction generation module; the helper _ ldm section corresponds to the piling configuration section.

The YAML type profile code is illustrated by taking a Batch Normalization (BN) operator test case as an example, and it should be noted that the Batch Normalization operator test case is only an illustrative example of the disclosure, and embodiments of the disclosure are not limited thereto.

For example, br _ bwd _ BN _ act obtained by adding the prefix br _ to the configuration value bwd _ BN _ act in the operator name configuration item case _ category in the YAML type configuration file code is the name of the test interface that needs to be called by the test case of the reverse BN operator, so as to dynamically obtain the object and the construction parameter container of the test interface called by the operator test case. For example, the construction method of the parameter can be divided into two types, one type of parameter is the construction of a simple type of parameter (e.g., int, float, str, pool, and list types), and the construction process obtains a configured parameter from a parameter namespace (argsNamespace) according to the name of the parameter, and then assigns the value of the parameter to be called; another parameter is the construction of a custom type (e.g., object of a custom class), which takes the configured parameter from the parameter namespace (ArgNamespace) according to the name of the parameter, and then constructs the object of the custom class and assigns the value to the parameter to be called. After all parameters are constructed, and the object of the operator calling interface obtained in the previous step, such as the automatic dynamic operator calling interface, is obtained, the parameter construction method and the automatic calling process are the same for all operators, so that different configuration information and configuration files realize the test process of different operators.

For example, the parameter configuration part gold _ cfg of the standard alignment data generation module of the reverse BN operator test case includes: an input tensor dimension ishape represented by 1x256x56x28, an input data type idtype represented by bf16, an input data pattern idata _ pattern represented by linear, and a channel represented by True having multiple channels with _ more _ ch.

For example, the tensor calculation unit (Usharp) configuration list Usharp _ list of the operator kernel source program generation module run _ cfg of the reverse BN operator test case includes 5 tensor calculation units U0, U1, U2, U3, and U4, for example, the tensor calculation unit (Usharp) may include a tensor dimension (tensor shape, Ushape), a data type, and a memory slice pointing to each pixel of the tensor. For example, the tensor dimension usahape of the tensor calculation unit U2 is 1 × 256 × 56 × 28, the TLR register range TLR _ ranges used is [0, 224], the TLR register data ready synchronization mode used is a mode BY _ TLR _ COUNT, and the buffer mode TLR _ buffer _ mode of the TLR register used is a SINGLE mode SINGLE. The blocking strategy tiling _ strategy of the test case of the reverse BN operator is 1X64X56X28, namely, the test case is only blocked on a channel dimension channel _ gram, the original 256 channels are divided into 4 blocks, and each block has 64 channels. For example, the activation function activation _ type may be a linear rectification unit (RELU) function.

For example, in order to simplify the test of a part of the processes, the instruction generating module of the reverse BN operator test case may introduce a pile driving module helper _ ldm, where pile driving is performed by a plurality of tensor calculation units, in order to test an output result of one of the tensor calculation units, an output of a previous tensor calculation unit corresponding to the tensor calculation unit to be tested is designated as a correct output, the designated correct output is used as an input of the tensor calculation unit to be tested, an output of the tensor calculation unit to be tested is obtained from the input, and the output of the tensor calculation unit to be tested is compared with a standard output corresponding to the tensor calculation unit to be tested in the standard comparison data to obtain a test result. The pile driving module can load a specified output of U2, and the test of the reverse BN operator calculates the outputs U1, U3 and U4 and compares the outputs with standard output data to verify and test whether the algorithm of the reverse BN operator meets the requirements. Therefore, when the test case runs the test flow of the reverse BN operator, the default _ compare is Uhards output by U1, U3 and U4. The pile driving module may also set a blocking policy, for example, the blocking policy is set to 1X64X28X28, that is, the channel _ gram and the column _ gram are blocked, the original 256 channels are divided into 4 blocks, and the original 56 columns are divided into 2 blocks, so that a total of 4X2=8 blocks is obtained.

For example, the operator library test system proposed by some embodiments of the present disclosure may be based on a test framework of an open source code, for example, the test framework may be Pytest, which is not limited in this respect by the embodiments of the present disclosure. For example, a test framework that employs Pytest as an operator library test system may run and manage test cases of an operator library, e.g., the test cases of the operator library may be derived from the base class of Pytest. For example, the testing framework of the open source code combines the configuration files, so that the testing methods of the operator library can be unified, and the testing work of the operator library is simplified.

As described above, the operator library comprises a large number of operators, and in order to ensure that the operator library can completely support various computing tasks of various artificial intelligence algorithm models, a large number of test cases are needed to verify the functions of the operator library. The manual test case compiling has large maintenance workload, the test cases have hard codes (hardcodes), and the repeatability of different test cases is high, so that the operator library test system has insufficient elasticity and is not easy to expand, modify and maintain. Therefore, it is necessary to generate a lot of test cases by an automated method to improve the production efficiency.

For example, an automated test parameter information file generator may be further designed according to the test parameter information file (e.g., the above-mentioned YAML type configuration file) of the target operator, that is, the generator may automatically generate a batch of test parameter information files (e.g., the YAML type configuration file), that is, a batch of generated test cases.

Fig. 17 is a schematic diagram of a testing system for operator libraries according to some embodiments of the present disclosure. As shown in FIG. 17, for example, a multiple test case configuration file may be automatically generated by a multiple test case generator. The generated multiple test case configuration files can be operated on a test platform so as to automatically complete the execution of the test flow of the corresponding target operator. Meanwhile, the single test case configuration file can also be operated on the test platform to execute the test flow of the corresponding target operator. The test platform of fig. 17 may be one embodiment of the tester 1000 of fig. 16.

For example, as an implementation of the multiple test case configuration file in fig. 17, the YAML type configuration file may be added with the following codes, so as to extend the YAML type configuration file to a YAML type configuration file capable of correspondingly producing multiple test cases:

run_loop_cfg: [golden_cfg.ishape]

connected_shapes:

- golden_cfg.ishape

- run_cfg.usharp_list.U0.ushape

- run_cfg.usharp_list.U1.ushape

- run_cfg.usharp_list.U2.ushape

- run_cfg.tiling_strategy.column_gran

golden_cfg:

ishape: [1x64x8x16, 2x64x16x16, 2x64x24x16, 2x64x56x56]

···

usharp_list:

U0:

ushape: [1x64x8x16, 2x64x16x16, 2x64x24x16, 2x64x56x56]

···

U1:

ushape:[1x128x8x16,2x128x16x16,2x128x24x16,2x128x56x56]

···

U2:

ushape:[1x64x16x32,2x64x32x32,2x64x48x32, 2x64x112x112]

···

tiling_strategy:

···

column_gran: [16, 16, 16, 56]

as shown in the above code, for example, a configuration item capable of circularly generating a plurality of YAML type configuration files may be added to the YAML type configuration file to obtain a multi-test case YAML type configuration file, and the configuration item may include, for example, a traversal configuration item run _ loop _ cfg and a parameter shape associated _ shapes. Wherein, run _ loop _ cfg may represent configuration items to be traversed, such as input shape, tensor dimension, Ushape, and the like; connected _ maps may represent configuration items that need to be associated during traversal, for example, during a loop traversal, each element of ishape in gold _ cfg needs to be associated with each element of U0 of usharp _ list in run _ cfg, U1, U2, and column _ gram of tiling _ category in run _ cfg in a one-to-one correspondence.

For example, the YAML type configuration file shown in the above code indicates that ishape is [1x64x8x16, 2x64x16x16, 2x64x24x16, 2x64x56x56], and since the configuration item to be traversed by the test case is ishape, the configuration item connected _ clips indicates a configuration item that needs to be linked in a process of cyclically generating a plurality of YAML type configuration files, that is, if ishape is a first element of the list, then linked run _ cfg.use _ list.u0.ushape is also a first element of the list, and 4 YAML type configuration files are generated, where each YAML type configuration file corresponds to a test case, that is, a total of 4 test cases are generated.

For example, depending on the configuration items to be traversed and the configuration items to be linked, the multiple test case generator in fig. 17 may execute the following three parts:

in the first section, a multiple test case configuration file (e.g., a multiple test case YAML type configuration file) is input.

And in the second part, traversing the configuration items and the associated configuration items as required, and recursively generating a plurality of parameter combinations, wherein the recursion may be, for example, a recursive algorithm, a backtracking algorithm, a dynamic programming algorithm, and the like, which is not limited by the present disclosure. For example, when 3 configuration items to be traversed are configured, the first configuration item to be traversed is a list [ a, B, C ] with a length of 3, the second configuration item to be traversed is a list [ D, E ] with a length of 2, and the third configuration item to be traversed is a list [ F, G, H, I ] with a length of 4, all combinations are generated, for example, using one or more of the two-pointer movement method, the recursive algorithm, the backtracking algorithm, and the like, such as 3x2x4=24 combinations [ "ADE", "ADG", "ADH", "ADI", "CEI" · · · and · CEI ]. For example, defining 5 configuration items to traverse and adding two configuration items to the list of 5 and 6, there are 3x2x4x5x6=720 combinations, and the generator will generate 720 test cases.

And a third part, respectively taking out corresponding elements from the configuration item list according to the generated multiple parameter combinations, generating multiple single test case configuration files, and submitting the multiple single test case configuration files to the test platform to execute the test flow. For example, a YAML type configuration file corresponding to 720 test cases can be generated according to the combination of 720 configuration items.

Thus, with the multi-test case generator, the generation of bulk YAML type configuration files can be accomplished without any programming.

For the present disclosure, there are also the following points to be explained:

(1) the drawings of the embodiments of the disclosure only relate to the structures related to the embodiments of the disclosure, and other structures can refer to the common design.

(2) Without conflict, embodiments of the present disclosure and features of the embodiments may be combined with each other to arrive at new embodiments.

The above are merely specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and the scope of the present disclosure should be determined by the scope of the claims.

Claims

1. A testing method of an operator library comprises the following steps:

receiving test parameter information of a target operator to be tested by using a unified test framework;

respectively calling a plurality of operator process subprograms for the target operator according to the test parameter information through the unified test framework;

operating the plurality of operator process subroutines to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result,

wherein, the operating the plurality of operator process subroutines to obtain the test instruction packet comprises:

generating standard comparison data for the target operator, wherein the standard comparison data comprises standard comparison input data and standard comparison output data;

generating an operator kernel source program corresponding to the target operator;

generating operator kernel description information corresponding to the target operator;

compiling the operator kernel source program and obtaining a compiled operator kernel program of the target operator;

and generating the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

2. The method of claim 1, wherein the plurality of operator process subroutines comprises:

a standard alignment data subroutine for generating the standard alignment data for the target operator;

a source program generation subprogram for generating the operator kernel source program corresponding to the target operator;

and the instruction packet generation subprogram is used for compiling the operator kernel source program and obtaining the compiled operator kernel program of the target operator, and generating the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

3. The method of claim 1, wherein said executing said plurality of operator process subroutines results in a test instruction packet comprising:

and using a testing tool to run the test instruction packet to obtain the test result.

4. The method of claim 3, wherein the using a test tool to run the test instruction packet to obtain a test result comprises:

inputting the standard comparison input data into the compiled operator kernel program;

executing the compiled operator kernel program based on the operator kernel description information and obtaining an execution result; and

and comparing the execution result with the standard comparison output data to obtain a test result of the target operator.

5. The method of claim 1, wherein the test parameter information comprises:

the test case description information is used for describing the attribute of an operator corresponding to the test case;

the standard comparison data configuration information is used for configuring parameters for generating standard comparison data;

and the operator kernel executes parameter configuration information, and is used for configuring and generating parameters of a source program of the operator kernel and configuring and generating parameters of the instruction packet for testing.

6. The method of claim 5, wherein the type of target operator comprises: tensor class, vector class, or scalar class.

7. The method of claim 5, wherein said receiving test parameter information for a target operator to be tested using a unifying test framework comprises:

recording the test parameter information in the same configuration file;

and reading the configuration file by using the unified testing framework so as to receive the testing parameter information of the target operator to be tested.

8. The method of claim 5, wherein said invoking a plurality of operator process subroutines for the target operator, respectively, according to the test parameter information comprises:

analyzing the test parameter information to obtain test input parameters, wherein the test input parameters comprise the analyzed test case description information, the analyzed standard comparison data configuration information and the analyzed operator kernel execution parameter configuration information;

constructing a parameter set according to the test input parameters, wherein the parameter set is used for an operator process subprogram corresponding to the type of the operator; and

invoking the plurality of operator process subroutines using the set of parameters.

9. The method of claim 8, wherein said constructing a set of parameters from said test input parameters comprises:

constructing a parameter container containing the test input parameters, wherein the parameter container comprises a plurality of parameters for the plurality of operator process subroutines and a guideline for indicating the plurality of parameters,

said invoking said plurality of operator process subroutines using said set of parameters comprises:

invoking the plurality of operator process subroutines using the direction of the plurality of parameters;

accessing the parameter container through direction of the plurality of parameters.

10. The method of claim 9, wherein,

the accessing the parameter container includes:

invoking an attribute access interception operation in response to at least one of said plurality of operator process subroutines invoking a respective first parameter of said plurality of parameters,

accessing the parameter container using direction of the first parameter in response to the attribute access interception operation confirming that the first parameter is not defined by at least one of the plurality of subprograms.

11. The method of claim 9, wherein said constructing a parameter container containing said test input parameters comprises:

adding a parameter configuration operation to the parameter container, wherein the parameter configuration operation comprises displaying parameter content or accessing parameter content using a referencing means.

12. The method of claim 1, wherein said invoking a plurality of operator process subroutines for the target operator, respectively, executing the plurality of operator process subroutines to obtain a test instruction packet comprises:

during the running process of the operator process subprograms, dynamically creating classes corresponding to the operator process subprograms and objects of the classes according to source programs of the operator process subprograms, acquiring parameter lists of the objects, constructing corresponding parameter objects according to the parameter lists, and automatically and dynamically calling the objects of the classes according to the parameter objects.

13. The method of claim 4, wherein the plurality of operator procedure subroutines of the target operator comprise a plurality of operational procedures, including a first class of operational procedures and a second class of operational procedures corresponding to the first class of operational procedures,

the executing the compiled operator kernel program and obtaining an execution result includes:

setting the execution result of the first type of operation process as a specified result;

and running the second type of operation process according to the specified result and acquiring the execution result of the second type of operation process.

14. A testing method of an operator library comprises the following steps:

acquiring configuration information for generating a plurality of test cases;

generating a plurality of test cases according to the configuration information in a recursion mode;

outputting the plurality of test cases, and testing by using the method of claim 1.

15. The method of claim 14, wherein said configuration information includes a plurality of attribute lists, said recursively generating a plurality of test cases according to said configuration information comprising:

selecting N attribute lists needing to be traversed in the attribute lists, wherein the attribute lists comprise a plurality of parallel attribute values;

setting M attribute lists needing to be associated in the attribute lists;

and recursively generating a plurality of test cases according to the N attribute lists needing to be traversed and the M attribute lists needing to be associated, wherein M and N are positive integers.

16. A test system for operator libraries, comprising:

the receiving module is configured to receive test parameter information of a target operator to be tested by using a unified test framework;

the execution processing module is configured to respectively call a plurality of operator process subprograms used for the target operator according to the test parameter information through the unified test framework, and operate the operator process subprograms to obtain a test instruction packet, wherein the test instruction packet is used for being operated to obtain a test result;

the test operation module is configured to operate the test instruction packet to obtain the test result;

wherein the execution processing module comprises:

a standard comparison data generation module configured to generate standard comparison data for the target operator, wherein the standard comparison data includes standard comparison input data and standard comparison output data;

an operator kernel source program generating module configured to generate an operator kernel source program corresponding to the target operator;

and the instruction generation module is configured to generate operator kernel description information corresponding to the target operator, compile the operator kernel source program to obtain a compiled operator kernel program, and generate the test instruction packet according to the standard comparison data, the operator kernel description information and the compiled operator kernel program.

17. The system of claim 16, wherein the test run module comprises:

a test operation input module configured to input the standard comparison input data to the compiled operator kernel program;

the test operation execution module is configured to execute the compiled operator kernel program and obtain an execution result based on the operator kernel description information; and

and the test operation comparison output module is configured to compare the execution result with the standard comparison output data and obtain a test result of the target operator.

18. The system of claim 17, wherein the type of operator comprises:

tensor class, vector class, or scalar class.

19. The system of claim 18, wherein the execution processing module comprises:

the parameter analysis module is configured to analyze the test parameter information to obtain a test input parameter, wherein the test input parameter comprises analyzed test case description information, analyzed standard comparison data configuration information and analyzed operator kernel execution parameter configuration information, the test case description information is used for describing attributes of an operator corresponding to the test case, the standard comparison data configuration information is used for configuring a parameter used for generating standard comparison data, and the operator kernel execution parameter configuration information is used for configuring a parameter of a source program for generating the operator kernel and configuring a parameter for generating the test instruction packet;

a parameter construction module configured to construct a parameter set according to the test input parameters, wherein the parameter set is used for an operator process subroutine corresponding to the type of the operator; and

an entry call module configured to call the plurality of operator process subroutines using the set of parameters.

20. The system of claim 19, wherein the parameter construction module comprises:

a parameter container construction unit configured to construct a parameter container containing the test input parameters, wherein the parameter container comprises a plurality of parameters for the plurality of subprograms and guidelines for indicating the plurality of parameters.

21. The system of claim 20, wherein the parameter container further comprises:

an attribute access intercepting unit configured to, in response to the attribute access intercepting unit,

22. The system of claim 20, wherein the parameter container construction unit comprises:

a parameter configuration operation adding unit configured to add a parameter configuration operation to the parameter container, the parameter configuration operation including displaying parameter content or accessing the parameter content using a referencing means.

23. The system of claim 19, wherein the plurality of operator process subroutines of the target operator comprise a plurality of operational processes including one or more first class operational processes and one or more second class operational processes, the test run execution module comprising:

and the piling testing unit is configured to set the execution result of the first type of operation process as a specified result, run the second type of operation process according to the specified result and acquire the execution result of the second type of operation process.

24. The system of claim 16, wherein said invoking a plurality of operator process subroutines for the target operator, respectively, executing the plurality of operator process subroutines to obtain a test instruction packet comprises:

25. An operator kernel test system, comprising:

the acquisition module is configured to acquire configuration information for generating a plurality of test cases;

the test case generation module is configured to generate a plurality of test cases according to the configuration information in a recursion mode;

a test case output module configured to output the plurality of test cases and perform a test using the system according to claim 16, respectively.

26. The system of claim 25, wherein the configuration information includes a plurality of attribute lists, the recursively generating a plurality of test cases according to the configuration information, the test case generating module comprising:

the attribute list selecting unit is configured to select N attribute lists needing to be traversed in the attribute lists, wherein the attribute lists comprise a plurality of parallel attribute values;

the attribute list association unit is configured to set M attribute lists needing to be associated in the attribute lists;

and the test case recursion generating unit is configured to recursively generate a plurality of test cases according to the N attribute lists needing to be traversed and the M attribute lists needing to be associated, wherein M and N are positive integers.

27. A test system for operator libraries, comprising:

a memory for non-transitory storage of computer-executable instructions; and

a processor for executing the computer-executable instructions,

wherein the computer-executable instructions, when executed by the processor, perform a method of testing the operator library according to any of claims 1-15.

28. A non-transitory storage medium, wherein the non-transitory storage medium non-transitory stores computer-executable instructions that, when executed by a computer, perform a method of testing an operator library according to any one of claims 1-15.