CN117312173A - Deep learning framework testing method and device, electronic equipment and storage medium - Google Patents

Abstract

The disclosure relates to a test method and device of a deep learning framework, electronic equipment and storage medium. The method comprises the following steps: injecting data grabbing codes into a deep learning model deployed in a first deep learning framework; training the deep learning model in the first deep learning frame, and grabbing information of a first intermediate variable set of the deep learning model through the data grabbing code; and determining a test result of the first deep learning framework according to the information of the first intermediate variable set.

Description

Testing methods, devices, electronic equipment and storage media for deep learning frameworks

Technical field

The present disclosure relates to the field of computer technology, and in particular, to a deep learning framework testing method, a deep learning framework testing device, electronic equipment, and a storage medium.

Background technique

The rapid development of deep learning has not only promoted the birth of many open source deep learning frameworks, such as TensorFlow, PyTorch, Caffe, etc., but also inspired many institutions to develop proprietary deep learning frameworks to meet specific needs. However, due to differences in the design principles and optimization methods of different deep learning frameworks, the training results of the same deep learning model under different deep learning frameworks may be different. Evaluating and validating the accuracy and capabilities of these deep learning frameworks is often a complex and time-consuming task.

Contents of the invention

The present disclosure provides a technical solution for testing a deep learning framework.

According to one aspect of the present disclosure, a method for testing a deep learning framework is provided, including:

Inject data scraping code into the deep learning model deployed in the first deep learning framework;

Train the deep learning model in the first deep learning framework, and capture the information of the first intermediate variable set of the deep learning model through the data capture code;

Determine the test result of the first deep learning framework according to the information of the first intermediate variable set.

In a possible implementation, determining the test result of the first deep learning framework based on the information of the first intermediate variable set includes:

Get the base value;

Compare the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework.

In one possible implementation,

Obtaining the benchmark value includes: injecting the data capture code into the deep learning model deployed in the second deep learning framework; training the deep learning model in the second deep learning framework, and passing The data capture code captures the information of the second intermediate variable set of the deep learning model;

Comparing the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework includes: comparing the information of the first intermediate variable set with the second intermediate variable Compare the set of information to determine the test result of the first deep learning framework.

In one possible implementation,

The first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework;

The second intermediate variable set includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

In one possible implementation,

The information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

The information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

In one possible implementation,

The method of using the data grabbing code to capture the information of the first intermediate variable set of the deep learning model includes: using the data grabbing code to capture the depth information trained in the first deep learning framework. The output value of the non-last network layer of the learning model;

The method of using the data grabbing code to capture the information of the second intermediate variable set of the deep learning model includes: using the data grabbing code to capture the depth information trained in the second deep learning framework. The output value of the non-last network layer of the learning model.

In one possible implementation,

The method further includes: using the data grabbing code to capture the first output value of the last network layer of the deep learning model trained in the first deep learning framework; using the data grabbing code to capture Taking the second output value of the last network layer of the deep learning model trained in the second deep learning framework;

Comparing the information of the first intermediate variable set with the information of the second intermediate variable set and determining the test result of the first deep learning framework includes: based on the information of the first intermediate variable set, The information of the second intermediate variable set, the first output value and the second output value determine the test result of the first deep learning framework.

In a possible implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes:

For a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, determine a difference value between the first intermediate variable and the second intermediate variable;

Wherein, the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In a possible implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes:

Taking the absolute values of the elements in the first intermediate variable and summing them up, the sum of the elements of the first intermediate variable is obtained;

The absolute values of the elements in the second intermediate variable are taken and then summed to obtain the sum of the elements of the second intermediate variable.

In a possible implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes:

The total number of elements in the first intermediate variable or the second intermediate variable is determined according to the first intermediate variable or the second intermediate variable.

In a possible implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes:

determining a standard difference value between the first intermediate variable and the second intermediate variable;

A first number of elements whose values are greater than a preset threshold among the standard difference values is determined.

In a possible implementation, determining the standard difference value between the first intermediate variable and the second intermediate variable includes:

Determine the absolute value of the difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and the absolute value of the first intermediate variable;

The product is subtracted from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

In a possible implementation, after determining the first number of elements whose values are greater than a preset threshold in the standard difference values, the method further includes:

A ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable is determined.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a baseline deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

According to one aspect of the present disclosure, a testing device for a deep learning framework is provided, including:

A data capture module, configured to inject data capture code into the deep learning model deployed in the first deep learning framework, train the deep learning model in the first deep learning framework, and capture the data through the The code captures the information of the first intermediate variable set of the deep learning model;

An accuracy comparison module, configured to determine the test result of the first deep learning framework based on the information of the first intermediate variable set.

In one possible implementation,

The data capture module is also used to: obtain the benchmark value;

The accuracy comparison module is specifically configured to compare the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework.

In one possible implementation,

The data capture module is specifically configured to: inject the data capture code into the deep learning model deployed in the second deep learning framework; train the deep learning model in the second deep learning framework, And capture the information of the second intermediate variable set of the deep learning model through the data capture code;

The accuracy comparison module is specifically configured to compare the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework.

In one possible implementation,

The first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework;

The second intermediate variable set includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

In one possible implementation,

The information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

The information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

In a possible implementation, the data capture module is specifically configured to: capture the non-last network layer of the deep learning model trained in the first deep learning framework through the data capture code. the output value;

The output value of the non-last network layer of the deep learning model trained in the second deep learning framework is captured through the data grabbing code.

In one possible implementation,

The data capture module is also configured to: capture the first output value of the last network layer of the deep learning model trained in the first deep learning framework through the data capture code; The grabbing code grabs the second output value of the last network layer of the deep learning model trained in the second deep learning framework;

The accuracy comparison module is specifically configured to: determine the first depth based on the information of the first intermediate variable set, the information of the second intermediate variable set, the first output value and the second output value. Test results of the learning framework.

In a possible implementation, the accuracy comparison module is specifically used to:

For a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, determine a difference value between the first intermediate variable and the second intermediate variable;

Wherein, the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In a possible implementation, the accuracy comparison module is specifically used to:

Taking the absolute values of the elements in the first intermediate variable and summing them up, the sum of the elements of the first intermediate variable is obtained;

The absolute values of the elements in the second intermediate variable are taken and then summed to obtain the sum of the elements of the second intermediate variable.

In a possible implementation, the accuracy comparison module is specifically used to:

The total number of elements in the first intermediate variable or the second intermediate variable is determined according to the first intermediate variable or the second intermediate variable.

In a possible implementation, the accuracy comparison module is specifically used to:

determining a standard difference value between the first intermediate variable and the second intermediate variable;

A first number of elements whose values are greater than a preset threshold among the standard difference values is determined.

In a possible implementation, the accuracy comparison module is specifically used to:

Determine the absolute value of the difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and the absolute value of the first intermediate variable;

The product is subtracted from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

In a possible implementation, the accuracy comparison module is also used to:

A ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable is determined.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a baseline deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

According to one aspect of the present disclosure, a testing device for a deep learning framework is provided, including:

The injection module is used to inject data capture code into the deep learning model deployed in the first deep learning framework;

A capture module, configured to train the deep learning model in the first deep learning framework, and capture the information of the first intermediate variable set of the deep learning model through the data capture code;

A determination module, configured to determine the test result of the first deep learning framework according to the information of the first intermediate variable set.

In a possible implementation, the determining module is specifically used to:

Get the base value;

Compare the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework.

In a possible implementation, the determining module is specifically used to:

Inject the data capture code into the deep learning model deployed in the second deep learning framework;

Train the deep learning model in the second deep learning framework, and capture the information of the second intermediate variable set of the deep learning model through the data capture code;

Compare the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework.

In one possible implementation,

The first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework;

The second intermediate variable set includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

In one possible implementation,

The information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

The information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

In one possible implementation,

The capture module is specifically configured to: capture the output value of the non-last network layer of the deep learning model trained in the first deep learning framework through the data capture code;

The determination module is specifically configured to: use the data capture code to capture the output value of the non-last network layer of the deep learning model trained in the second deep learning framework.

In one possible implementation,

The crawling module is also configured to: crawl through the data crawling code the first output value of the last network layer of the deep learning model trained in the first deep learning framework; crawl through the data Get the code to capture the second output value of the last network layer of the deep learning model trained in the second deep learning framework;

The determination module is specifically configured to: determine the first deep learning based on the information of the first intermediate variable set, the information of the second intermediate variable set, the first output value and the second output value. Framework test results.

In a possible implementation, the determining module is specifically used to:

For a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, determine a difference value between the first intermediate variable and the second intermediate variable;

Wherein, the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In a possible implementation, the determining module is specifically used to:

Taking the absolute values of the elements in the first intermediate variable and summing them up, the sum of the elements of the first intermediate variable is obtained;

The absolute values of the elements in the second intermediate variable are taken and then summed to obtain the sum of the elements of the second intermediate variable.

In a possible implementation, the determining module is specifically used to:

The total number of elements in the first intermediate variable or the second intermediate variable is determined according to the first intermediate variable or the second intermediate variable.

In a possible implementation, the determining module is specifically used to:

determining a standard difference value between the first intermediate variable and the second intermediate variable;

A first number of elements whose values are greater than a preset threshold among the standard difference values is determined.

In a possible implementation, the determining module is specifically used to:

Determine the absolute value of the difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and the absolute value of the first intermediate variable;

The product is subtracted from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

In a possible implementation, the determining module is also used to:

A ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable is determined.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a benchmark deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

According to an aspect of the present disclosure, an electronic device is provided, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the memory to store executable instructions to perform the above method.

According to an aspect of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

According to an aspect of the present disclosure, a computer program product is provided, including computer readable code, or a non-volatile computer readable storage medium carrying the computer readable code, when the computer readable code is in an electronic device When running, the processor in the electronic device executes the above method.

In an embodiment of the present disclosure, by injecting data capture code into the deep learning model deployed in the first deep learning framework, the deep learning model is trained in the first deep learning framework, and through the data capture The code captures the information of the first intermediate variable set of the deep learning model, and determines the test result of the first deep learning framework based on the information of the first intermediate variable set, thereby capturing through the data capture code The information on the intermediate variable set of the deep learning model trained in the deep learning framework can provide detailed information of the deep learning model trained in the deep learning framework, and help users determine which layer or layer determines the accuracy of the deep learning model. Which step of operation is triggered, so that the source of the accuracy difference can be traced, and the accuracy and functions of different deep learning frameworks can be accurately evaluated, and the efficiency of deep learning framework development and optimization can be improved. In addition, embodiments of the present disclosure do not require complete training of the deep learning model on the deep learning framework. For example, only one iteration can be trained, thereby saving time for evaluating the deep learning framework and achieving rapid deep learning. Framework evaluation, thereby further improving the efficiency of deep learning framework development and optimization. Among them, one iteration can mean that the deep learning model performs forward propagation and back propagation on a batch of data. By using the deep learning framework testing method provided by the embodiments of the present disclosure, users can quickly and accurately verify the function and reliability of the self-developed deep learning framework, which greatly improves the efficiency of developing and optimizing the deep learning framework. In addition, the testing method of the deep learning framework provided by the present invention is easy to use and only needs to inject data capture code without complicated settings and operations.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the disclosure.

Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of the drawings

The accompanying drawings herein are incorporated into and constitute a part of this specification. They illustrate embodiments consistent with the disclosure and, together with the description, serve to explain the technical solutions of the disclosure.

Figure 1 shows a flow chart of a testing method of a deep learning framework provided by an embodiment of the present disclosure.

Figure 2 shows a schematic diagram of the testing method of the deep learning framework provided by the embodiment of the present disclosure. The first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a benchmark deep learning framework.

Figure 3 shows a schematic diagram of the first deep learning framework and the second deep learning framework being the same type of deep learning framework based on different computing platforms in the testing method of the deep learning framework provided by the embodiment of the present disclosure.

FIG. 4 shows a schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure.

FIG. 5 shows another schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure.

Figure 6 shows another schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure.

FIG. 7 shows a block diagram of a testing device of the deep learning framework provided by an embodiment of the present disclosure.

FIG. 8 shows another block diagram of a testing device of the deep learning framework provided by an embodiment of the present disclosure.

FIG. 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings identify functionally identical or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" as used herein means "serving as an example, example, or illustrative." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or superior to other embodiments.

The term "and/or" in this article is just an association relationship that describes related objects, indicating that three relationships can exist. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C, which can mean including from A, Any one or more elements selected from the set composed of B and C.

In addition, in order to better explain the present disclosure, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits that are well known to those skilled in the art are not described in detail in order to emphasize the subject matter of the disclosure.

In related technologies, the method of evaluating the accuracy of a deep learning framework includes training the same deep learning model under each deep learning framework and evaluating it on the same data set. Although this method is intuitive, it cannot provide specific information about the accuracy difference. For example, it cannot determine which layer or step operation the difference is caused by, nor can it measure the specific degree of the difference. Furthermore, this approach is time-costly and requires complete model training and evaluation on each deep learning framework.

In order to solve technical problems similar to those described above, embodiments of the present disclosure provide a method for testing a deep learning framework by injecting data capture code into the deep learning model deployed in the first deep learning framework. The deep learning model is trained in the first deep learning framework, the information of the first intermediate variable set of the deep learning model is captured through the data capture code, and the information of the first intermediate variable set is determined based on the information of the first intermediate variable set. The test results of the first deep learning framework are described, whereby the information of the intermediate variable set of the deep learning model trained in the deep learning framework is captured through the data grabbing code, and detailed information of the deep learning model trained in the deep learning framework can be provided. , helps users determine which layer or step operation causes the accuracy of the deep learning model, so that the source of the accuracy difference can be traced, and the accuracy and functions of different deep learning frameworks can be accurately evaluated, and the deep learning framework can be improved. Development and optimization efficiency. In addition, embodiments of the present disclosure do not require complete training of the deep learning model on the deep learning framework. For example, only one iteration can be trained, thereby saving time for evaluating the deep learning framework and achieving rapid deep learning. Framework evaluation, thereby further improving the efficiency of deep learning framework development and optimization. Among them, one iteration can mean that the deep learning model performs forward propagation and back propagation on a batch of data.

By using the deep learning framework testing method provided by the embodiments of the present disclosure, users can quickly and accurately verify the function and reliability of the self-developed deep learning framework, which greatly improves the efficiency of developing and optimizing the deep learning framework. In addition, the testing method of the deep learning framework provided by the present invention is easy to use and only needs to inject data capture code without complicated settings and operations.

The testing method of the deep learning framework provided by the embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings.

Figure 1 shows a flow chart of a testing method of a deep learning framework provided by an embodiment of the present disclosure. In a possible implementation, the execution subject of the testing method of the deep learning framework may be a testing device of the deep learning framework. For example, the testing method of the deep learning framework may be executed by a terminal device or a server or other electronic equipment. . The terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device or a portable device. Wearable devices, etc. In a possible implementation, the execution subject of the testing method of the deep learning framework may be a testing tool of the deep learning framework. In some possible implementations, the testing method of the deep learning framework can be implemented by the processor calling computer-readable instructions stored in the memory. As shown in Figure 1, the testing method of the deep learning framework includes steps S11 to S13.

In step S11, data grabbing code is injected into the deep learning model deployed in the first deep learning framework.

In step S12, the deep learning model is trained in the first deep learning framework, and the information of the first intermediate variable set of the deep learning model is captured through the data capture code.

In step S13, the test result of the first deep learning framework is determined based on the information of the first intermediate variable set.

In embodiments of the present disclosure, a deep learning framework may represent a software library for designing, training, and validating deep learning models. In some application scenarios, the deep learning framework may also be called a deep learning training framework, etc., which is not limited here.

The first deep learning framework in the embodiment of the present disclosure may be an open source deep learning framework, such as TensorFlow, PyTorch, Caffe, Keras, Theano, CNTK, MXNet, etc., or it may be a self-developed deep learning framework.

In embodiments of the present disclosure, data grabbing code is injected into the deep learning model deployed in the first deep learning framework.

In a possible implementation, data capture code can be injected into multiple network layers deployed in the deep learning model of the first deep learning framework to capture information on intermediate variables corresponding to the multiple network layers, Get the information of the first intermediate variable set. For example, data capture code can be injected into each network layer deployed in the deep learning model of the first deep learning framework to capture information on intermediate variables corresponding to each network layer and obtain information on the first intermediate variable set. For another example, data capture code can be injected into each network layer containing intermediate variables in the deep learning model deployed in the first deep learning framework to capture the information of the intermediate variables corresponding to each network layer to obtain the first intermediate variable. collection of information.

In one possible implementation, the data fetching code can be called a hook. Among them, a hook is a processing mechanism, usually a function, that can be called when a specific event or condition occurs, or inserted into the normal program flow at a specific point. In deep learning frameworks, hooks can be used to capture intermediate variables at specific layers or stages of the model.

In an example, in the case of training using MMCV, the data scraping code can be:

In one possible implementation, the data grabbing code can be injected into the deep learning model deployed in the first deep learning framework through the data grabbing module in the testing tool of the deep learning framework.

In a possible implementation, the information of the first intermediate variable set of the deep learning model trained in the first deep learning framework can be captured through the data capture module in the testing tool of the deep learning framework. .

In a possible implementation, when each layer of the deep learning model deployed in the first deep learning framework performs forward propagation and back propagation, the intermediate variables of this layer can be captured through the data grabbing code.

In a possible implementation, after the information of the first intermediate variable set of the deep learning model is captured through the data capture code, the information of the first intermediate variable set may be stored.

In a possible implementation, determining the test result of the first deep learning framework based on the information of the first intermediate variable set includes: obtaining a benchmark value; Compare with the benchmark value to determine the test result of the first deep learning framework.

In this implementation, the baseline value may represent a value of a baseline used to determine the test results of the first deep learning framework. As an example of this implementation, the data grabbing code can be injected into the deep learning model deployed in the second deep learning framework, the deep learning model can be trained in the second deep learning framework, and The information of the second intermediate variable set of the deep learning model is captured through the data capture code, and the information of the second intermediate variable set is used as a reference value. As another example of this implementation, the reference value may be a preset value.

In this implementation, by injecting data capture code into the deep learning model deployed in the first deep learning framework, the deep learning model is trained in the first deep learning framework, and the data capture code is The code captures the information of the first intermediate variable set of the deep learning model, obtains a benchmark value, and compares the information of the first intermediate variable set with the benchmark value to determine the test result of the first deep learning framework , from which the benchmark value can be used to determine which layer or step of operation the accuracy difference is caused by, so that the source of the accuracy difference can be traced, and then the accuracy and function of the deep learning framework can be accurately evaluated, and the efficiency of the development and optimization of the deep learning framework can be improved. .

In a possible implementation, obtaining the benchmark value includes: injecting the data capture code into the deep learning model deployed in the second deep learning framework; in the second deep learning framework Train the deep learning model, and capture the information of the second intermediate variable set of the deep learning model through the data capture code; compare the information of the first intermediate variable set with the reference value, Determining the test result of the first deep learning framework includes: comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework.

In this implementation, the second deep learning framework can be an open source deep learning framework, such as TensorFlow, PyTorch, Caffe, Keras, Theano, CNTK, MXNet, etc., or it can be a self-developed deep learning framework.

In this implementation, the reference value includes information for the second set of intermediate variables.

In this implementation, the data crawling code is injected into the deep learning model deployed in the first deep learning framework, and the data crawling code is injected into the deep learning model deployed in the second deep learning framework. .

As an example of this implementation, data capture code can be injected into multiple network layers deployed in the deep learning model of the first deep learning framework to capture information about intermediate variables corresponding to the multiple network layers, and we obtain The information of the first intermediate variable set; data capture code can be injected into multiple network layers deployed in the deep learning model of the second deep learning framework to capture the information of the intermediate variables corresponding to the multiple network layers, and we obtain Information about the second intermediate variable set. For example, data capture code can be injected into each network layer in the deep learning model deployed in the first deep learning framework to capture the information of the intermediate variables corresponding to each network layer to obtain the information of the first intermediate variable set; Data capture code can be injected into each network layer in the deep learning model deployed in the second deep learning framework to capture information on intermediate variables corresponding to each network layer and obtain information on the second intermediate variable set. For another example, data capture code can be injected into each network layer containing intermediate variables in the deep learning model deployed in the first deep learning framework to capture the information of the intermediate variables corresponding to each network layer to obtain the first intermediate variable. set of information; the data capture code can be injected into each network layer containing intermediate variables in the deep learning model deployed in the second deep learning framework to capture the information of the intermediate variables corresponding to each network layer to obtain the second intermediate Variable set information.

In this implementation, the data grabbing code is injected into the deep learning model deployed in the first deep learning framework, and the data grabbing code is injected into the deep learning model deployed in the second deep learning framework. Code to train the deep learning model in the first deep learning framework, and capture the information of the first intermediate variable set of the deep learning model through the data capture code, and in the second deep learning framework Train the deep learning model, capture the information of the second intermediate variable set of the deep learning model through the data capture code, and combine the information of the first intermediate variable set with the second intermediate variable set. Compare the information to determine the test results of the first deep learning framework, thereby using the data capture code to capture the information of the intermediate variable set of the same deep learning model trained in the two deep learning frameworks, which can provide deep learning The specific information about the accuracy difference of the framework can help users determine which layer or step operation the difference is caused by, so that the source of the accuracy difference can be traced, and the accuracy and functions of different deep learning frameworks can be accurately evaluated to improve deep learning. Efficiency in framework development and optimization. In addition, embodiments of the present disclosure do not require complete training of the deep learning model on the deep learning framework. For example, only one iteration can be trained, thereby saving time for evaluating the deep learning framework and achieving rapid deep learning. Framework evaluation, thereby further improving the efficiency of deep learning framework development and optimization. Among them, one iteration can mean that the deep learning model performs forward propagation and back propagation on a batch of data.

The testing method of the deep learning framework provided by this implementation can not only provide detailed information during the training process of the deep learning model, but also quickly and efficiently compare the accuracy between multiple deep learning frameworks. By using the testing method of the deep learning framework provided by this implementation, users can quickly and accurately verify the functionality and reliability of the self-developed deep learning framework, greatly improving the efficiency of developing and optimizing the deep learning framework. In addition, the testing method of the deep learning framework provided by this implementation is easy to use, and only needs to inject data grabbing code, without the need for complex settings and operations.

As an example of this implementation, the data grabbing module in the test tool of the deep learning framework can be used to inject the data grabbing code into the deep learning model deployed in the first deep learning framework, and then inject the data grabbing code into the deep learning model deployed in the second deep learning framework. Inject the data capture code into the deep learning model of the learning framework.

As an example of this implementation, the information of the first intermediate variable set of the deep learning model trained in the first deep learning framework can be captured through the data capture module in the testing tool of the deep learning framework, and information about a second set of intermediate variables of the deep learning model trained in the second deep learning framework.

As an example of this implementation, when each layer of the deep learning model deployed in the first deep learning framework performs forward propagation and back propagation, the intermediate variables of this layer can be captured through the data grabbing code; when deploying When each layer of the deep learning model of the second deep learning framework performs forward propagation and back propagation, the intermediate variables of this layer can be captured through the data grabbing code.

As an example of this implementation, after the information of the first intermediate variable set of the deep learning model is captured through the data capture code, the information of the first intermediate variable set can be stored; After the code captures the information of the second intermediate variable set of the deep learning model, the information of the second intermediate variable set can be stored.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a baseline deep learning framework.

In this implementation, the first deep learning framework may be a self-developed deep learning framework to be tested, and the second deep learning framework may be a benchmark deep learning framework. For example, the second deep learning framework can be TensorFlow, PyTorch, Caffe, etc.

For example, a company has developed a deep learning framework and hopes that its performance will exceed that of open source deep learning frameworks. Then, the deep learning framework developed by the company can be used as the first deep learning framework, and a certain open source deep learning framework (such as PyTorch) can be used as the second deep learning framework.

Figure 2 shows a schematic diagram of the testing method of the deep learning framework provided by the embodiment of the present disclosure. The first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a benchmark deep learning framework. In the example shown in Figure 2, you can use the self-developed deep learning framework as the first deep learning framework and Pytorch as the second deep learning framework. The data capture code can be injected into the deep learning model deployed in the self-developed deep learning framework, and the data capture code can be injected into the deep learning model deployed in Pytorch. The information of the intermediate variables in a training iteration of the deep learning model trained in the self-developed deep learning framework can be captured through the data capture code to obtain the information of the first intermediate variable set; the information of the first intermediate variable set can be obtained through the data The capture code captures the information of the intermediate variables in one training iteration of the deep learning model trained in Pytorch, and obtains the information of the second intermediate variable set. The information of the first intermediate variable set can be compared with the information of the second intermediate variable set to obtain the test results of the self-developed deep learning framework.

By adopting this implementation method, specific information about the accuracy difference between the self-developed deep learning framework and the benchmark deep learning framework can be provided, so that users can tune and improve the self-developed deep learning framework in a targeted manner.

As an example of this implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

Among them, the type of computing platform can be CPU, CUDA, etc., which is not limited here. Different types of computing platforms correspond to different types of computing devices.

In this example, the first deep learning framework and the second deep learning framework perform training or calculation (computing) based on the same type of computing platform/computing architecture. For example, the first deep learning framework and the second deep learning framework are both based on CPU for training or operation; for example, the first deep learning framework and the second deep learning framework are both based on CUDA; and so on.

In this example, when the first deep learning framework is a self-developed deep learning framework and the second deep learning framework is a benchmark deep learning framework, the self-developed deep learning framework and the benchmark deep learning framework are Training or computing based on the same type of computing platform helps to more accurately determine the accuracy difference between the self-developed deep learning framework and the benchmark deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

For example, the first deep learning framework and the second deep learning framework are both Pytorch, the first deep learning framework is based on CPU, and the second deep learning framework is based on CUDA; for another example, the first deep learning framework and the second deep learning framework are both TensorFlow , the first deep learning framework is based on CUDA10.2, the second deep learning framework is based on CUDA11.1; and so on.

Figure 3 shows a schematic diagram of the first deep learning framework and the second deep learning framework being the same type of deep learning framework based on different computing platforms in the testing method of the deep learning framework provided by the embodiment of the present disclosure. In the example shown in Figure 3, the first deep learning framework and the second deep learning framework are both Pytorch, the first deep learning framework is based on CPU, and the second deep learning framework is based on CUDA. The data scraping code may be injected into the deep learning model deployed in the first deep learning framework, and the data scraping code may be injected into the deep learning model deployed in the second deep learning framework. The information of the intermediate variables in a training iteration of the deep learning model trained in the first deep learning framework can be captured through the data capture code to obtain the information of the first intermediate variable set; the information of the first intermediate variable set can be obtained through the data capture The code is retrieved to capture the information of the intermediate variables in one training iteration of the deep learning model trained in the second deep learning framework, and obtain the information of the second intermediate variable set. The information of the first intermediate variable set can be compared with the information of the second intermediate variable set to obtain the test results of the self-developed deep learning framework.

By adopting this implementation, it is possible to provide specific information on the accuracy differences between deep learning frameworks of the same type based on different computing platforms.

In a possible implementation, the first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework; the second intermediate variable set It includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

The first parameter set may include learnable parameters of each network layer of the deep learning model (i.e., parameters to be trained), and the second parameter set may include learnable parameters of each network layer of the deep learning model. (i.e. the parameters to be trained). Since the same deep learning model is trained on the first deep learning framework and the second deep learning framework, the parameters in the first parameter set correspond to the parameters in the second parameter set one-to-one.

In this implementation, by based on the information of the first parameter set to be trained in the deep learning model trained in the first deep learning framework, and the depth training in the second deep learning framework The information of the second parameter set to be trained in the learning model is determined to determine the test results of the first deep learning framework, thereby helping the user to determine which layer or step operation caused the accuracy difference, so that the accuracy difference can be traced It can accurately evaluate the accuracy and functions of different deep learning frameworks and improve the efficiency of deep learning framework development and optimization.

In a possible implementation, the information of the first parameter set includes: the values and/or gradients of parameters in the first parameter set; the information of the second parameter set includes: the second parameter set The values and/or gradients of the parameters.

As an example of this implementation, the information of the first parameter set includes: the values of the parameters in the first parameter set; the information of the second parameter set includes: the values of the parameters in the second parameter set.

In this example, by according to the values of parameters in the first parameter set of the deep learning model trained in the first deep learning framework, and the deep learning model trained in the second deep learning framework The values of the parameters in the second parameter set determine the test results of the first deep learning framework, thereby helping to track and understand the training process and results of the deep learning model.

As another example of this implementation, the information of the first parameter set includes: the gradient of the parameters in the first parameter set; the information of the second parameter set includes: the gradient of the parameters in the second parameter set. .

In this example, by according to the gradient of the parameters in the first parameter set of the deep learning model trained in the first deep learning framework, and the deep learning model trained in the second deep learning framework The gradient of the parameters in the second parameter set determines the test result of the first deep learning framework, thereby helping to track and understand the training process and results of the deep learning model.

As another example of this implementation, the information of the first parameter set includes: the values and gradients of the parameters in the first parameter set; the information of the second parameter set includes: the parameters of the second parameter set values and gradients.

In this example, by based on the values and gradients of parameters in the first parameter set of the deep learning model trained in the first deep learning framework, and the deep learning model trained in the second deep learning framework The values and gradients of parameters in the second parameter set of the learning model determine the test results of the first deep learning framework, thereby helping to track and understand the training process and results of the deep learning model.

In a possible implementation, using the data grabbing code to capture the information of the first intermediate variable set of the deep learning model includes: using the data grabbing code to capture the first set of intermediate variables in the deep learning model. The output value of the non-last network layer of the deep learning model trained in the deep learning framework; the information of the second intermediate variable set of the deep learning model captured through the data capture code includes: The data capture code captures the output value of the non-last network layer of the deep learning model trained in the second deep learning framework.

In this implementation, the first intermediate variable set includes: the output of the non-last network layer of the deep learning model trained in the first deep learning framework; the information of the first intermediate variable set includes : the output value of the non-last network layer of the deep learning model trained in the first deep learning framework; the second intermediate variable set includes: the depth of the deep learning model trained in the second deep learning framework The output of the non-last network layer of the learning model; the information of the second intermediate variable set includes: the output value of the non-last network layer of the deep learning model trained in the second deep learning framework.

That is, in this implementation, the first intermediate variable set may include: the output of the first network layer to the penultimate network layer of the deep learning model trained in the first deep learning framework; the second The intermediate variable set may include: outputs from the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

The information of the first intermediate variable set may include: the output values from the first network layer to the penultimate network layer of the deep learning model trained in the first deep learning framework; the information of the second intermediate variable set It may include: output values from the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

In this implementation, by based on the output value of the non-last network layer of the deep learning model trained in the first deep learning framework, and the deep learning trained in the second deep learning framework The output value of the non-last network layer of the model determines the test result of the first deep learning framework, thereby helping the user to determine which layer or step operation caused the accuracy difference, so that the source of the accuracy difference can be traced , which can accurately evaluate the accuracy and functionality of different deep learning frameworks and improve the efficiency of deep learning framework development and optimization.

In another possible implementation, the first intermediate variable set includes: inputs from non-first network layers of the deep learning model trained in the first deep learning framework; the first intermediate variable set The information of the variable set includes: the input values of non-first network layers of the deep learning model trained in the first deep learning framework; the second intermediate variable set includes: the input values of the non-first network layer of the deep learning model trained in the first deep learning framework; The input of the non-first network layer of the deep learning model trained in the second deep learning framework; the information of the second intermediate variable set includes: the non-first network layer of the deep learning model trained in the second deep learning framework. The input value of the network layer.

In this implementation, the first intermediate variable set may include: inputs from the second network layer to the last network layer of the deep learning model trained in the first deep learning framework; the second intermediate variable set may Including: inputs from the second network layer to the last network layer of the deep learning model trained in the second deep learning framework.

The information of the first intermediate variable set may include: input values from the second network layer to the last network layer of the deep learning model trained in the first deep learning framework; the information of the second intermediate variable set may include : the input values from the second network layer to the last network layer of the deep learning model trained in the second deep learning framework.

In a possible implementation, the first intermediate variable set may include: learnable parameters of each network layer of the deep learning model trained in the first deep learning framework, and The output of the first network layer to the penultimate network layer of the deep learning model; the second intermediate variable set may include: the learnable values of each network layer of the deep learning model trained in the second deep learning framework parameters, and the outputs of the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

The information of the first intermediate variable set may include: values of learnable parameters of each network layer of the deep learning model trained in the first deep learning framework and gradients of the parameters, and in the first deep learning framework The output values of the first network layer to the penultimate network layer of the trained deep learning model; the information of the second intermediate variable set may include: each network of the deep learning model trained in the second deep learning framework The values of the learnable parameters of the layer and the gradient of the parameters, as well as the output values of the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

In a possible implementation, the method further includes: using the data capture code to capture the first output value of the last network layer of the deep learning model trained in the first deep learning framework. ; Capture the second output value of the last network layer of the deep learning model trained in the second deep learning framework through the data grabbing code; said combining the information of the first intermediate variable set with Comparing the information of the second intermediate variable set to determine the test result of the first deep learning framework includes: based on the information of the first intermediate variable set, the information of the second intermediate variable set, the third An output value and the second output value determine the test result of the first deep learning framework.

In this implementation, the first output value may represent the final output result of the deep learning model trained in the first deep learning framework, and the second output value may represent the final output result of the deep learning model trained in the second deep learning framework. The final output result of the deep learning model.

In this implementation, the first output value of the last network layer of the deep learning model trained in the first deep learning framework is captured through the data grabbing code. Get the second output value of the last network layer of the deep learning model trained in the second deep learning framework, based on the information of the first intermediate variable set, the information of the second intermediate variable set, the The first output value and the second output value are used to determine the test result of the first deep learning framework, thereby combining the final output result of the deep learning model trained in the first deep learning framework and the The final output result of the deep learning model trained in the second deep learning framework is used to obtain the test results of the first deep learning framework, thereby helping to more accurately evaluate the accuracy of the deep learning framework to be tested.

In this embodiment of the present disclosure, the information of the first intermediate variable set can be compared with the information of the second intermediate variable set to determine the test result of the first deep learning framework. By providing the user with the test results of the first deep learning framework, the user can quickly discover the specific accuracy difference between the first deep learning framework and the second deep learning framework, and can perform targeted tuning and improvement.

In a possible implementation, the information of the first intermediate variable set can be compared with the information of the second intermediate variable set through the accuracy comparison module in the testing tool of the deep learning framework to determine the third intermediate variable set. Test results of a deep learning framework.

In a possible implementation, the information of the first intermediate variable set and the information of the second intermediate variable set can be read from a storage system to compare the information of the first intermediate variable set with the information of the second intermediate variable set. The information of the second intermediate variable set is compared to determine the test result of the first deep learning framework.

In an embodiment of the present disclosure, after obtaining the information of the first intermediate variable set and the information of the second intermediate variable set, the information of the first intermediate variable set and the second intermediate variable set may be The information on the corresponding intermediate variables in the information is compared separately.

In a possible implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes: for the The first intermediate variable in the first intermediate variable set and the second intermediate variable in the second intermediate variable set are used to determine the difference value between the first intermediate variable and the second intermediate variable; wherein, the first intermediate variable An intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In this implementation, the corresponding elements in the first intermediate variable and the second intermediate variable can be subtracted and then the absolute value is obtained to obtain the difference value corresponding to the element; each element in the first intermediate variable and the second intermediate variable can be The corresponding difference values are summed to obtain the difference value between the first intermediate variable and the second intermediate variable.

In one example, the difference value between the first intermediate variable and the second intermediate variable may be represented by L1_diff.

In this implementation, the test result of the first deep learning framework may include a difference value between the first intermediate variable and the second intermediate variable.

In this implementation, for a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, a relationship between the first intermediate variable and the second intermediate variable is determined. The difference value of , wherein the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model, thereby providing the user with the deep learning model trained in the first deep learning framework and The difference values between corresponding intermediate variables in the deep learning model trained in the second deep learning framework help users quickly discover the bottleneck of the deep learning framework.

As an example of this implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes: The elements in the first intermediate variable are taken as absolute values and then summed to obtain the sum of the elements of the first intermediate variable; the elements in the second intermediate variable are taken as absolute values and then summed to obtain the second intermediate variable The sum of the elements.

In one example, the sum of the elements of the first intermediate variable can be represented by L1_data1, and the sum of the elements of the second intermediate variable can be represented by L1_data2.

In this example, the test result of the first deep learning framework may include the sum of elements of the first intermediate variable and the sum of elements of the second intermediate variable.

In this example, by taking the absolute values of the elements in the first intermediate variable and summing them, for the first intermediate variable in the first intermediate variable set and the second intermediate variable in the second intermediate variable set, Obtain the sum of the elements of the first intermediate variable, take the absolute value of the elements in the second intermediate variable and sum them up to obtain the sum of the elements of the second intermediate variable, thereby providing the user with the first The sum of the elements of the deep learning model trained in the deep learning framework and the corresponding intermediate variables in the deep learning model trained in the second deep learning framework helps users quickly discover the bottlenecks of the deep learning framework.

As an example of this implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes: according to the The first intermediate variable or the second intermediate variable determines the total number of elements in the first intermediate variable or the second intermediate variable.

Since the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model, the total number of elements in the first intermediate variable is equal to the total number of elements in the second intermediate variable. Thus, in this example, the total number of elements in the first intermediate variable may be determined based on the first intermediate variable, or the total number of elements in the second intermediate variable may be determined based on the second intermediate variable.

In one example, the total number of elements in the first intermediate variable or the second intermediate variable may be represented by total.

In this example, the test result of the first deep learning framework may include the total number of elements in the first intermediate variable or the second intermediate variable.

In this example, by determining, for a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, the first intermediate variable or the second intermediate variable, The total number of elements in the first intermediate variable or the second intermediate variable, so that the user can intuitively know the number of elements in the first intermediate variable and the second intermediate variable.

As an example of this implementation, comparing the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework includes: determining the A standard difference value between the first intermediate variable and the second intermediate variable; determining a first number of elements in the standard difference value whose value is greater than a preset threshold.

In this example, the standard difference value can be used to measure whether the corresponding elements in the corresponding intermediate variables are close, that is, it can be used to measure whether the precision difference between the corresponding elements in the corresponding intermediate variables is large.

In one example, the standard difference value can be expressed as diff_v.

In one example, the preset threshold includes 0.01, and the first number of elements with a value greater than 0.01 in the standard difference value can be represented by 0.01_n.

In another example, the preset threshold includes 0.001, and the first number of elements with a value greater than 0.001 in the standard difference value can be represented by 0.001_n.

In another example, the preset threshold includes 0.0001, and the first number of elements with a value greater than 0.0001 in the standard difference value can be represented by 0.0001_n.

In another example, the preset threshold includes 0.00001, and the first number of elements with a value greater than 0.00001 in the standard difference value can be represented by 0.00001_n.

In this example, the test results of the first deep learning framework may include a first number of elements whose values are greater than a preset threshold among the standard difference values between the first intermediate variable and the second intermediate variable.

In this example, for a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, the distance between the first intermediate variable and the second intermediate variable is determined. The standard difference value determines the first number of elements whose value is greater than the preset threshold among the standard difference values, thereby facilitating the user to determine whether the corresponding elements in the corresponding intermediate variables are close.

In one example, determining the standard difference value between the first intermediate variable and the second intermediate variable includes: determining the absolute value of the difference between the first intermediate variable and the second intermediate variable. ; Determine the product of the preset tolerance ratio and the absolute value of the first intermediate variable; subtract the product from the absolute value of the difference to obtain the relationship between the first intermediate variable and the second intermediate variable standard difference value.

For example, diff_v=|data1-data2|-rtol×|data1|,

Among them, diff_v represents the standard difference value, data1 represents the first intermediate variable, data2 represents the second intermediate variable, and rtol represents the preset tolerance ratio. rtol×|data1| can be called relative tolerance.

Among them, the preset tolerance ratio can be greater than 0 and less than 0.01. For example, the preset tolerance ratio can be 0.00001, etc., which is not limited here.

In some application scenarios, the preset tolerance ratio may also be called relative tolerance, etc., which is not limited here.

In this example, by determining the absolute value of the difference between the first intermediate variable and the second intermediate variable, the product of the preset tolerance ratio and the absolute value of the first intermediate variable is determined, and the The absolute value of the difference is subtracted from the product to obtain the standard difference value between the first intermediate variable and the second intermediate variable. The standard difference value determined thereby helps to determine the corresponding intermediate variable. Whether the corresponding elements are close.

In one example, after determining a first number of elements whose values are greater than a preset threshold in the standard difference values, the method further includes: determining a relationship between the first number and the first intermediate variable or the first intermediate variable. The ratio of the total number of elements in the second intermediate variable.

In one example, the preset threshold includes 0.01. The first number of elements with a value greater than 0.01 in the standard difference value can be expressed as 0.01_n. The first number is related to the first intermediate variable or the second intermediate variable. The ratio of the total number of elements in the variable can be expressed as 0.01_r. For example, if the first number of elements with a value greater than 0.01 in the standard difference value is 1000, and the elements in the first intermediate variable are 10000, then 0.01_r=0.1. That is, among the first intermediate variable and the second intermediate variable, 10% of the elements have a large accuracy difference.

In another example, the preset threshold includes 0.001, the first number of elements with a value greater than 0.001 in the standard difference value can be expressed as 0.001_n, and the first number is related to the first intermediate variable or the second The ratio of the total number of elements in the intermediate variable can be expressed as 0.001_r.

In another example, the preset threshold includes 0.0001, the first number of elements with a value greater than 0.0001 in the standard difference value can be represented by 0.0001_n, and the first number is related to the first intermediate variable or the second The ratio of the total number of elements in the intermediate variable can be expressed as 0.0001_r.

In another example, the preset threshold includes 0.00001, and the first number of elements with a value greater than 0.00001 in the standard difference value can be represented by 0.00001_n. The first number is related to the first intermediate variable or the second The ratio of the total number of elements in the intermediate variable can be expressed as 0.00001_r.

In this example, the test result of the first deep learning framework may include a ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable.

In this example, by determining the ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable, the accuracy in the first intermediate variable and the second intermediate variable can be provided to the user. The proportion of elements that differ greatly.

In one example, the method further includes: in response to the value of any element in the standard difference value being greater than a preset threshold, issuing prompt information.

In a possible implementation, after determining the test result of the first deep learning framework, the method may further include: generating an accuracy difference report based on the test result of the first deep learning framework.

FIG. 4 shows a schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure. Figure 4 shows the accuracy difference report corresponding to the output value and the input value. In Figure 4, Item_Name can represent the name of the intermediate variable, the number after feature can represent the position of the network layer in the deep learning model, bottom can represent the output value, top can represent the input value, and L1_data1 can represent the element of the first intermediate variable. L1_data2 can represent the sum of elements of the second intermediate variable, L1_diff can represent the difference value between the first intermediate variable and the second intermediate variable, and total can represent the total number of elements in the first intermediate variable (i.e., the second intermediate variable (the total number of elements in ), 0.01_n can represent the number of elements with a standard difference value greater than 0.01, 0.001_n can represent the number of elements with a standard difference value greater than 0.001, 0.0001_n can represent the standard difference value with a median value greater than 0.0001 The number of elements, 0.00001_n can represent the number of elements with a standard difference value greater than 0.00001, 0.01_r can represent the ratio of the number of elements with a standard difference value greater than 0.01 to the total number of elements in the first intermediate variable, 0.001_r can Represents the ratio of the number of elements with a median standard difference value greater than 0.001 to the total number of elements in the first intermediate variable. 0.0001_r can represent the ratio of the number of elements with a median standard difference value greater than 0.0001 to the total number of elements in the first intermediate variable. , 0.00001_r can represent the ratio of the number of elements with a standard difference value greater than 0.00001 to the total number of elements in the first intermediate variable.

FIG. 5 shows another schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure. Figure 5 shows the accuracy difference report corresponding to the parameter value. In Figure 5, Item_Name can represent the name of the intermediate variable, the number after feature can represent the position of the network layer in the deep learning model, weight can represent the weight, bias can represent the bias, and L1_data1 can represent one of the elements of the first intermediate variable. and, L1_data2 can represent the sum of elements of the second intermediate variable, L1_diff can represent the difference value between the first intermediate variable and the second intermediate variable, and total can represent the total number of elements in the first intermediate variable (that is, in the second intermediate variable The total number of elements), 0.01_n can represent the number of elements with a standard difference value greater than 0.01, 0.001_n can represent the number of elements with a standard difference value greater than 0.001, 0.0001_n can represent the elements with a standard difference value greater than 0.0001 The number of elements, 0.00001_n can represent the number of elements with a standard difference value greater than 0.00001, 0.01_r can represent the ratio of the number of elements with a standard difference value greater than 0.01 to the total number of elements in the first intermediate variable, 0.001_r can represent The ratio of the number of elements with the median standard difference value greater than 0.001 to the total number of elements in the first intermediate variable. 0.0001_r can represent the ratio of the number of elements with the median standard difference value greater than 0.0001 to the total number of elements in the first intermediate variable. 0.00001_r can represent the ratio of the number of elements with a standard difference value greater than 0.00001 to the total number of elements in the first intermediate variable.

Figure 6 shows another schematic diagram of an accuracy difference report in the testing method of the deep learning framework provided by the embodiment of the present disclosure. Figure 6 shows the accuracy difference report corresponding to the gradient. In Figure 6, Item_Name can represent the name of the intermediate variable, weight can represent the weight, bias can represent the bias, L1_data1 can represent the sum of the elements of the first intermediate variable, L1_data2 can represent the sum of the elements of the second intermediate variable, and L1_diff can Represents the difference value between the first intermediate variable and the second intermediate variable, total can represent the total number of elements in the first intermediate variable (that is, the total number of elements in the second intermediate variable), and 0.01_n can represent that the median standard difference value is greater than 0.01 The number of elements, 0.001_n can represent the number of elements with the standard difference value greater than 0.001, 0.0001_n can represent the number of elements with the standard difference value greater than 0.0001, 0.00001_n can represent the elements with the standard difference value greater than 0.00001 The number of The ratio of the total number of elements in , 0.0001_r can represent the ratio of the number of elements with the standard difference value greater than 0.0001 to the total number of elements in the first intermediate variable, 0.00001_r can represent the number of elements with the standard difference value greater than 0.00001 and The ratio of the total number of elements in the first intermediate variable.

The deep learning framework testing method provided by the embodiments of the present disclosure can be applied to technical fields such as software testing, and is not limited here. The deep learning framework testing method provided by the embodiments of the present disclosure can be applied to deep learning software libraries, autonomous driving systems, image recognition or speech recognition products, etc., to optimize their deep learning frameworks.

The following describes the testing method of the deep learning framework provided by the embodiment of the present disclosure through a specific application scenario.

In this application scenario, a company developed a deep learning framework and hoped that its performance would surpass the open source deep learning framework. In this application scenario, the deep learning framework developed by the company can be used as the first deep learning framework, and an open source deep learning framework (such as PyTorch) can be used as the second deep learning framework.

Through the data capture module of the test tool of the deep learning framework, the data capture code can be injected into the deep learning model deployed in the first deep learning framework, and in the deep learning model deployed in the second deep learning framework , inject the data scraping code. The data capture module can be used to capture the information of the first intermediate variable set of the deep learning model trained in the first deep learning framework in one training iteration, and all the information of the deep learning model trained in the second deep learning framework. Describe the information of the second intermediate variable set of the deep learning model in a training iteration, and store the information of the first intermediate variable set and the information of the second intermediate variable set.

Wherein, the first intermediate variable set may include: learnable parameters of each network layer of the deep learning model trained in the first deep learning framework, and the first deep learning model trained in the first deep learning framework. The output of one network layer to the penultimate network layer; the second intermediate variable set may include: the learnable parameters of each network layer of the deep learning model trained in the second deep learning framework, and the The output of the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

The information of the first intermediate variable set may include: values of learnable parameters of each network layer of the deep learning model trained in the first deep learning framework and gradients of the parameters, and in the first deep learning framework The output values of the first network layer to the penultimate network layer of the trained deep learning model; the information of the second intermediate variable set may include: each network of the deep learning model trained in the second deep learning framework The values of the learnable parameters of the layer and the gradient of the parameters, as well as the output values of the first network layer to the penultimate network layer of the deep learning model trained in the second deep learning framework.

In this application scenario, the test results of the first deep learning framework can be determined based on the information of the first intermediate variable set and the information of the second intermediate variable set, and the test results of the first deep learning framework can be quickly Locate accuracy differences and guide optimization.

It can be understood that the above-mentioned method embodiments mentioned in this disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the details will not be described in this disclosure. Those skilled in the art can understand that in the above-mentioned methods of specific embodiments, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides a deep learning framework testing device, electronic equipment, computer-readable storage media, and computer program products. The above can be used to implement any deep learning framework testing method provided by the present disclosure. The corresponding technical solutions and For technical effects, please refer to the corresponding records in the methods section and will not be described again.

FIG. 7 shows a block diagram of a testing device of the deep learning framework provided by an embodiment of the present disclosure. As shown in Figure 7, the testing device of the deep learning framework includes:

The data capture module 71 is configured to inject data capture code into the deep learning model deployed in the first deep learning framework, train the deep learning model in the first deep learning framework, and capture the data through the data capture module. Get the code to capture the information of the first intermediate variable set of the deep learning model;

The accuracy comparison module 72 is configured to determine the test result of the first deep learning framework based on the information of the first intermediate variable set.

In the embodiment of the present disclosure, the testing device of the deep learning framework may also be called a testing tool of the deep learning framework.

In one possible implementation,

The data capture module 71 is also used to: obtain the reference value;

The accuracy comparison module 72 is specifically configured to compare the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework.

In one possible implementation,

The data capture module 71 is specifically configured to: inject the data capture code into the deep learning model deployed in the second deep learning framework; train the deep learning model in the second deep learning framework , and capture the information of the second intermediate variable set of the deep learning model through the data capture code;

The accuracy comparison module 72 is specifically configured to compare the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework.

In one possible implementation,

The first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework;

The second intermediate variable set includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

In one possible implementation,

The information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

The information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

In a possible implementation, the data capture module 71 is specifically configured to: capture the non-last network of the deep learning model trained in the first deep learning framework through the data capture code. The output value of the layer;

The output value of the non-last network layer of the deep learning model trained in the second deep learning framework is captured through the data grabbing code.

In one possible implementation,

The data capture module 71 is also configured to: capture the first output value of the last network layer of the deep learning model trained in the first deep learning framework through the data capture code; The data capture code captures the second output value of the last network layer of the deep learning model trained in the second deep learning framework;

The accuracy comparison module 72 is specifically configured to: determine the first output value according to the information of the first intermediate variable set, the information of the second intermediate variable set, the first output value and the second output value. Test results of deep learning frameworks.

In a possible implementation, the accuracy comparison module 72 is specifically used to:

For a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, determine a difference value between the first intermediate variable and the second intermediate variable;

Wherein, the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In a possible implementation, the accuracy comparison module 72 is specifically used to:

Taking the absolute values of the elements in the first intermediate variable and summing them up, the sum of the elements of the first intermediate variable is obtained;

The absolute values of the elements in the second intermediate variable are taken and then summed to obtain the sum of the elements of the second intermediate variable.

In a possible implementation, the accuracy comparison module 72 is specifically used to:

The total number of elements in the first intermediate variable or the second intermediate variable is determined according to the first intermediate variable or the second intermediate variable.

In a possible implementation, the accuracy comparison module 72 is specifically used to:

determining a standard difference value between the first intermediate variable and the second intermediate variable;

A first number of elements whose values are greater than a preset threshold among the standard difference values is determined.

In a possible implementation, the accuracy comparison module 72 is specifically used to:

Determine the absolute value of the difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and the absolute value of the first intermediate variable;

The product is subtracted from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

In a possible implementation, the accuracy comparison module 72 is also used to:

A ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable is determined.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a benchmark deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

FIG. 8 shows another block diagram of a testing device of the deep learning framework provided by an embodiment of the present disclosure. As shown in Figure 8, the testing device of the deep learning framework includes:

Injection module 81 is used to inject data capture code into the deep learning model deployed in the first deep learning framework;

The capture module 82 is used to train the deep learning model in the first deep learning framework, and capture the information of the first intermediate variable set of the deep learning model through the data capture code;

The determination module 83 is configured to determine the test result of the first deep learning framework according to the information of the first intermediate variable set.

In a possible implementation, the determining module 83 is specifically used to:

Get the base value;

Compare the information of the first intermediate variable set with the reference value to determine the test result of the first deep learning framework.

In a possible implementation, the determination module 83 is specifically used to:

Inject the data grabbing code into the deep learning model deployed in the second deep learning framework;

Train the deep learning model in the second deep learning framework, and capture the information of the second intermediate variable set of the deep learning model through the data capture code;

Compare the information of the first intermediate variable set with the information of the second intermediate variable set to determine the test result of the first deep learning framework.

In one possible implementation,

The first intermediate variable set includes: a first parameter set to be trained in the deep learning model trained in the first deep learning framework;

The second intermediate variable set includes: a second parameter set to be trained in the deep learning model trained in the second deep learning framework.

In one possible implementation,

The information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

The information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

In one possible implementation,

The capture module 82 is specifically configured to: capture the output value of the non-last network layer of the deep learning model trained in the first deep learning framework through the data capture code;

The determination module 83 is specifically configured to: use the data capture code to capture the output value of the non-last network layer of the deep learning model trained in the second deep learning framework.

In one possible implementation,

The capture module 82 is also configured to: capture the first output value of the last network layer of the deep learning model trained in the first deep learning framework through the data capture code; The grabbing code grabs the second output value of the last network layer of the deep learning model trained in the second deep learning framework;

The determination module 83 is specifically configured to: determine the first depth according to the information of the first intermediate variable set, the information of the second intermediate variable set, the first output value and the second output value. Test results of the learning framework.

In a possible implementation, the determining module 83 is specifically used to:

For a first intermediate variable in the first intermediate variable set and a second intermediate variable in the second intermediate variable set, determine a difference value between the first intermediate variable and the second intermediate variable;

Wherein, the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

In a possible implementation, the determining module 83 is specifically used to:

Taking the absolute values of the elements in the first intermediate variable and summing them up, the sum of the elements of the first intermediate variable is obtained;

The absolute values of the elements in the second intermediate variable are taken and then summed to obtain the sum of the elements of the second intermediate variable.

In a possible implementation, the determining module 83 is specifically used to:

The total number of elements in the first intermediate variable or the second intermediate variable is determined according to the first intermediate variable or the second intermediate variable.

In a possible implementation, the determination module 83 is specifically used to:

determining a standard difference value between the first intermediate variable and the second intermediate variable;

A first number of elements whose values are greater than a preset threshold among the standard difference values is determined.

In a possible implementation, the determining module 83 is specifically used to:

Determine the absolute value of the difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and the absolute value of the first intermediate variable;

The product is subtracted from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

In a possible implementation, the determining module 83 is also used to:

A ratio of the first quantity to the total number of elements in the first intermediate variable or the second intermediate variable is determined.

In a possible implementation, the first deep learning framework is a self-developed deep learning framework, and the second deep learning framework is a baseline deep learning framework.

In a possible implementation, the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

In a possible implementation, the first deep learning framework and the second deep learning framework are the same type of deep learning framework based on different computing platforms.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiment. For its specific implementation and technical effects, refer to the description of the above method embodiment. It’s concise and I won’t go into details here.

Embodiments of the present disclosure also provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above method is implemented. Wherein, the computer-readable storage medium may be a non-volatile computer-readable storage medium, or may be a volatile computer-readable storage medium.

An embodiment of the present disclosure also provides a computer program, which includes a computer readable code. When the computer readable code is run in an electronic device, a processor in the electronic device executes the above method.

Embodiments of the present disclosure also provide a computer program product, including computer readable code, or a non-volatile computer readable storage medium carrying the computer readable code, when the computer readable code is run in an electronic device , the processor in the electronic device executes the above method.

An embodiment of the present disclosure also provides an electronic device, including: one or more processors; a memory for storing executable instructions; wherein the one or more processors are configured to call the executable instructions stored in the memory. instructions to execute the above method.

The electronic device may be provided as a terminal, a server, or other forms of equipment.

FIG. 9 shows a block diagram of an electronic device 1900 provided by an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal. Referring to FIG. 9 , electronic device 1900 includes a processing component 1922 , which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, executable by processing component 1922 . The application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the above-described methods.

Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input/output interface 1958 (I/O interface). The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^TM ), a graphical user interface-based operating system (MacOS X ^TM ) introduced by Apple, a multi-user multi-process computer operating system ( Unix ^TM ), a free and open source Unix-like operating system (Linux ^TM ), an open source Unix-like operating system (FreeBSD ^TM ) or similar.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

The present disclosure may be a system, method, and/or computer program product. A computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

Computer-readable storage media may be tangible devices that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or Flash memory), Static Random Access Memory (SRAM), Compact Disk Read Only Memory (CD-ROM), Digital Versatile Disk (DVD), Memory Stick, Floppy Disk, Mechanical Coding Device, such as a printer with instructions stored on it. Protruding structures in hole cards or grooves, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing/processing device .

Computer program instructions for performing operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or instructions in one or more programming languages. Source code or object code written in any combination of object-oriented programming languages - such as Smalltalk, C++, etc., and conventional procedural programming languages - such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server implement. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as an Internet service provider through the Internet). connect). In some embodiments, by utilizing state information of computer-readable program instructions to personalize an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), the electronic circuit can Computer readable program instructions are executed to implement various aspects of the disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine that, when executed by the processor of the computer or other programmable data processing apparatus, , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executed on a computer, other programmable data processing apparatus, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that contains one or more components for implementing the specified logical function(s). Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions.

The computer program product can be implemented specifically through hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a Software Development Kit (SDK), etc. wait.

The above description of various embodiments tends to emphasize the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described again here.

If the technical solutions of the disclosed embodiments involve personal information, the products applying the technical solutions of the disclosed embodiments must clearly inform the personal information processing rules and obtain the individual's independent consent before processing the personal information. If the technical solutions of the embodiments of the present disclosure involve sensitive personal information, the products applying the technical solutions of the embodiments of the present disclosure must obtain the individual's separate consent before processing the sensitive personal information, and at the same time meet the requirement of "express consent". For example, setting up clear and conspicuous signs on personal information collection devices such as cameras to inform them that they have entered the scope of personal information collection, and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed to have agreed to the collection of his or her personal information; or On personal information processing devices, when using obvious logos/information to inform personal information processing rules, obtain personal authorization through pop-up messages or asking individuals to upload their personal information; among them, personal information processing rules may include personal information processing rules. Information such as information processors, purposes of processing personal information, methods of processing, and types of personal information processed.

The embodiments of the present disclosure have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims (20)

1. A method for testing a deep learning framework, comprising:

injecting data grabbing codes into a deep learning model deployed in a first deep learning framework;

training the deep learning model in the first deep learning frame, and grabbing information of a first intermediate variable set of the deep learning model through the data grabbing code;

and determining a test result of the first deep learning framework according to the information of the first intermediate variable set.

2. The method of claim 1, wherein determining the test result of the first deep learning framework based on the information of the first set of intermediate variables comprises:

Obtaining a reference value;

and comparing the information of the first intermediate variable set with the reference value to determine a test result of the first deep learning framework.

3. The method of claim 2, wherein the step of determining the position of the substrate comprises,

the obtaining the reference value includes: injecting the data grabbing code in the deep learning model deployed in a second deep learning framework; training the deep learning model in the second deep learning frame, and grabbing information of a second intermediate variable set of the deep learning model through the data grabbing code;

comparing the information of the first intermediate variable set with the reference value, and determining a test result of the first deep learning framework, wherein the test result comprises the following steps: and comparing the information of the first intermediate variable set with the information of the second intermediate variable set, and determining the test result of the first deep learning framework.

4. The method of claim 3, wherein the step of,

the first set of intermediate variables includes: a first set of parameters to be trained in the deep learning model trained in the first deep learning framework;

the second set of intermediate variables includes: a second set of parameters to be trained in the deep learning model trained in the second deep learning framework.

5. The method of claim 4, wherein the step of determining the position of the first electrode is performed,

the information of the first parameter set includes: values and/or gradients of parameters in the first parameter set;

the information of the second parameter set includes: values and/or gradients of parameters in the second parameter set.

6. A method according to any one of claims 3 to 5,

the capturing information of the first intermediate variable set of the deep learning model through the data capturing code includes: capturing output values of a non-last network layer of the deep learning model trained in the first deep learning frame through the data capturing code;

the capturing information of the second intermediate variable set of the deep learning model through the data capturing code includes: capturing output values of a non-last network layer of the deep learning model trained in the second deep learning framework through the data capture code.

7. A method according to any one of claims 3 to 6,

the method further comprises the steps of: capturing a first output value of a last network layer of the deep learning model trained in the first deep learning framework through the data capturing code; capturing a second output value of a last network layer of the deep learning model trained in the second deep learning framework through the data capturing code;

Comparing the information of the first intermediate variable set with the information of the second intermediate variable set, and determining a test result of the first deep learning framework comprises: and determining a test result of the first deep learning framework according to the information of the first intermediate variable set, the information of the second intermediate variable set, the first output value and the second output value.

8. The method of any of claims 3 to 7, wherein comparing the information of the first set of intermediate variables with the information of the second set of intermediate variables determines a test result of the first deep learning framework, comprising:

determining a difference value between a first intermediate variable in the first set of intermediate variables and a second intermediate variable in the second set of intermediate variables;

wherein the first intermediate variable and the second intermediate variable are the same intermediate variable in the deep learning model.

9. The method of claim 8, wherein comparing the information of the first set of intermediate variables with the information of the second set of intermediate variables determines a test result of the first deep learning framework, comprising:

Taking absolute values of elements in the first intermediate variable, and summing to obtain the sum of the elements of the first intermediate variable;

and taking absolute values of the elements in the second intermediate variable, and summing to obtain the sum of the elements of the second intermediate variable.

10. The method of claim 8 or 9, wherein comparing the information of the first set of intermediate variables with the information of the second set of intermediate variables determines a test result of the first deep learning framework, comprising:

and determining the total number of elements in the first intermediate variable or the second intermediate variable according to the first intermediate variable or the second intermediate variable.

11. The method of any of claims 8 to 10, wherein comparing the information of the first set of intermediate variables with the information of the second set of intermediate variables determines a test result of the first deep learning framework, comprising:

determining a standard deviation value between the first intermediate variable and the second intermediate variable;

and determining a first number of elements with values larger than a preset threshold value in the standard deviation value.

12. The method of claim 11, wherein the determining a standard deviation value between the first intermediate variable and the second intermediate variable comprises:

Determining an absolute value of a difference between the first intermediate variable and the second intermediate variable;

determining a product of a preset tolerance ratio and an absolute value of the first intermediate variable;

and subtracting the product from the absolute value of the difference to obtain a standard difference value between the first intermediate variable and the second intermediate variable.

13. The method according to claim 11 or 12, wherein after said determining the first number of elements of the standard deviation value having a value greater than a preset threshold value, the method further comprises:

a ratio of the first number to a total number of elements in the first intermediate variable or the second intermediate variable is determined.

14. The method of any one of claims 3 to 13, wherein the first deep learning frame is a self-lapping deep learning frame and the second deep learning frame is a reference deep learning frame.

15. The method of claim 14, wherein the first deep learning framework and the second deep learning framework are trained or operated based on the same type of computing platform.

16. The method of any of claims 3 to 15, wherein the first and second deep learning frameworks are the same type of deep learning framework based on different computing platforms.

17. A test device for a deep learning framework, comprising:

the data grabbing module is used for injecting data grabbing codes into a deep learning model deployed in a first deep learning frame, training the deep learning model in the first deep learning frame and grabbing information of a first intermediate variable set of the deep learning model through the data grabbing codes;

and the precision comparison module is used for determining the test result of the first deep learning framework according to the information of the first intermediate variable set.

18. A test device for a deep learning framework, comprising:

the injection module is used for injecting data grabbing codes into the deep learning model deployed in the first deep learning framework;

the grabbing module is used for training the deep learning model in the first deep learning frame and grabbing information of a first intermediate variable set of the deep learning model through the data grabbing code;

and the determining module is used for determining the test result of the first deep learning framework according to the information of the first intermediate variable set.

19. An electronic device, comprising:

One or more processors;

a memory for storing executable instructions;

wherein the one or more processors are configured to invoke the memory-stored executable instructions to perform the method of any of claims 1 to 16.

20. A computer readable storage medium having stored thereon computer program instructions, which when executed by a processor, implement the method of any of claims 1 to 16.