TW202145142A - Method and apparatus of quantization training, image processing, and storage medium - Google Patents

Abstract

The present disclosure provides a method and an apparatus of quantization training, image processing, and a storage medium. The quantization training method includes: obtaining a first quantization model by performing at least one round of iteration quantization training on a neural network model; obtaining a test result of the first quantization model by testing the first quantization model through simulating a hardware deployment environment.

Description

Quantization training, image processing method and device, storage medium

The present disclosure relates to the field of quantitative training, and in particular, to quantitative training, image processing methods and devices, and storage media.

As more and more neural network models need to be deployed on mobile devices, the efficiency of inference has become a key issue. In the case of deploying a neural network model to a mobile device, the structure of the neural network model needs to be simplified, and commonly used methods include quantization.

Quantizing the neural network model is to approximate the high-precision parameters of the neural network model with lower-precision parameters. The high-precision parameters may include floating-point parameters, and the low-precision parameters may include integer parameters. The quantized neural network model can process more data per unit time, and the storage space occupied can be further reduced, and so on.

At present, the quantization training process for neural network models is generally to specify the total number of quantization training based on experience. After the total number of times is reached, the obtained quantization model is converted into a test model corresponding to the actual hardware environment, and the Run the test model in the actual hardware environment to get the running result.

According to a first aspect of the embodiments of the present disclosure, there is provided a quantization training method, the method comprising: a model training device performs at least one round of iterative quantization training on a neural network model to obtain a first quantization model; the model training device simulates hard The physical deployment environment is used to test the first quantitative model to obtain a test result of the first quantitative model.

In some optional embodiments, the model training device tests the first quantitative model by simulating a hardware deployment environment, and obtains a test result of the first quantitative model, including: the model training device calls the The objective function tests the first quantitative model to obtain a test result of the first quantitative model, wherein the objective function is used to simulate a hardware deployment environment.

In some optional embodiments, the method further includes: performing a conversion process on the first quantization model to obtain a first test model, wherein the conversion process includes removing at least one target unit of the first quantization model , the target unit is used to perform a quantization operation and/or a dequantization operation on at least one of the output data and network parameters of the network layer of the neural network model; the model training device is deployed by simulating hardware environment, testing the first quantitative model to obtain the test result of the first quantitative model, including: the model training device tests the first test model by simulating a hardware deployment environment, and obtains the Test results of the first quantitative model.

In some optional embodiments, the conversion process is implemented by an objective function for simulating the hardware deployment environment.

In some optional embodiments, the model training device tests the first quantitative model by simulating a hardware deployment environment, and obtains a test result of the first quantitative model, including: the model training device simulates The hardware deployment environment uses fixed-point data obtained by performing fixed-point processing on test samples and network parameters of the first quantitative model to test the first quantitative model to obtain a test result of the first quantitative model.

In some optional embodiments, the method further includes: the model training device performs at least one round of iterative quantization training on the first quantization model to obtain a second quantization model; the model training device simulates a hardware deployment environment by , and test the second quantization model to obtain a test result of the second quantization model.

In some optional embodiments, the method further includes: obtaining, based at least in part on a test result of the first quantization model, a training strategy analysis result of performing quantitative training on the neural network model, wherein the training The policy analysis result includes at least one of the following: terminating the quantization training of the neural network model, adjusting the quantization method of at least one network layer in the neural network model, adjusting the Quantitative training method.

In some optional embodiments, the model training device performs at least one round of iterative quantization training on the first quantization model to obtain a second quantization model, including: the model training device is performing all the steps on the first quantization model. During the testing process, at least one round of iterative quantization training is performed on the first quantization model in parallel to obtain the second quantization model.

In some optional embodiments, the testing of the first quantitative model to obtain a test result of the first quantitative model includes any one of the following: in response to performing at least one round of iteration on the neural network model The number of times of quantization training reaches a preset number of times, and the obtained first quantization model is tested to obtain a test result of the first quantization model; or in response to determining that the first quantization model satisfies the test condition based on a preset test strategy , test the first quantization model to obtain a test result of the first quantization model.

According to a second aspect of the embodiments of the present disclosure, there is provided an image processing method, comprising: inputting an image to be processed into a quantization model, and obtaining an image processing result output by the quantization model; wherein, the quantization model is obtained by the above-mentioned No. A quantitative model obtained by the method described in one aspect.

According to a third aspect of the embodiments of the present disclosure, there is provided a quantization training device, the device comprising: a first quantization training module, used by the model training device to perform at least one round of iterative quantization training on a neural network model to obtain a first quantization model a first test module for the model training device to test the first quantized model by simulating a hardware deployment environment to obtain a test result of the first quantized model.

In some optional embodiments, the first test module includes: a first test sub-module for the model training device to test the first quantitative model by calling an objective function to obtain the first Quantify the test results of the model, wherein the objective function is used to simulate a hardware deployment environment.

In some optional embodiments, the apparatus further includes: a model conversion module, configured to perform conversion processing on the first quantization model to obtain a first test model, wherein the conversion processing includes removing the first test model. at least one target unit of the quantization model, the target unit is used to perform a quantization operation and/or a dequantization operation on at least one of the output data and network parameters of the network layer of the neural network model; the first A test module includes: a second test sub-module for the model training device to test the first test model by simulating a hardware deployment environment to obtain a test result of the first quantitative model.

In some optional embodiments, the conversion process is implemented by an objective function for simulating the hardware deployment environment.

In some optional embodiments, the first test module includes: a third test sub-module for the model training device to simulate a hardware deployment environment and use the test samples and The fixed-point data obtained by performing fixed-point processing on network parameters is used to test the first quantization model to obtain a test result of the first quantization model.

In some optional embodiments, the apparatus further includes: a second quantization training module, used for the model training device to perform at least one round of iterative quantization training on the first quantization model to obtain a second quantization model; a second quantization model; A test module is used for the model training device to test the second quantization model by simulating a hardware deployment environment to obtain a test result of the second quantization model.

In some optional embodiments, the apparatus further includes: a determination module configured to obtain a training strategy analysis result for quantitative training of the neural network model based at least in part on the test result of the first quantitative model , wherein the training strategy analysis result includes at least one of the following: terminating the quantization training of the neural network model, adjusting the quantization method of at least one network layer in the neural network model, adjusting the neural network model Quantized training method for subsequent iterations of the road model.

In some optional embodiments, the second quantization training module includes: a quantization training sub-module for the model training device to perform the test on the first quantization model in parallel The first quantization model performs at least one round of iterative quantization training to obtain the second quantization model.

In some optional embodiments, the first test module includes any one of the following: a fourth test sub-module, configured to respond to at least one round of iterative quantization training on the neural network model reaching a preset number of times number of times, the obtained first quantitative model is tested to obtain the test result of the first quantitative model; or a fifth test sub-module is used for determining that the first quantitative model meets the requirements based on the preset test strategy. Test conditions, test the first quantitative model to obtain a test result of the first quantitative model.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, the apparatus comprising: an image processing module, configured to input an image to be processed into a quantization model, and obtain an image processing result output by the quantization model ; wherein, the quantization model is a quantization model obtained by the method described in the first aspect.

According to a fifth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, the storage medium stores a computer program, and the computer program is used to execute the quantization training method described in the above-mentioned first aspect or the above-mentioned second aspect. The described image processing method.

According to a sixth aspect of the embodiments of the present disclosure, there is provided a quantization training apparatus, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the memory in the memory The stored executable instructions implement the quantization training method described in the first aspect.

According to a seventh aspect of the embodiments of the present disclosure, there is provided an image processing apparatus, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call the memory The executable instructions stored in the image processing method implement the image processing method described in the second aspect.

The technical solutions provided by the embodiments of the present disclosure may include the following beneficial effects:

In the embodiment of the present disclosure, at least one round of iterative quantization training can be performed on the neural network model by the model training device to obtain the first quantized model. test, so as to obtain the test result of the first quantitative model. In the present disclosure, after at least one round of iterative quantization training is performed by the model training device, the hardware deployment environment can be simulated, and the first quantization model can be directly tested under the training framework without deploying the first quantization model on the actual hardware device. Obtaining the test result of the first quantitative model is beneficial to improve the development efficiency of the neural network model.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure.

Exemplary embodiments will be described in detail herein, examples of which are illustrated in the accompanying drawings. Where the following description refers to the drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the illustrative examples below are not intended to represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatus and methods consistent with some aspects of the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a," "the," and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It will also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It will be understood that although the terms first, second, third, etc. may be used in this disclosure to describe various information, these information should not be limited by these terms. These terms are only used to distinguish information of the same type from one another. For example, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information, without departing from the scope of the present disclosure. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining."

At present, the neural network model needs to be deployed on the actual mobile device for testing after completing the quantitative training. Specifically, before the test, the neural network model can be iteratively quantized training, after obtaining the quantized model, the quantized model can be converted into a test model corresponding to the actual hardware environment, and then the test model can be deployed to the actual hardware environment. Test on mobile device and get test results. Based on the test results, it is determined whether the performance of the quantitative model is as expected.

Quantization schemes can generally be divided into post-training quantization and Quantization-aware training. Post-training quantization refers to directly quantizing the parameters of the neural network model after training the neural network model with floating-point parameters. This scheme has better quantization effect for models with large parameters, and the performance loss is small. However, models with small parameters will lead to significant performance degradation. Quantization-aware training simulates the quantization behavior in the training of the neural network model. In the training, the fixed-point parameters are stored in the form of floating-point parameters. Finally, the fixed-point parameters are directly used for operation during model inference.

Let's take Quantization-aware training as an example to introduce the process of quantization training: first, all tensor operations are implemented through modules, and then all multiplexing modules are changed to separate, that is, modules are not allowed. Group reuse, and change all functional interfaces in training frameworks (such as pytorch) to modules that implement the same function.

However, it is necessary to first obtain the quantized model through the complete training process, and then convert the trained quantized model into a deployable model and place it in the deployment environment to test its performance in the deployment environment. This solution is used for model training or parameter adjustment. It is very inconvenient, and it is necessary to ensure that the intermediate conversion link cannot be wrong. In addition, only after the model is fully trained can the results of the model in the real environment be known, and the model training process cannot be corrected as soon as possible.

In addition, the disadvantage of using training frameworks such as pytorch is that the quantized model including floating-point parameters is used for testing, while the quantized model including fixed-point parameters is used for inference in the deployment environment, so the test results and the inference results on the deployed hardware will be different. There are certain differences, and the test results cannot truly reflect the performance of the quantitative model in the actual hardware deployment environment. In addition, the quantitative model is prone to overfitting during the training process, and once the overfitting occurs, it is not easy to adjust the parameters in the quantitative model.

The embodiments of the present disclosure provide a quantization training scheme. For example, as shown in FIG. 1 , FIG. 1 shows a quantization training method according to an exemplary embodiment, and the quantization training method may be applied to a model training device. In the embodiment of the present disclosure, both the quantization training and the testing of the quantization model can be performed on the same model training device. The model training device may be an electronic device deployed with a training platform framework including a quantization algorithm or a quantization tool; the electronic device may include, but is not limited to, a terminal device using the x86 architecture, such as a personal computer (PC), mobile phone, Portable devices, etc. The quantitative training method includes the following steps:

In step 101, the model training device performs at least one round of iterative quantization training on the neural network model to obtain a first quantized model.

The model training device can use quantization training methods, including but not limited to quantization-aware training, to quantify the neural network model during the training process. In the embodiment of the present disclosure, the model training device may perform at least one round of iterative quantization training on the neural network model, thereby obtaining the first quantization model. Specifically, the model training device may perform limited iteration training, such as one or several times, to obtain the first quantized model.

In step 102, the model training device tests the first quantitative model by simulating a hardware deployment environment, and obtains a test result of the first quantitative model.

In the embodiment of the present disclosure, an actual hardware deployment environment can be simulated on the model training device, so that the first quantitative model can be tested directly on the model training device without converting the first quantitative model to the actual hardware deployment environment carry out testing. According to the test result of the first quantization model, a training strategy analysis of the quantization training may be performed, for example, at least one round of iterative quantization training may be performed on the first quantization model again.

In some embodiments, simulating the hardware deployment environment refers to computing logic based on the hardware deployment environment, encapsulating at least one interface in a modular manner, and during testing, the at least one interface and the first quantitative model can be called. The interface corresponding to the network structure runs the first quantitative model to obtain the running result, and the running result is used as the test result of the first quantitative model. Among them, at least one interface is respectively used to realize the functions of different network layers, and the network layers include but are not limited to convolution layer (conv), pooling layer (pooling), linear layer (linear), activation function layer (prelu), etc. .

For example, if the first quantitative model needs to run on a graphics processor (Graphics Processing Unit, GPU) of the mobile device, the hardware deployment environment is the hardware environment of the GPU. The functions of different network layers can be realized through at least one interface based on the computing logic of the GPU, and the model training device can call the interface corresponding to the network structure of the first quantization model in the at least one interface to run the first quantization model, thereby A measurement result of the first quantitative model is obtained.

In the above embodiment, after at least one round of iterative quantization training is performed by the model training device, the hardware deployment environment can be simulated, and the first quantization model can be tested directly under the training framework, without deploying the first quantization model to the actual hardware device. The test result of the first quantitative model can be obtained on the above, which is beneficial to improve the development efficiency of the neural network model.

In some optional embodiments, for the above step 102, the model training device can set at least one interface encapsulated in a modular way based on the computing logic of the hardware deployment environment in the objective function, that is, the objective function can use for simulating a hardware deployment environment. After the first quantization model is obtained, the interface corresponding to the network structure of the first quantization model in the objective function can be called by means of a function call, thereby obtaining the test result of the first quantization model.

In a possible implementation manner, the quantization training framework of the model training platform adopts the pytorch framework, and accordingly, the objective function may be the model.quant() function in the quantization training framework.

In the above embodiment, the model training platform can call the target module to test the first quantitative model, and obtain the test result of the first quantitative model. By simulating the hardware deployment environment on the same hardware device, not only can iterative quantitative training be performed, but also the test results of the first quantitative model can be directly obtained, which is beneficial to improve the development efficiency of the neural network model.

In some optional embodiments, during the iterative quantization training process, a pseudo-quantization layer is added to the output of each network layer of the neural network model, and the pseudo-quantization layer can simulate the effects brought by the quantization of the network layer. Loss of precision.

For example, the original output data of network layer 1 has the first precision, assuming that the first precision is FP32, and the value is 1.1, the pseudo-quantization layer quantizes the original output data to obtain the second precision value, assuming that the second precision is uint8, the numerical value is 1, but the network layer 2 that takes the output of network layer 1 as input also needs to use the value of the first precision, so it is also necessary to dequantize the value obtained by the quantization of network layer 1 through the pseudo-quantization layer, and the second The precision value 1 is converted to the first precision, and the value remains unchanged. According to the above process, the precision loss of the network layer 1 can be determined as 1.1-1=0.1 through the pseudo quantization layer. In the same way, the accuracy loss of other network layers can be determined.

In some optional embodiments, such as shown in FIG. 2 , before step 102 is performed, the method may further include step 103 .

In step 103, a conversion process is performed on the first quantization model to obtain a first test model.

In the embodiment of the present disclosure, in the process of testing the first quantization model, the pseudo quantization layer can be removed, and the same model structure as in the actual deployment environment can be used for testing, so that the neural network model obtained by quantization training can be directly simulated Run on the actual deployment hardware to get more realistic test results. Therefore, in the embodiment of the present disclosure, it is necessary to perform conversion processing on the first quantization model, and the conversion processing includes removing at least one pseudo-quantization layer of the first quantization model. In the embodiment of the present disclosure, the pseudo-quantization layer can pass through the module It is realized in a quantized way, that is, each pseudo-quantization layer can correspond to a target unit, and accordingly, the conversion process includes removing at least one target unit of the first quantization model, and the target unit is used for the neural network model. At least one of the output data of the network layer and the network parameters is subjected to a quantization operation and/or a dequantization operation.

Accordingly, step 102 may include:

The model training device tests the first test model by simulating a hardware deployment environment to obtain a test result of the first quantitative model.

For example, the original network structure of the neural network model is shown in FIG. 3A , and the first quantization model including at least one target unit is shown in FIG. 3B , in which the target unit 1 quantifies the network parameters of the convolution layer, that is, the weight value. operation and/or dequantization operation, the target unit 2 performs a quantization operation and/or a dequantization operation on the output data of the activation function layer. In the conversion process, the target unit needs to be removed from the first quantization model, and the first test model is obtained as shown in FIG. 3C .

In the present disclosure, the first test model is tested by the model training device, so as to obtain the test result of the first quantitative model.

In the above embodiment, the first quantization model needs to perform a quantization operation and/or a dequantization operation on at least one of the output data and network parameters of the network layer of the neural network model. However, during the testing process, It is no longer necessary to perform quantization operation and/or dequantization operation. Therefore, the first quantization model can be converted to remove at least one target unit in the first quantization model to obtain a first test model, and test the first test model. , so that the required test result of the first quantization model can be obtained, and the purpose of testing the first quantization model on the model training device is achieved.

In some optional embodiments, the process of converting the first quantization model by the model training device is implemented by using an objective function for simulating the hardware deployment environment.

After obtaining the first quantized model, the model training device calls the objective function, thereby converting the first quantized model to remove at least one target unit, so as to test the obtained first test model and obtain the test result of the first quantized model .

In some optional embodiments, when testing the first quantization model, the test samples and network parameters of the first quantization model may be fixed-pointed to obtain fixed-point data, and then the The model is tested and the test results are obtained.

Fixed-point processing refers to converting data from a first precision to a second precision, where the first precision is higher than the second precision, for example, the first precision is floating point precision FP32, and the second precision is integer precision uint8.

The model training device needs to perform fixed-point processing on the test samples and network parameters input by the entire first quantization model, such as weight values, and then obtain fixed-point data, and test based on the fixed-point data.

For example, in Fig. 3B, the accuracy of the test samples (input values) and network parameters (weight values corresponding to the input values) of the first quantization model are both FP32, and the accuracy of the fixed-point data after fixed-point processing is uint8. In Fig. 3C The first test model may be tested based on the fixed-point data of uint8 to obtain a test result of the first quantitative model.

In the above embodiment, the model training device needs to use the fixed-point data obtained by performing fixed-point processing on the test samples and network parameters of the first quantitative model to test the first quantitative model, and obtain the test of the first quantitative model. As a result, the testing process is more reasonable and accurate.

In some optional embodiments, such as shown in FIG. 4 , the above method may further include steps 104 and 105 .

In step 104, the model training device performs at least one round of iterative quantization training on the first quantization model to obtain a second quantization model.

In the embodiment of the present disclosure, after obtaining the first quantization model, the model training device may continue to perform at least one round of iterative quantization training on the first quantization model, thereby obtaining the second quantization model.

In a possible implementation manner, the model training device may, when the test result of the first quantitative model indicates that the first quantitative model does not meet the quantitative training requirements, for example, when the accuracy of processing the test samples does not meet the design requirements of the neural network model , and continue to perform at least one round of iterative quantization training on the first quantization model to obtain the second quantization model.

In another possible implementation manner, the model training device may also perform at least one round of iterative quantization training on the first quantization model while testing the first quantization model to obtain the second quantization model. If the test result of the first quantization model indicates that the first quantization model does not meet the quantization training requirements, it means that at least one round of iterative quantization training for the first quantization model has been started in advance, and the usability is higher.

In step 105, the model training device tests the second quantization model by simulating a hardware deployment environment to obtain a test result of the second quantization model.

By simulating the hardware deployment environment, the model training device can test the second quantitative model, and obtain the test result of the second quantitative model. According to the test result of the second quantization model, a training strategy analysis of the quantization training may be performed, for example, at least one round of iterative quantization training may be performed on the second quantization model again.

In some optional embodiments, such as shown in FIG. 5 , after obtaining the test result of the first quantitative model in step 102 , the above method may further include step 106 .

In step 106, based at least in part on the test results of the first quantization model, a training strategy analysis result for quantitative training of the neural network model is obtained.

Specifically, based on the test results of at least one test, it can be determined whether the current quantitative training scheme of the neural network model is feasible, such as whether the design of the loss function is reasonable, whether the design of the quantitative scheme is reasonable, etc., or the adjustment scheme of the current quantitative training scheme can be determined. , for example, whether to modify one or more of the network structure, network hyperparameters, loss function, quantization scheme, etc., or to further give adjustment strategies or other detailed information. In some optional examples, it can also be determined whether the current quantization iterative training needs to be stopped, for example, it is determined that early stopping is required, or it is determined that the neural network model has reached the expectation, etc., which is not limited in this embodiment of the present disclosure.

In this embodiment of the present disclosure, the training strategy analysis result includes at least one of the following: terminating the quantization training of the neural network model, adjusting the quantization method of at least one network layer in the neural network model, adjusting the Quantized training method for subsequent iterations of the neural network model.

In the embodiment of the present disclosure, training and testing are performed on the same platform, that is, on the same model training device, without model conversion and deployment, thereby optimizing the development process of the neural network model, and in the training process In the test, it can be determined whether it is necessary to continue the quantitative iterative training of the neural network model according to the test result of the first quantitative model and the evaluation index can be more conveniently combined. The evaluation index includes, but is not limited to, an evaluation index such as early stop.

For example, the test result of the first quantization model indicates that the first quantization model does not meet the quantization training requirement, so the quantization training of the neural network model cannot be terminated. For another example, if the test result of the first quantization model indicates that the first quantization model is overfitting, the quantization training of the neural network model may be terminated in advance according to the evaluation index.

According to the test result of the first quantization model, the model training device can also adjust the quantization method of at least one network layer in the neural network model. The quantization method can be adjusted to Quantization-aware training.

According to the test result of the first quantization model, the model training device can also adjust the quantization training method of the subsequent iterations of the neural network model. The quantization training method includes but is not limited to the number of iterative quantization training, the adjustment of the loss function, and the like. For example, N rounds of iterative quantization training are performed on the neural network model before, to obtain a first quantization model, and M rounds of iterative quantization training can be performed on the first quantization model, where M and N are both positive integers and may be equal or unequal.

In the above embodiment, according to the test result of the first quantitative model, the analysis result of the training strategy for the quantitative training of the neural network model can be obtained, and the quantitative training can be adjusted based on the analysis result, so that the quantitative training of the neural network model can be adjusted. more reasonable.

In some optional embodiments, the model training device may stop iterative quantization training based on different training timings, thereby obtaining the first quantized model.

In an optional implementation manner, the model training device may test the obtained first quantization model when the number of times of performing at least one round of iterative quantization training on the neural network model reaches a preset number of times, and obtain the first quantization model. 's test results. The preset number of times can be much smaller than the total number of times of iterative quantization training for the neural network model set previously, for example, the total number of times is 1000 times, and the preset number of times can be any positive integer less than 1000, based on the combination of the test results of the first quantization model Evaluation indicators such as early stopping can terminate the iterative quantitative training of the neural network model in advance, thereby avoiding the problem of overfitting in the final quantitative model.

In another possible implementation manner, the model training device may obtain the first quantitative model when it is determined based on a preset test strategy that the first quantitative model satisfies the test condition, so as to test the first quantitative model. The test conditions include, but are not limited to, the change of the loss function corresponding to the neural network model is small, or the accuracy of the neural network model is far less than the preset accuracy requirement, or the preset total number of iterative quantizations has been reached, etc.

In the above embodiment, the model training device can test the obtained first quantized model based on the training timing, so as to obtain the test result of the first quantized model, which is beneficial to improve the development efficiency of the neural network model.

In some optional embodiments, the present disclosure also provides an image processing method, which can input an image to be processed into a quantization model, so as to obtain an image processing result output by the quantization model.

The quantization model is a quantization model obtained by performing at least one round of iterative quantization training on the neural network model through the quantization training method described above.

The image to be processed can be an image collected in a visual task, and the collected image can be analyzed by visual task analysis, wherein the visual task analysis includes but is not limited to image classification, image semantic segmentation, Human key point detection, etc., with high availability.

Corresponding to the foregoing method embodiments, the present disclosure also provides device embodiments.

As shown in FIG. 6, FIG. 6 is a block diagram of a quantization training apparatus according to an exemplary embodiment of the present disclosure. The apparatus includes: a first quantization training module 210, which is used for the model training device to perform at least one round of the neural network model. Iterative quantitative training to obtain a first quantitative model; a first test module 220 for the model training device to test the first quantitative model by simulating a hardware deployment environment to obtain a test of the first quantitative model result.

In some optional embodiments, the first test module includes: a first test sub-module for the model training device to test the first quantitative model by calling an objective function to obtain the first Quantify the test results of the model, wherein the objective function is used to simulate a hardware deployment environment.

In some optional embodiments, the apparatus further includes: a model conversion module, configured to perform conversion processing on the first quantization model to obtain a first test model, wherein the conversion processing includes removing the first test model. at least one target unit of the quantization model, the target unit is used to perform a quantization operation and/or a dequantization operation on at least one of the output data and network parameters of the network layer of the neural network model; the first A test module includes: a second test sub-module for the model training device to test the first test model by simulating a hardware deployment environment to obtain a test result of the first quantitative model.

In some optional embodiments, the conversion process is implemented by an objective function for simulating the hardware deployment environment.

In some optional embodiments, the first test module includes: a third test sub-module for the model training device to simulate a hardware deployment environment and use the test samples and The fixed-point data obtained by performing fixed-point processing on network parameters is used to test the first quantization model to obtain a test result of the first quantization model.

In some optional embodiments, the apparatus further includes: a second quantization training module, used for the model training device to perform at least one round of iterative quantization training on the first quantization model to obtain a second quantization model; a second quantization model; A test module is used for the model training device to test the second quantization model by simulating a hardware deployment environment to obtain a test result of the second quantization model.

In some optional embodiments, the apparatus further includes: a determination module configured to obtain a training strategy analysis result for quantitative training of the neural network model based at least in part on the test result of the first quantitative model , wherein the training strategy analysis result includes at least one of the following: terminating the quantization training of the neural network model, adjusting the quantization method of at least one network layer in the neural network model, adjusting the neural network model Quantized training method for subsequent iterations of the road model.

In some optional embodiments, the second quantization training module includes: a quantization training sub-module for the model training device to perform the test on the first quantization model in parallel The first quantization model performs at least one round of iterative quantization training to obtain the second quantization model.

In some optional embodiments, the first test module includes any one of the following: a fourth test sub-module, configured to respond to at least one round of iterative quantization training on the neural network model reaching a preset number of times number of times, the obtained first quantitative model is tested to obtain the test result of the first quantitative model; or a fifth test sub-module is used for determining that the first quantitative model meets the requirements based on the preset test strategy. Test conditions, test the first quantitative model to obtain a test result of the first quantitative model.

The present disclosure also provides an image processing device, the device comprising: an image processing module for inputting an image to be processed into a quantization model to obtain an image processing result output by the quantization model; wherein, the quantization model The model is a quantized model obtained by the quantization training method described above.

For the apparatus embodiments, since they basically correspond to the method embodiments, reference may be made to the partial descriptions of the method embodiments for related parts. The device embodiments described above are only illustrative, wherein the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place , or can be distributed over multiple network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of the present disclosure. Those of ordinary skill in the art can understand and implement it without creative effort.

Embodiments of the present disclosure also provide a computer-readable storage medium, where a computer program is stored in the storage medium, and the computer program is used to execute the above-described quantization training method or image processing method.

In some optional embodiments, embodiments of the present disclosure provide a computer program product, comprising computer readable code, when the computer readable code is executed on a device, the processor in the device executes to implement any of the above embodiments Provides instructions for quantizing training methods or image processing methods.

The computer program product can be embodied in hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

An embodiment of the present disclosure further provides a quantization training apparatus, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to achieve the above The quantized training method is described.

FIG. 7 is a schematic diagram of a hardware structure of a quantization training apparatus according to an embodiment of the present disclosure. The quantization training device 310 of the neural network model includes a processor 311 , and may further include an input device 312 , an output device 313 and a memory 314 . The input device 312 , the output device 313 , the memory 314 and the processor 311 are connected to each other through bus bars.

Memory includes but is not limited to random access memory (RAM), read-only memory (ROM), erasable programmable read only memory (EPROM) ), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

Input means are used for inputting data and/or signals, and output means are used for outputting data and/or signals. The output device and the input device can be independent devices or an integral device.

The processor may include one or more processors, for example, including one or more central processing units (central processing units, CPUs). In the case where the processor is a CPU, the CPU may be a single-core CPU or a Multi-core CPU.

Memory is used to store program code and data for network devices.

The processor is used for calling the program code and data in the memory to execute the steps in the above method embodiments. For details, refer to the description in the method embodiment, which is not repeated here.

It can be understood that FIG. 7 only shows a simplified design of a quantization training apparatus. In practical applications, the quantization training device may also include other necessary elements, including but not limited to any number of input/output devices, processors, controllers, memories, etc., all of which can realize the quantization of the embodiments of the present disclosure. Training devices are within the scope of this disclosure.

An embodiment of the present disclosure further provides an image processing apparatus, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the executable instructions stored in the memory to achieve The image processing method described above.

Other embodiments of the present disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common general knowledge or techniques in the technical field not disclosed by this disclosure . The specification and examples are to be regarded as exemplary only, with the true scope and spirit of the disclosure being indicated by the following claims.

The above descriptions are only preferred embodiments of the present disclosure, and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the present disclosure. within the scope of protection.

101: Perform at least one round of iterative quantization training on the neural network model to obtain a first quantization model 102: by simulating a hardware deployment environment, test the first quantitative model to obtain a test result of the first quantitative model 103: converting the first quantization model to obtain the first test model 104: perform at least one round of iterative quantization training on the first quantization model to obtain a second quantization model 105: by simulating a hardware deployment environment, test the second quantization model to obtain a test result of the second quantization model 106: based on the test result of the first quantization model, obtain the training strategy analysis result of the quantization training on the neural network model 210: The first quantitative training module 220: Brother One Test Module 310: Quantization training device 311: Processor 312: Input device 313: Output device 314: Memory

FIG. 1 is a flowchart of a quantization training method according to an exemplary embodiment of the present disclosure. FIG. 2 is a flowchart of another quantization training method according to an exemplary embodiment of the present disclosure. FIG. 3A is a schematic structural diagram of a neural network model according to an exemplary embodiment of the present disclosure. FIG. 3B is a schematic structural diagram of a first quantization model according to an exemplary embodiment of the present disclosure. FIG. 3C is a schematic structural diagram of a first test model according to an exemplary embodiment of the present disclosure. FIG. 4 is a flowchart of another quantization training method according to an exemplary embodiment of the present disclosure. FIG. 5 is a flowchart of another quantization training method according to an exemplary embodiment of the present disclosure. FIG. 6 is a block diagram of a quantization training apparatus according to an exemplary embodiment of the present disclosure. FIG. 7 is a schematic structural diagram of a quantization training apparatus according to an exemplary embodiment of the present disclosure.

101: Perform at least one round of iterative quantization training on the neural network model to obtain a first quantization model

102: by simulating a hardware deployment environment, test the first quantitative model to obtain a test result of the first quantitative model

Claims (15)

A quantitative training method, including: Perform at least one round of iterative quantization training on the neural network model to obtain a first quantization model; By simulating a hardware deployment environment, the first quantitative model is tested to obtain a test result of the first quantitative model. The quantization training method according to claim 1, wherein the first quantization model is tested by simulating a hardware deployment environment to obtain a test result of the first quantization model, comprising: A test result of the first quantitative model is obtained by calling an objective function to test the first quantitative model, wherein the objective function is used to simulate a hardware deployment environment. The quantization training method according to claim 1 or 2, further comprising: A conversion process is performed on the first quantization model to obtain a first test model, wherein the conversion process includes removing at least one target unit of the first quantization model, and the target unit is used for the neural network model. at least one of the output data and network parameters of the network layer is subjected to a quantization operation and/or a dequantization operation; By simulating a hardware deployment environment, the first quantitative model is tested to obtain a test result of the first quantitative model, including: By simulating a hardware deployment environment, the first test model is tested to obtain a test result of the first quantitative model. The quantization training method according to claim 3, wherein the conversion process is realized by an objective function for simulating the hardware deployment environment. The quantitative training method according to claim 1 or 2, wherein the first quantitative model is tested by simulating a hardware deployment environment to obtain a test result of the first quantitative model, including: By simulating the hardware deployment environment, using the fixed-point data obtained by performing fixed-point processing on the test samples and network parameters of the first quantitative model, the first quantitative model is tested to obtain the test of the first quantitative model. result. The quantization training method according to claim 1 or 2, further comprising: Perform at least one round of iterative quantization training on the first quantization model to obtain a second quantization model; By simulating a hardware deployment environment, the second quantization model is tested to obtain a test result of the second quantization model. The quantization training method according to claim 1 or 2, further comprising: Based on the test result of the first quantitative model, a training strategy analysis result of performing quantitative training on the neural network model is obtained, wherein the training strategy analysis result includes at least one of the following: terminating the neural network Quantization training of the model, adjusting the quantization method of at least one network layer in the neural network model, and adjusting the quantization training method of subsequent iterations of the neural network model. The quantization training method according to claim 6, wherein performing at least one round of iterative quantization training on the first quantization model to obtain a second quantization model, comprising: In the process of performing the test on the first quantization model, at least one round of iterative quantization training is performed on the first quantization model in parallel to obtain the second quantization model. The quantization training method according to claim 1 or 2, wherein the test is performed on the first quantization model to obtain a test result of the first quantization model, including any one of the following: In response to performing at least one round of iterative quantization training on the neural network model reaching a preset number of times, testing the obtained first quantization model to obtain a test result of the first quantization model; or In response to determining that the first quantitative model satisfies the test condition based on the preset test strategy, the first quantitative model is tested to obtain a test result of the first quantitative model. An image processing method, comprising: Input the image to be processed into the first quantization model, and obtain the image processing result output by the first quantization model; wherein, the first quantization model is obtained by the quantization training method described in any one of claims 1-9 quantification model. A quantitative training device, comprising: a first quantization training module, used to perform at least one round of iterative quantization training on the neural network model to obtain a first quantization model; The first test module is used to test the first quantitative model by simulating a hardware deployment environment to obtain a test result of the first quantitative model. An image processing device, comprising: The image processing module is used to input the image to be processed into the quantization model, and obtain the image processing result output by the quantization model; wherein, the quantization model is trained through the quantization of any one of claims 1-9 The quantized model trained by the method. A computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the quantization training method described in any one of the above claims 1-9 or the image processing described in the above claim 10 method. A quantitative training device, comprising: processor; memory for storing said processor-executable instructions; Wherein, the processor is configured to call executable instructions stored in the memory to implement the quantization training method according to any one of claims 1-9. An image processing device, comprising: processor; memory for storing said processor-executable instructions; Wherein, the processor is configured to invoke the executable instructions stored in the memory to implement the image processing method of claim 10 .

Images