CN114492783A - Pruning method and device for multitask neural network model - Google Patents

Abstract

The embodiment of the invention discloses a pruning method and a pruning device of a multitask neural network model, wherein the method comprises the following steps: obtaining a model to be pruned; executing the steps A-C aiming at each convolution layer of the model to be pruned, and determining the sensitivity information of each convolution layer; a, aiming at the pruning rate of each preset layer, pre-pruning the convolutional layer based on the pruning rate of the preset layer and each convolutional kernel in the convolutional layer; b, determining loss values of tasks corresponding to the pre-pruned model to be pruned corresponding to the pruning rate of the preset layer based on the first training data; c, determining sensitivity information of the convolutional layer under the pruning rate of the preset layer based on the loss value of each task; grouping convolution kernels of the model to be pruned based on channel information corresponding to the model to be pruned; and determining a target pruning model based on the current model pruning rate, the grouping condition of convolution kernels, the pruning norm value corresponding to each convolution kernel and the sensitivity information of each convolution layer so as to realize the pruning of the multitask neural network model.

Description

A method and device for pruning a multi-task neural network model

technical field

The present invention relates to the field of computer technology, and in particular, to a method and device for pruning a multi-task neural network model.

Background technique

The development of deep learning has made deep neural networks more and more applied to computer vision tasks such as image recognition, detection and tracking, and network models tend to be designed in a wider and deeper direction. The success of deep learning largely depends on the large number of parameters of the model and the computing equipment with powerful performance. However, due to the huge memory requirements and computational consumption of deep neural networks, it is difficult to deploy them on low-storage, low-power hardware platforms (such as mobile devices), which greatly limits their applications. Therefore, it is an important problem to study how to effectively compress the neural network model while keeping the performance of the existing deep neural network model unchanged.

Then, how to provide a pruning method for multi-task neural network models has become an urgent problem to be solved.

SUMMARY OF THE INVENTION

The invention provides a method and device for pruning a multi-task neural network model, so as to realize the pruning of the multi-task neural network model. The specific technical solutions are as follows:

In a first aspect, an embodiment of the present invention provides a method for pruning a multi-task neural network model, the method comprising:

Obtain the trained multi-task neural network model as the model to be pruned;

For each convolutional layer of the model to be pruned, the following steps A-C are performed to determine the sensitivity information of each convolutional layer under different preset layer pruning rates;

Step A: for each preset layer pruning rate, based on the preset layer pruning rate and each convolution kernel in the convolution layer, pre-prune the convolution layer;

Step B: Based on the first training data, determine the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate;

Step C: Determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned;

grouping the convolution kernels in the convolution layer of the to-be-pruned model based on the channel information corresponding to the to-be-pruned model;

Based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers The sensitivity information under the rate is used to determine the target pruning model.

Optionally, the pruning rate based on the current model, the grouping situation of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolution layer, and each convolution kernel. The sensitivity information of layers under different layer pruning rates, and the steps to determine the target pruning model, including:

Based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers Sensitivity information at the rate, prune the to-be-pruned model to obtain the current pruning model;

determining, based on the second training data, detection information corresponding to each second training data corresponding to the current pruning model;

If it is determined that the current pruning model meets the preset pruning condition based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, the current pruning model The pruning model is determined as the target pruning model.

Optionally, the method further includes:

If, based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, it is determined that the current pruning model does not meet the preset pruning conditions, and the current pruning model is adjusted. The value of the model pruning rate to get the new current model pruning rate;

Based on the new current model pruning rate, the grouping of convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers According to the sensitivity information at the rate, prune the model to be pruned to obtain a new current pruning model;

Based on the third training data, determining the detection information corresponding to each third training data corresponding to the new current pruning model;

If it is determined that the new current pruning model meets the preset pruning condition based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, determining the new current pruning model as the target pruning model;

If based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions , and return to the step of adjusting the value of the current model pruning rate to obtain a new current model pruning rate.

Optionally, the step of determining the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned includes:

Based on the loss value of each task corresponding to the pre-pruned model to be pruned and the preset objective function, determine the loss value corresponding to the pre-pruned model to be pruned;

Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The task weights corresponding to each task; L represents the number of tasks of the model to be pruned.

Optionally, the step of pre-pruning the convolution layer based on the preset layer pruning rate and each convolution kernel in the convolution layer includes:

For each convolution kernel of the convolution layer, based on each weight parameter in the convolution kernel, determine the pruning norm value corresponding to the convolution kernel;

Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to preset pruning rules, wherein the preset pruning The rule is: Pre-pruning is performed preferentially on the corresponding convolution kernel with a small pruning norm value.

In a second aspect, an embodiment of the present invention provides a device for pruning a multi-task neural network model, the device comprising:

The obtaining module is configured to obtain the trained multi-task neural network model as the model to be pruned;

The first determining module is configured to, for each convolutional layer of the model to be pruned, determine the sensitivity information of each convolutional layer under different preset layer pruning rates through the following units;

a pre-pruning unit, configured to perform pre-pruning on the convolution layer based on the preset layer pruning rate and each convolution kernel in the convolution layer for each preset layer pruning rate;

The first determining unit is configured to determine, based on the first training data, the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate;

The second determining unit is configured to determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned;

a grouping module, configured to group the convolution kernels in the convolutional layer of the model to be pruned based on the channel information corresponding to the model to be pruned;

The second determination module is configured to be based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each The sensitivity information of the convolutional layer at different layer pruning rates determines the target pruning model.

Optionally, the second determining module includes:

The first pruning unit is configured to be based on the current model pruning rate, the grouping situation of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each convolution kernel. Sensitivity information of a convolutional layer under different layer pruning rates, prune the to-be-pruned model to obtain the current pruning model;

a third determining unit, configured to determine, based on the second training data, detection information corresponding to each second training data corresponding to the current pruning model;

a fourth determination unit, configured to determine that the current pruning model reaches a preset level based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data The pruning condition is to determine the current pruning model as the target pruning model.

Optionally, the second determining module further includes:

An adjustment unit, configured to determine that the current pruning model does not reach the preset pruning value based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data branch condition, adjust the value of the current model pruning rate to obtain the new current model pruning rate;

The second pruning unit is configured to be based on the new current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each Sensitivity information of a convolutional layer under different layer pruning rates, pruning the to-be-pruned model to obtain a new current pruning model;

a fifth determining unit, configured to determine, based on the third training data, detection information corresponding to each third training data corresponding to the new current pruning model;

a sixth determining unit, configured to determine the new current pruning based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data The model reaches the preset pruning condition, and the new current pruning model is determined as the target pruning model;

If based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions , and return to trigger the adjustment unit.

Optionally, the second determining unit is specifically configured to determine the loss corresponding to the pre-pruned model to be pruned based on the loss value of each task corresponding to the pre-pruned model to be pruned and the preset objective function. value;

Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The task weights corresponding to each task; L represents the number of tasks of the model to be pruned.

Optionally, the pre-pruning unit is specifically configured to, for each convolution kernel of the convolution layer, determine the pruning norm value corresponding to the convolution kernel based on each weight parameter in the convolution kernel;

Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to preset pruning rules, wherein the preset pruning The rule is: Pre-pruning is performed preferentially on the corresponding convolution kernel with a small pruning norm value.

It can be seen from the above content that the method and device for pruning a multi-task neural network model provided by the embodiments of the present invention obtain a trained multi-task neural network model as the model to be pruned; for each volume of the model to be pruned Layering, perform the following steps A-C to determine the sensitivity information of each convolutional layer under different preset layer pruning rates; Step A: for each preset layer pruning rate, based on the preset layer pruning rate and each convolution kernel in the convolution layer, pre-pruning the convolution layer; step B: based on the first training data, determine the pre-pruned model to be pruned corresponding to the pruning rate of the preset layer. Corresponding to the loss value of each task; Step C: Based on the loss value of each task corresponding to the pre-pruned model to be pruned, determine the sensitivity information of the convolutional layer at the preset layer pruning rate; Based on the model to be pruned Corresponding channel information, group the convolution kernels in the convolutional layer of the model to be pruned; based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the volume of each convolutional layer in each layer The pruning norm value corresponding to the accumulation kernel and the sensitivity information of each convolutional layer under different layer pruning rates are used to determine the target pruning model.

By applying the embodiments of the present invention, the task loss values corresponding to multiple tasks in the network to be pruned are used to jointly determine the sensitivity information of each layer of convolutional layers under different preset layer pruning rates, and then, based on the current model pruning The branch rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the sensitivity information of each convolutional layer at different layer pruning rates, Determine the target pruning model to achieve pruning of multi-task neural network models. In addition, combining the task loss values corresponding to multiple tasks as the pruning target, comprehensively considering the accuracy of multiple tasks for sensitivity analysis and pruning, the generalization performance of the sparse model is effectively preserved, and the pruned model can be balanced for multiple tasks. Accuracy loss, reducing the difficulty of retraining. Of course, it is not necessary for any product or method of the present invention to achieve all of the advantages described above at the same time.

The innovative points of the embodiments of the present invention include:

1. Use the task loss values corresponding to multiple tasks in the network to be pruned to jointly determine the sensitivity information of each layer of convolutional layers under different preset layer pruning rates, and then, based on the current model pruning rate, to be The grouping of the convolution kernels of the pruning model, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the sensitivity information of each convolutional layer under different layer pruning rates, determine the target pruning model to implement pruning of multi-task neural network models. In addition, the task loss value corresponding to multiple tasks is used as the pruning target, and the sensitivity analysis and pruning of the model are performed comprehensively considering the accuracy of multiple tasks, which effectively retains the generalization performance of the sparse model, and can balance multiple models after pruning. The accuracy loss of the task reduces the difficulty of retraining.

2. Set the preset pruning conditions, and based on the current loss value corresponding to the current pruning model, determine that the current pruning model meets the preset pruning conditions to determine whether the pruning model to be pruned is successfully pruned. In the case of successful model pruning, determine the current pruning model as the target pruning model; if it is determined that the pruning model to be pruned is unsuccessful, adjust the value of the current pruning model, and re-treat the pruning model Pruning is performed until it is determined that the pruning model to be pruned is successfully pruned, and the target pruning model is obtained, so as to obtain the target pruning model with good detection information and small storage space requirement.

3. Using the preset objective function, based on the loss value of each task and its corresponding task weight, determine the loss value corresponding to the pre-pruned model to be pruned, and then determine that the convolutional layer is under the preset layer pruning rate The sensitivity information of the model is comprehensively considered, and the sensitivity analysis and pruning of the model are carried out considering the accuracy of multiple tasks, and the effective pruning of the model is realized.

Description of drawings

In order to illustrate the embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that are required in the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic flowchart of a method for pruning a multi-task neural network model according to an embodiment of the present invention;

FIG. 2 is a schematic structural diagram of a device for pruning a multi-task neural network model according to an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

It should be noted that the terms "comprising" and "having" and any modifications thereof in the embodiments of the present invention and the accompanying drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the steps or units listed, but optionally also includes steps or units not listed, or optionally also includes For other steps or units inherent to these processes, methods, products or devices.

The invention provides a method and device for pruning a multi-task neural network model, so as to realize the pruning of the multi-task neural network model. The embodiments of the present invention will be described in detail below.

FIG. 1 is a schematic flowchart of a method for pruning a multi-task neural network model according to an embodiment of the present invention. The method may include the following steps:

S101: Obtain a trained multi-task neural network model as a model to be pruned.

The method for pruning a multi-task neural network model provided by the embodiment of the present invention can be applied to any electronic device with computing capability, and the electronic device can be a terminal or a server.

Wherein, the multi-task neural network model is a neural network model that can realize at least two detection tasks. The detection tasks may include, but are not limited to, tasks such as classification, localization, and semantic segmentation. The implementation of the present invention does not limit the specific task types of the multi-task neural network model.

The trained multi-task neural network model may refer to a multi-task neural network model that has been trained based on the training data and the detection accuracy of the detection information is not lower than a preset accuracy threshold.

S102: For each convolutional layer of the model to be pruned, perform the following steps S1021-S1023 to determine the sensitivity information of each convolutional layer under different preset layer pruning rates.

S1021: For each preset layer pruning rate, pre-prune the convolution layer based on the preset layer pruning rate and each convolution kernel in the convolution layer;

S1022: Based on the first training data, determine the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate;

S1023: Determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss values of each task corresponding to the pre-pruned model to be pruned.

The preset layer pruning rate is used to indicate the required pruning ratio of the pre-pruned convolutional layer during the pre-pruning process for each convolutional layer. A pruned convolutional layer requires the proportion of pre-pruned convolutional kernels to the total convolutional kernels of the convolutional layer. The value range of the preset layer pruning rate is (0, 1), the preset layer pruning rate is multiple, and the pruning rate of every two adjacent preset layers among the multiple preset layer pruning rates The difference between can be the same.

In this step, the electronic device may determine, for each convolutional layer of the model to be pruned, the sensitivity information of each convolutional layer under different preset layer pruning rates, that is, for each convolutional layer of the model to be pruned layer, for each preset layer pruning rate in turn, based on the preset layer pruning rate and each convolution kernel in the convolution layer, the convolution layer is pre-pruned, that is, based on the convolution layer in the convolution layer. For each convolution kernel, determine the convolution kernel to be pre-pruned corresponding to the preset layer pruning rate, and the pre-pruning of the convolution kernel to be pre-pruned corresponding to the preset layer pruning rate, for example: Setting the weight value corresponding to the convolution kernel to be pre-pruned corresponding to the pruning rate of the preset layer to 0, to realize the pre-pruning of the convolution kernel to be pre-pruned corresponding to the pruning rate of the preset layer; and then , based on the first training data, determine the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate, that is, the task loss values corresponding to multiple tasks; The loss value of each task corresponding to the model to be pruned, determine the sensitivity information of the convolutional layer under the pruning rate of the preset layer, and determine that each convolutional layer of the model to be pruned is pruned at each preset layer. Sensitivity information under branch rate. For example, if there are 10 convolutional layers in the model to be pruned, and the preset layer pruning rate includes an integer percentage from 1% to 99%, that is, 99, then the sensitivity of all convolutional layers of the model to be pruned The number of degree information includes 10*99=990.

Wherein, the first training data may be partial data of the training data of the model to be pruned after training, or may be other training data corresponding to the model to be pruned, which is all possible. The first training data may include a sample image and its corresponding calibration information, wherein the sample image includes the target to be detected by the model to be pruned, and the calibration information corresponding to the sample image includes the detection label corresponding to each task of the model to be pruned. details. For example, the target to be detected by the model to be pruned is a person, and the detection labels corresponding to each task of the model to be pruned are the identity of the person and the location of the person. Correspondingly, the sample image includes each person, and the calibration information corresponding to the sample image includes : Personnel identification information and location information of the persons included in the image.

Correspondingly, in one implementation, the above-mentioned process of determining the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate based on the first training data may be: Each sample image in the training data is input into the pre-pruned model to be pruned corresponding to the preset layer pruning rate, and the prediction information corresponding to the sample image is obtained, wherein the prediction information includes the prediction information of the to-be-pruned model. The prediction information corresponding to each task; for each task, based on the preset loss function corresponding to the task, the prediction information corresponding to the task corresponding to the sample image, and the calibration information corresponding to the task corresponding to the sample image in the first training data , determine the task sub-loss value corresponding to the task corresponding to the sample image, so as to obtain the task sub-loss value corresponding to the task corresponding to the sample image; and for each task, based on the task sub-loss value corresponding to the task corresponding to each sample image value to determine the task loss value corresponding to the task. The preset loss function corresponding to each task may be any type of loss function in the related art, which is not limited herein.

Among them, the smaller the change of the task loss value corresponding to the task before and after pre-pruning, the smaller the change of the task loss value, which represents the pre-pruning of the convolution layer in the convolution layer corresponding to the pruning rate of the preset layer, and the detection of the task corresponding to the task loss value. The smaller the influence of information, that is, the convolution kernel corresponding to the pruning rate of the preset layer of pre-pruning, the smaller the influence of the task detection information corresponding to the task loss value. Correspondingly, the corresponding loss value corresponding to the task can be represented. For the task of the pre-pruning, the sensitivity of the convolution kernel corresponding to the pruning rate of the preset layer of the pre-pruning is not high.

In an implementation manner of the present invention, the S1021 may include the following steps:

For each convolution kernel of the convolution layer, based on each weight parameter in the convolution kernel, determine the pruning norm value corresponding to the convolution kernel;

Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to the preset pruning rule, wherein the preset pruning rule is: : Prioritize pre-pruning for convolution kernels with small corresponding pruning norm values.

In this implementation manner, the electronic device may use the L1-norm algorithm for each convolution kernel of the convolution layer to determine the pruning norm value corresponding to the convolution kernel based on each weight parameter in the convolution kernel. Specifically, the absolute value of each weight parameter in the convolution kernel may be added, and the obtained result may be used as the pruning norm value corresponding to the convolution kernel. Among them, the larger the pruning norm value corresponding to the convolution kernel, the more important the convolution kernel is, and the lower the probability of choosing to prune the convolution kernel in the pre-pruning and subsequent pruning processes.

Subsequently, based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to a preset pruning rule. That is, pre-pruning is preferentially performed on the convolution kernel with a small corresponding pruning norm value. That is, in the convolutional layer, according to the preset layer pruning rate, the convolution kernel and its corresponding output feature map channel are pruned from the corresponding pruning norm value from small to large.

Among them, the above-mentioned trimming of the convolution kernel and its corresponding output feature map channel may be: setting each weight parameter contained in the convolution kernel to 0, or setting the weight value corresponding to the convolution kernel to 0, and setting the volume The weight value corresponding to the output feature map channel corresponding to the product kernel is 0.

S103: Based on the channel information corresponding to the model to be pruned, group the convolution kernels in the convolution layer of the model to be pruned.

In one implementation, there may be multiple convolutional layers connected by the ElementwiseLayer function in the model to be pruned. In view of the data processing method of the ElementwiseLayer function, for the multiple convolutional layers connected by the ElementwiseLayer function in the model to be pruned, When a convolution kernel in a certain layer needs to be clipped, the corresponding channel identification in other convolutional layers needs to be the same convolution kernel as the channel identification corresponding to the convolution kernel to be clipped. In view of this, the electronic device can pre-determine channel information corresponding to the model to be pruned, and further, based on the channel information corresponding to the model to be pruned, group the convolution kernels in the convolution layer of the model to be pruned. Among them, the convolution kernels corresponding to the same channel identification are divided into one group, the convolution kernels corresponding to different channel identifications are divided into different groups, and the convolution kernels of the convolutional layers not connected by the ElementwiseLayer function are the volumes that do not correspond to the channel identification. Kernels can be grouped into groups.

S104: Based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers The sensitivity information under the rate is used to determine the target pruning model.

The current model pruning rate may be pre-stored in the local or connected storage device of the electronic device, and the model pruning rate is used to limit the subtraction of the convolution kernel and its corresponding output during the pruning process of the model to be pruned. The ratio of feature map channels, which is used to limit the total number of convolution kernels and their corresponding output feature map channels in the process of pruning the model to be pruned, accounting for the convolution kernel and its corresponding output feature map channels included in the model to be pruned. The ratio of the total number of corresponding output feature map channels.

The pruning norm value corresponding to each convolution kernel in each convolution layer is: the value determined based on each weight parameter in the convolution kernel, which is used to characterize the importance of the convolution kernel in the convolution layer where it is located. The larger the value of the pruning norm corresponding to the convolution kernel in the convolution layer, the more important the convolution kernel is, and the smaller the probability of choosing to prune the convolution kernel in the pre-pruning and subsequent pruning process.

After the electronic device groups the convolution kernels in the convolution layer of the model to be pruned, the grouping situation of the convolution kernels of the model to be pruned can be obtained. Perform pruning or retain at the same time. For convolution kernels that do not correspond to channel identifiers, perform pruning or retention based on their corresponding pruning norm values.

The electronic device can determine the pruning rate corresponding to each convolutional layer based on the current model pruning rate and the sensitivity information of each convolutional layer at different pruning rates, where the sensitivity information of the convolutional layer is The less sensitive it is, the higher the corresponding pruning rate. Further, for each convolutional layer, the electronic device is based on the pruning rate corresponding to the convolutional layer, the grouping of the convolutional kernels of the model to be pruned, and the pruning corresponding to each convolutional kernel in each convolutional layer. Norm value, determine the specific convolution kernel that needs to be pruned for each convolution layer and its corresponding output feature map channel; based on the determined specific convolution kernel that needs to be pruned for each convolution layer, The corresponding output feature map channel is pruned to obtain the target pruning model.

Wherein, the above-mentioned principle of determining the specific convolution kernel that needs to be pruned for each convolutional layer and its corresponding output feature map channel may include: preferentially performing the pruning for the convolution kernel with a small corresponding pruning norm value in the convolutional layer. Pruning, and when a convolution kernel in a convolutional layer needs to be pruned, and the convolution kernel belongs to the convolution kernel group corresponding to a channel identifier, all corresponding convolution kernels in the convolution kernel group need to be pruned. The kernel and its corresponding output feature map channel are pruned.

By applying the embodiments of the present invention, the task loss values corresponding to multiple tasks in the network to be pruned are used to jointly determine the sensitivity information of each layer of convolutional layers under different preset layer pruning rates, and then, based on the current model pruning The branch rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the sensitivity information of each convolutional layer at different layer pruning rates, Determine the target pruning model to achieve pruning of multi-task neural network models. In addition, combining the task loss values corresponding to multiple tasks as the pruning target, comprehensively considering the accuracy of multiple tasks for sensitivity analysis and pruning, the generalization performance of the sparse model is effectively preserved, and the pruned model can be balanced for multiple tasks. Accuracy loss, reducing the difficulty of retraining.

Considering that the current model pruning rate may not be set properly, it may appear that the current model pruning rate setting is too small. In theory, under the constraint of detection accuracy, the model to be pruned can also prune more In the case of the convolution kernel and/or its corresponding output feature map channel, a model with a smaller amount of pruned data and a low detection accuracy can be obtained; or the current model pruning rate setting may be too large, The data volume of the model obtained after pruning is reduced, but the detection accuracy of the model obtained after pruning is not high. In the above cases, the models obtained after pruning can be considered as inadequate models. In view of this, in another embodiment of the present invention, the S104 may include the following steps:

Based on the current model pruning rate, the grouping of convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning rate of each convolutional layer at different layers Sensitivity information, prune the model to be pruned to obtain the current pruning model;

Based on the second training data, determine the detection information corresponding to each second training data corresponding to the current pruning model;

If it is determined that the current pruning model meets the preset pruning conditions based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, the current pruning model is determined as Target pruning model.

Wherein, the second training data may be partial data of the training data of the model to be pruned after training, or may be other training data corresponding to the model to be pruned, which is all possible. The second training data may include the same data as the first training data, or may include data different from the first training data.

The above-mentioned process of determining the detection information corresponding to each second training data corresponding to the current pruning model based on the second training data may be: inputting the sample images in each second training data into the current pruning model to obtain Detection information of each task corresponding to each second training data.

The above-mentioned process of determining that the current pruning model reaches the preset pruning condition based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data may be: based on increasing the The model obtained after pruning, the second training data, the detection information of each task corresponding to each second training data determined, the calibration information in each second training data, and the The preset accuracy determination method corresponding to the task determines the accuracy of the new pruned model for the task. If the accuracy of the task corresponding to the model to be pruned is judged to be the difference between the accuracy of the new pruned model If the value is not within the acceptable precision reduction range, and the difference is greater than the upper limit of the acceptable precision reduction range, it can be determined that the current pruning model meets the preset pruning conditions.

It can be understood that the corresponding preset accuracy determination methods for different tasks may be different, and the specific preset accuracy determination methods can refer to the determination methods for determining the accuracy of each task of the neural network model in the related art.

Taking the tasks of the model to be pruned, including the target detection task and the semantic segmentation task as an example, the pruning accuracy corresponding to the target detection task can be expressed by the mAP (Mean Average Precision) value; the pruning accuracy corresponding to the semantic segmentation task It can be represented by mIOU (Mean Intersection-Over-Union) value. For the calculation process of the mAP value and the mIOU value, please refer to the calculation process of the mAP value of the target detection task and the mIOU value of the semantic segmentation task in the neural network model in the related art, which will not be repeated here.

In another embodiment of the present invention, the S104 may further include the following steps:

If it is determined based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, it is determined that the current pruning model does not meet the preset pruning conditions, and the current model pruning rate is adjusted. The value of , to get the new current model pruning rate;

Based on the new current model pruning rate, the grouping of convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers According to the sensitivity information at the rate, prune the model to be pruned to obtain a new current pruning model;

Based on the third training data, determine the detection information corresponding to each third training data corresponding to the new current pruning model;

If it is determined that the new current pruning model meets the preset pruning conditions based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, the new current pruning model The pruning model is determined as the target pruning model;

If, based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions, and then return the The steps of adjusting the value of the current model pruning rate to obtain the new current model pruning rate.

If the electronic device judges that the difference between the accuracy of the task corresponding to the model to be pruned and the accuracy of the new model after pruning is not within the acceptable range of accuracy reduction, it can be determined that the current pruning model does not meet the predetermined accuracy. Set pruning conditions.

Correspondingly, the value of the current model pruning rate can be adjusted to obtain a new current model pruning rate, based on the new current model pruning rate, the grouping of the convolution kernels of the model to be pruned, each layer. The pruning norm value corresponding to each convolution kernel in the convolutional layer and the sensitivity information of each convolutional layer under different layer pruning rates are pruned, and the new current pruning model is obtained by pruning the model to be pruned; The third training data is to determine the detection information corresponding to each third training data corresponding to the new current pruning model; if the detection information corresponding to each third training data corresponding to the current pruning model and the the calibration information, determine that the new current pruning model reaches the preset pruning conditions, and determine the new current pruning model as the target pruning model; if the third training data corresponding to the current pruning model corresponds to The detection information and the calibration information in each third training data, determine that the new current pruning model does not meet the preset pruning conditions, return the value of the adjusted current model pruning rate, and obtain a new current model pruning branch rate steps.

Among them, if it is judged that the difference between the accuracy of the task corresponding to the model to be pruned and the new accuracy of pruning is not within the acceptable accuracy reduction range, and the accuracy of the task corresponding to the model to be pruned is the same as the new accuracy If the difference of the accuracy of the model after pruning exceeds the upper limit of the acceptable accuracy reduction range, the value of the current model pruning rate can be reduced to obtain the new current model pruning rate; The difference between the accuracy of the task corresponding to the pruned model and the accuracy of the new pruned model is within the acceptable accuracy drop range, and the accuracy of the modified task corresponding to the model to be pruned is the same as the accuracy of the new pruned model. If the difference in the accuracy of the post-branching model is lower than the upper limit of the acceptable accuracy drop range, the value of the current model pruning rate can be increased to obtain a new current model pruning rate.

The third training data may be the same data as the second training data, or may be data corresponding to the model to be pruned that is different from the second training data.

In another embodiment of the present invention, the S1023 may include the following steps:

Based on the loss value of each task corresponding to the pre-pruned model to be pruned and the preset objective function, determine the loss value corresponding to the pre-pruned model to be pruned;

Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The corresponding task weight; L represents the number of tasks to be pruned.

In one case, the change of the corresponding loss value of the model to be pruned before and after pre-pruning can be determined as the sensitivity information under the pruning rate of the preset layer. For example, it may be: the difference between the loss value corresponding to the model to be pruned after pre-pruning and the loss value corresponding to the model to be pruned before pre-pruning is determined as the pruning of the convolutional layer at the preset layer. Sensitivity information under branch rate. Correspondingly, the loss value corresponding to the model to be pruned after pre-pruning, compared with the loss value corresponding to the model to be pruned before pre-pruning, the smaller the increase is, it indicates that the convolutional layer is pruned at the preset layer. The smaller the impact on the accuracy of the detection information of the pruned model at the lower rate, the lower the sensitivity of the convolutional layer under the preset layer pruning rate.

In one case, the task weight corresponding to each task can be selected according to the range of the task loss value corresponding to the task, so that the weighted task loss value is at the same data level.

Corresponding to the above method embodiments, the embodiments of the present invention provide an apparatus for pruning a multi-task neural network model. As shown in FIG. 2 , the apparatus may include:

The obtaining module 210 is configured to obtain the trained multi-task neural network model as the model to be pruned;

The first determining module 220 is configured to, for each convolutional layer of the model to be pruned, determine the sensitivity information of each convolutional layer under different preset layer pruning rates through the following units;

The pre-pruning unit 221 is configured to pre-prune the convolution layer based on the preset layer pruning rate and each convolution kernel in the convolution layer for each preset layer pruning rate;

The first determining unit 222 is configured to determine the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate based on the first training data;

The second determination unit 223 is configured to determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned;

The grouping module 230 is configured to group the convolution kernels in the convolutional layers of the to-be-pruned model based on the channel information corresponding to the to-be-pruned model;

The second determination module 240 is configured to be based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each The sensitivity information of a convolutional layer at different layer pruning rates determines the target pruning model.

By applying the embodiments of the present invention, the task loss values corresponding to multiple tasks in the network to be pruned are used to jointly determine the sensitivity information of each layer of convolutional layers under different preset layer pruning rates, and then, based on the current model pruning The branch rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the sensitivity information of each convolutional layer at different layer pruning rates, Determine the target pruning model to achieve pruning of multi-task neural network models. In addition, combining the task loss values corresponding to multiple tasks as the pruning target, comprehensively considering the accuracy of multiple tasks for sensitivity analysis and pruning, the generalization performance of the sparse model is effectively preserved, and the pruned model can be balanced for multiple tasks. Accuracy loss, reducing the difficulty of retraining.

In another embodiment of the present invention, the second determining module 240 includes:

The first pruning unit (not shown in the figure) is configured to be based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, and the corresponding convolution kernels in each convolution layer. The pruning norm value and the sensitivity information of each convolutional layer under different layer pruning rates, prune the to-be-pruned model to obtain the current pruning model;

a third determining unit (not shown in the figure), configured to determine, based on the second training data, detection information corresponding to each second training data corresponding to the current pruning model;

The fourth determination unit (not shown in the figure) is configured to determine the current pruning model based on detection information corresponding to each second training data corresponding to the current pruning model and calibration information in each second training data The current pruning model reaches the preset pruning condition, and the current pruning model is determined as the target pruning model.

In another embodiment of the present invention, the second determining module 240 further includes:

An adjustment unit (not shown in the figure), configured to determine the current pruning model based on detection information corresponding to each second training data corresponding to the current pruning model and calibration information in each second training data If the branch model does not meet the preset pruning conditions, adjust the value of the current model pruning rate to obtain the new current model pruning rate;

The second pruning unit (not shown in the figure) is configured to be based on the new current model pruning rate, the grouping of the convolution kernels of the model to be pruned, and the corresponding convolution kernels in each convolution layer. The pruning norm value and the sensitivity information of each convolutional layer under different layer pruning rates, prune the to-be-pruned model to obtain a new current pruning model;

a fifth determining unit (not shown in the figure), configured to determine, based on the third training data, detection information corresponding to each third training data corresponding to the new current pruning model;

The sixth determination unit (not shown in the figure) is configured to determine if based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data. The new current pruning model reaches preset pruning conditions, and the new current pruning model is determined as the target pruning model;

If based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions , and return to trigger the adjustment unit.

In another embodiment of the present invention, the second determining unit 223 is specifically configured to determine the pre-pruned model based on the loss value of each task corresponding to the pre-pruned model to be pruned and the preset objective function. The loss value corresponding to the model to be pruned;

Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The task weights corresponding to each task; L represents the number of tasks of the model to be pruned.

In another embodiment of the present invention, the pre-pruning unit 221 is specifically configured to, for each convolution kernel of the convolution layer, determine the convolution kernel based on each weight parameter in the convolution kernel The corresponding pruning norm value;

Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to preset pruning rules, wherein the preset pruning The rule is: Pre-pruning is performed preferentially on the corresponding convolution kernel with a small pruning norm value.

The foregoing system and device embodiments correspond to the system embodiments, and have the same technical effects as the method embodiments. For specific descriptions, refer to the method embodiments. The apparatus embodiment is obtained based on the method embodiment, and the specific description can refer to the method embodiment section, which will not be repeated here. Those of ordinary skill in the art can understand that the accompanying drawing is only a schematic diagram of an embodiment, and the modules or processes in the accompanying drawing are not necessarily necessary to implement the present invention.

Those skilled in the art may understand that: the modules in the apparatus in the embodiment may be distributed in the apparatus in the embodiment according to the description of the embodiment, and may also be located in one or more apparatuses different from this embodiment with corresponding changes. The modules in the foregoing embodiments may be combined into one module, or may be further split into multiple sub-modules.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims (10)

1. a pruning method of multi-task neural network model, is characterized in that, described method comprises: Obtain the trained multi-task neural network model as the model to be pruned; For each convolutional layer of the model to be pruned, the following steps A-C are performed to determine the sensitivity information of each convolutional layer under different preset layer pruning rates; Step A: for each preset layer pruning rate, based on the preset layer pruning rate and each convolution kernel in the convolution layer, pre-prune the convolution layer; Step B: Based on the first training data, determine the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate; Step C: Determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned; grouping the convolution kernels in the convolution layer of the to-be-pruned model based on the channel information corresponding to the to-be-pruned model; Based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers The sensitivity information under the rate is used to determine the target pruning model. 2. The method according to claim 1, wherein the pruning rate based on the current model, the grouping situation of the convolution kernels of the model to be pruned, the convolution kernels in each convolution layer The corresponding pruning norm value and the sensitivity information of each convolutional layer at different layer pruning rates determine the steps of the target pruning model, including: Based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers Sensitivity information at the rate, prune the to-be-pruned model to obtain the current pruning model; determining, based on the second training data, detection information corresponding to each second training data corresponding to the current pruning model; If it is determined that the current pruning model meets the preset pruning condition based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, the current pruning model The pruning model is determined as the target pruning model. 3. The method of claim 2, wherein the method further comprises: If, based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data, it is determined that the current pruning model does not meet the preset pruning conditions, and the current pruning model is adjusted. The value of the model pruning rate to get the new current model pruning rate; Based on the new current model pruning rate, the grouping of convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and the pruning of each convolutional layer at different layers According to the sensitivity information at the rate, prune the model to be pruned to obtain a new current pruning model; Based on the third training data, determining the detection information corresponding to each third training data corresponding to the new current pruning model; If it is determined that the new current pruning model meets the preset pruning condition based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, determining the new current pruning model as the target pruning model; If based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions , and return to the step of adjusting the value of the current model pruning rate to obtain a new current model pruning rate. 4. The method according to any one of claims 1-3, characterized in that, based on the loss value of each task corresponding to the pre-pruned model to be pruned, it is determined that the convolutional layer is pruned at the preset layer. The steps for sensitivity information under branch rate include: Based on the loss value of each task corresponding to the pre-pruned model to be pruned and the preset objective function, determine the loss value corresponding to the pre-pruned model to be pruned; Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The task weights corresponding to each task; L represents the number of tasks of the model to be pruned. 5. The method of any one of claims 1-4, wherein the convolution layer is pre-pruned based on the preset layer pruning rate and each convolution kernel in the convolution layer steps, including: For each convolution kernel of the convolution layer, based on each weight parameter in the convolution kernel, determine the pruning norm value corresponding to the convolution kernel; Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to preset pruning rules, wherein the preset pruning The rule is: Pre-pruning is performed preferentially on the corresponding convolution kernel with a small pruning norm value. 6. A pruning device for a multi-task neural network model, wherein the device comprises: The obtaining module is configured to obtain the trained multi-task neural network model as the model to be pruned; The first determining module is configured to, for each convolutional layer of the model to be pruned, determine the sensitivity information of each convolutional layer under different preset layer pruning rates through the following units; a pre-pruning unit, configured to perform pre-pruning on the convolution layer based on the preset layer pruning rate and each convolution kernel in the convolution layer for each preset layer pruning rate; The first determining unit is configured to determine, based on the first training data, the loss value of each task corresponding to the pre-pruned model to be pruned corresponding to the preset layer pruning rate; The second determining unit is configured to determine the sensitivity information of the convolutional layer at the preset layer pruning rate based on the loss value of each task corresponding to the pre-pruned model to be pruned; a grouping module, configured to group the convolution kernels in the convolutional layer of the model to be pruned based on the channel information corresponding to the model to be pruned; The second determination module is configured to be based on the current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each The sensitivity information of the convolutional layer at different layer pruning rates determines the target pruning model. 7. The apparatus of claim 6, wherein the second determining module comprises: The first pruning unit is configured to be based on the current model pruning rate, the grouping situation of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each convolution kernel. Sensitivity information of a convolutional layer under different layer pruning rates, prune the to-be-pruned model to obtain the current pruning model; a third determining unit, configured to determine, based on the second training data, detection information corresponding to each second training data corresponding to the current pruning model; a fourth determination unit, configured to determine that the current pruning model reaches a preset level based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data The pruning condition is to determine the current pruning model as the target pruning model. 8. The apparatus of claim 7, wherein the second determining module further comprises: An adjustment unit, configured to determine that the current pruning model does not reach the preset pruning value based on the detection information corresponding to each second training data corresponding to the current pruning model and the calibration information in each second training data branch condition, adjust the value of the current model pruning rate to obtain the new current model pruning rate; The second pruning unit is configured to be based on the new current model pruning rate, the grouping of the convolution kernels of the model to be pruned, the pruning norm value corresponding to each convolution kernel in each convolutional layer, and each Sensitivity information of a convolutional layer under different layer pruning rates, pruning the to-be-pruned model to obtain a new current pruning model; a fifth determining unit, configured to determine, based on the third training data, detection information corresponding to each third training data corresponding to the new current pruning model; a sixth determining unit, configured to determine the new current pruning based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data The model reaches the preset pruning condition, and the new current pruning model is determined as the target pruning model; If based on the detection information corresponding to each third training data corresponding to the new current pruning model and the calibration information in each third training data, it is determined that the new current pruning model does not meet the preset pruning conditions , and return to trigger the adjustment unit. 9 . The device according to claim 6 , wherein the second determining unit is specifically configured to be based on the loss value of each task corresponding to the pre-pruned model to be pruned and a preset value. 10 . The objective function determines the loss value corresponding to the pre-pruned model to be pruned; Based on the loss value corresponding to the pre-pruned model to be pruned, the sensitivity information of the convolutional layer at the preset layer pruning rate is determined; wherein, the expression of the preset objective function is:

Among them, Objective represents the loss value corresponding to the pre-pruned model to be pruned, loss _i represents the task loss value corresponding to the i-th task in the to-be-pruned model, and w _i represents the i-th task in the to-be-pruned model The task weights corresponding to each task; L represents the number of tasks of the model to be pruned. 10. The apparatus according to any one of claims 6-9, wherein the pre-pruning unit is specifically configured to, for each convolution kernel of the convolution layer, based on the Each weight parameter determines the pruning norm value corresponding to the convolution kernel; Based on the preset layer pruning rate and the pruning norm value corresponding to each convolution kernel in the convolution layer, the convolution layer is pre-pruned according to preset pruning rules, wherein the preset pruning The rule is: Pre-pruning is performed preferentially on the corresponding convolution kernel with a small pruning norm value.

Images