CN117151185A - Deep neural network self-adaptive quantization method and system based on joint optimization - Google Patents

Deep neural network self-adaptive quantization method and system based on joint optimization Download PDF

Info

Publication number
CN117151185A
CN117151185A CN202310943969.1A CN202310943969A CN117151185A CN 117151185 A CN117151185 A CN 117151185A CN 202310943969 A CN202310943969 A CN 202310943969A CN 117151185 A CN117151185 A CN 117151185A
Authority
CN
China
Prior art keywords
activation
weight
neural network
model
quantization
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310943969.1A
Other languages
Chinese (zh)
Inventor
张蕊
刘畅
张曦珊
支天
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Institute of Computing Technology of CAS
Original Assignee
Institute of Computing Technology of CAS
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Institute of Computing Technology of CAS filed Critical Institute of Computing Technology of CAS
Priority to CN202310943969.1A priority Critical patent/CN117151185A/en
Publication of CN117151185A publication Critical patent/CN117151185A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Molecular Biology (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Computational Linguistics (AREA)
  • Mathematical Physics (AREA)
  • Biophysics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Medical Informatics (AREA)
  • Multimedia (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides a deep neural network self-adaptive quantization method and system based on joint optimization, comprising the following steps: and (3) jointly optimizing the dividing parameters and the neural network parameters so as to jointly perform data division of the deep neural network and neural network parameter training. And storing the partition parameters after training, quantifying the activation or weight of the execution model by using the partition parameters to obtain a final model, and executing corresponding tasks by using the final model. The invention can better fit the data distribution of the activation and the weight of the deep neural network, quantize the activation values and the weight of all linear layers in the deep neural network into low bits, thereby reducing quantization errors.

Description

Deep neural network self-adaptive quantization method and system based on joint optimization
Technical Field
The invention relates to the technical field of quantization acceleration of a deep neural network and the technical field of picture classification, in particular to a deep neural network self-adaptive quantization method and system based on joint optimization.
Background
In the deep learning field, in order to reduce training complexity and speed up training, quantized perceptual training is used for a deep neural network, that is, an activation value and a weight of a linear layer (including a convolution layer and a full connection layer) in the deep neural network are quantized into a fixed-point number or a floating-point number with a low bit width (such as 16 bits and 8 bits) in the training process.
Existing deep neural networks quantize the perception training effort, using uniform quantization or non-uniform quantization in a fixed format (such as floating point number FP 8). In practical use, however, the data distribution of different deep neural networks is different. These existing quantization methods allocate quantization intervals in the same manner, lack adaptation to different data, and therefore cannot effectively allocate quantization intervals when using low bit widths, and thus cannot obtain good quantization effects.
Disclosure of Invention
The invention aims to solve the problems that the prior art is difficult to allocate proper quantization intervals to the activation and weight of the deep neural network, ensure that the activation and weight are quantized into low-bit-width data and maintain the network accuracy effect. In practical applications, to maintain the network accuracy effect, quantization errors, that is, MSE mean square errors of quantized data and pre-quantized data, need to be reduced. Based on the purpose, the invention provides a self-adaptive quantized perceptual training method based on joint optimization.
The invention provides a deep neural network self-adaptive quantization method based on joint optimization, which comprises the following steps:
step 1, acquiring a deep neural network model to be trained as a target model, and counting the maximum absolute value Z of activation or weight in all linear layers of the target model; initializing the segmentation parameters of the activation or weight settings of each linear layer according to the maximum absolute value Z
Step 2, when the target model is in the t training iteration, the activation of the ith linear layer or the segmentation division parameter value of the weight isTo->Dividing the activation or weight of the ith linear layer into two parts, dividing +.> Region as center section, regions on both sides +.>And->As a surrounding section; quantifying the activation or weight in the target model to a low accuracy, wherein the quantization interval of the activation or weight at the center segment is smaller than the quantization interval of the activation or weight at the surrounding segments during the quantization process;
step 3, in the forward propagation process of the t training iteration of the target model, calculating a full-precision forward propagation result of the activation or weight and a quantized low-precision forward propagation result at the same time; obtaining the quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization error of each layer of the target model, multiplying the sum by a preset coefficient, and then adding the sum with a loss function of the training result of the target model to obtain a final loss function;
step 4, judging whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, storing the current target model as an execution model, andpreserving current segmentation partitioning parametersDividing parameter using current segment->Quantifying the activation or weight of the execution model to obtain a final model, and executing corresponding tasks by the final model; otherwise t is added with 1, and the step 2 is executed again.
According to the depth neural network self-adaptive quantization method based on joint optimization, the target model is an image classification depth neural network model, a target picture to be classified is input into the final model, and the final model performs a picture classification task to obtain the category of the target picture.
The depth neural network self-adaptive quantization method based on joint optimization, wherein
The quantization interval of the activation or weight at the center segment is:
the quantization interval of the activation or weight of the surrounding segments is:
the method for adaptively quantifying the deep neural network based on joint optimization comprises the following steps of:
the activation or weight of the segment located around is quantized as:
wherein the method comprises the steps ofIs the full-precision activation or weighting value of the ith linear layer in the t-th iteration,/->Is the low-precision fixed point number of the ith linear layer after quantization;
quantization error for each linear layer of the object modelSumming, multiplying the sum by a preset coefficient gamma and then adding the sum to a loss function of the training result of the target model>Adding to obtain a final loss function; the back propagation process calculates the gradient of the deep neural network parameters while calculating the segmentation parameters +.>Is a gradient of (2); calculated segmentation parameters after back propagation +.>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating, wherein the updating formula is as follows:
where η is the learning rate, then usedThe t+1st iteration is calculated.
The invention also provides a deep neural network self-adaptive quantization system based on joint optimization, which comprises the following steps:
the initial module is used for acquiring a deep neural network model to be trained as a target model, and counting the maximum absolute value Z of activation or weight in all linear layers of the target model; initializing the segmentation parameters of the activation or weight settings of each linear layer according to the maximum absolute value Z
The iterative training module is used for activating an ith linear layer or segmenting and dividing weight parameters into values when the target model is subjected to the t-th training iterationTo->Dividing the activation or weight of the ith linear layer into two parts, dividing +.>Region as center section, regions on both sides +.>And->As a surrounding section; quantifying the activation or weight in the target model to a low accuracy, wherein the quantization interval of the activation or weight at the center segment is smaller than the quantization interval of the activation or weight at the surrounding segments during the quantization process;
the loss calculation module is used for simultaneously calculating a full-precision forward propagation result of the activation or weight and a quantized low-precision forward propagation result in the forward propagation process of the t-th training iteration of the target model; obtaining the quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization error of each layer of the target model, multiplying the sum by a preset coefficient, and then adding the sum with a loss function of the training result of the target model to obtain a final loss function;
the task execution module is used for judging whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, the current target model is saved as an execution model, and the current segmentation dividing parameters are savedDividing parameter using current segment->Quantifying the activation or weight of the execution model to obtain a final model, and executing corresponding tasks by the final model; otherwise, t is added with 1, and the iterative training module is called again.
The target model is an image classification depth neural network model, a target picture to be classified is input into the final model, and the final model performs a picture classification task to obtain the category of the target picture.
The deep neural network self-adaptive quantification system based on joint optimization, wherein
The quantization interval of the activation or weight at the center segment is:
the quantization interval of the activation or weight of the surrounding segments is:
the joint optimization-based deep neural network adaptive quantization system, wherein the activation or weight at the central segment is quantized as follows:
the activation or weight of the segment located around is quantized as:
wherein the method comprises the steps ofIs the full-precision activation or weighting value of the ith linear layer in the t-th iteration,/->Is the low-precision fixed point number of the ith linear layer after quantization;
quantization error for each linear layer of the object modelSumming, multiplying the sum by a preset coefficient gamma and then adding the sum to a loss function of the training result of the target model>Adding to obtain a final loss function; the back propagation process calculates the gradient of the deep neural network parameters while calculating the segmentation parameters +.>Is a gradient of (2); calculated segmentation parameters after back propagation +.>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating, wherein the updating formula is as follows:
where η is the learning rate, then usedThe t+1st iteration is calculated.
The invention also provides a storage medium for storing a program for executing any one of the deep neural network adaptive quantization methods.
The invention also provides a client which is used for any deep neural network self-adaptive quantization system.
The advantages of the invention are as follows:
compared with the prior art, the method and the device can better fit the data distribution of the activation and the weight of the deep neural network, quantize the activation values and the weight of all linear layers in the deep neural network into low bits, thereby reducing quantization errors and obtaining less than 1% of reasoning accuracy loss.
Drawings
FIG. 1 is a schematic diagram of a segment division of the present invention;
FIG. 2 is a flow chart of the method of the present invention.
Detailed Description
The inventors have conducted accelerated studies of deep neural network quantization and found that existing works use uniform quantization or non-uniform quantization in a fixed format (such as floating point number FP 8), which lack adaptation to different data.
According to the invention, through researching the activation (and weight) distribution of the deep neural network, the distribution is found to be similar to Gaussian distribution, and the deep neural network can be divided into a central section and a peripheral section, and the characteristics of the different sections are different. The data value of the central section is smaller, the data quantity is large, and the data distribution change is large; the data value of the surrounding segment is larger, the data quantity is small, and the data distribution change is small. Meanwhile, the variance of the activation (and weight) distribution of the neural network with different layer depths is different, namely the height, width and length of the 'central segment' distribution are different. If uniform quantization at fixed intervals or non-uniform quantization in a fixed format (such as floating point number FP 8) is used as in the prior art, larger quantization errors may be caused, which may impair the effect of the deep neural network. Therefore, in order to reduce quantization errors when quantizing a deep neural network, it is necessary to adaptively adjust quantization intervals of different regions. However, how to allocate quantization intervals of different areas is a difficulty, if the number of divided areas is too large, the parameters to be optimized are too large, and it is difficult to find an optimal set of parameters.
The inventor finds that the optimal quantization interval distribution parameter can be adaptively realized in a combined optimization mode through research. According to the activation of the deep neural network and the shape characteristics of the weight distribution, the data distribution can be divided into a central segment and a peripheral segment. And then in the process of quantized perception training, different partitioning modes can be realized for different deep neural network distributions by adaptively learning a proper data distribution partitioning parameter, namely a segmented parameter, so that the most proper quantization interval of the deep neural network is adaptively obtained.
When learning the dividing parameters, in order to organically combine the learning data distribution dividing parameters with the training deep neural network parameters, and reduce the calculation cost of training the deep neural network due to the increase of the learning dividing parameters, the inventor proposes a self-adaptive quantization method based on joint optimization, and the dividing parameters and the neural network parameters are jointly optimized, so that the data division of the deep neural network and the neural network parameter training are jointly performed. Based on the above findings and thinking, the inventors propose an adaptive quantization method based on quantization training. In particular, in order to achieve the technical effects, the invention comprises the following key technical points:
the key point 1 is to adaptively divide the activation value (or weight) of each layer into a central segment and a peripheral segment, and quantize the two segments by using different quantization intervals. The absolute value of the data of the central section is small, the data volume is large, and the data is uniformly quantized by adopting a smaller quantization interval; the data of the 'surrounding segments' have large absolute values and small data size, and are quantized by adopting larger quantization intervals. The smaller quantization interval is used for the central segment, so that quantization errors become smaller; while the use of a larger quantization interval for the "surrounding segments" does not increase too much quantization error. Therefore, the whole quantization error can be reduced, and the problem that the network accuracy is reduced due to the overlarge quantization error in the existing quantization method is partially solved.
And the key point 2 is used for adaptively acquiring the segmentation dividing parameters through joint optimization. Selecting optimal segmentation dividing parameters by utilizing quantization errors, wherein the optimization process is to multiply the quantization errors by coefficients, then add the coefficients into a network loss function and optimize the network loss function together through a network optimizer, wherein the invention does not limit the specific form of the network loss function, and any loss function related to the deviation between a training result of a measurement model and a target value can be used as the network loss function; the segmentation dividing parameters are obtained in a self-adaptive mode through a joint optimization mode, so that the quantization method is better fit with different data distribution shapes of different depth neural networks, quantization errors of activation and weight are reduced, and the problem that network accuracy is reduced due to overlarge quantization errors of the existing quantization method is partially solved.
The quantization training method can be applied to training a currently common image classification depth neural network model, such as a ResNet-50 model, wherein the input model is image data, and the model outputs a classification result of a corresponding image. After the quantization training method is used, the quantization model with smaller precision loss can be obtained through combined self-adaptive quantization training, the calculated amount of training is reduced, the model training speed is accelerated, and the energy consumption generated by training the model is reduced. After training, in the use stage of the model, the calculation amount of the model can be reduced, the prediction speed of image classification is increased, the energy consumption generated by classification is reduced, and the space size used for storing the model is reduced.
In order to make the above features and effects of the present invention more clearly understood, the following specific examples are given with reference to the accompanying drawings.
Firstly defining a segmentation dividing parameter, defining the activation and weight area of a deep neural network as a central segment, and uniformly quantizing by using a smaller quantization interval; the portions other than the "center segment" are defined as "surrounding segments", and are uniformly quantized using a larger quantization interval, in a specific division manner as shown in fig. 1. In this case, the "center segment" has a small data value and a large data amount, and the data distribution is changed greatly, so that the quantization error of the "center segment" becomes small after using a small quantization interval. The data value of the 'surrounding segment' is larger, the data volume is small, the data distribution change is small, and the quantization error is not increased too much by using larger quantization interval. In summary, the adaptive quantization method based on joint optimization distributes the quantization interval used by the original 'peripheral segment' to the 'central segment', and on the premise that the data distribution change of the 'peripheral segment' is small and the data distribution change of the 'central segment' is large, the quantization error reduced by the 'central segment' is larger than the quantization error increased by the 'peripheral segment', so that the overall quantization error is reduced, and the accuracy loss of the quantized deep neural network is reduced.
The segmentation parameters are obtained by joint optimization of network parameters in the deep neural network training process. The optimization objective of the segmentation partitioning parameters is to minimize the quantization error MSE, so as to ensure that the quantization error is small enough and reduce the resulting decline of the model accuracy. For a deep neural network, respectively learning respective segmentation dividing parameters for the activation and the weight of each layer, namely, each layer is provided with independent segmentation dividing parameters; and summing the quantization errors of each layer of the deep neural network, multiplying the quantization errors by a coefficient set manually, and adding the sum with a loss function of the deep neural network as a final loss function for training the deep neural network. In this way, in the training process, the segmentation dividing parameters are jointly optimized with the network parameters, and finally the dividing parameters of the central segment and the peripheral segment which have the minimum quantization error and can be most suitable for the activation and weight distribution shapes are obtained. In the reasoning stage, the activation and weight of the deep neural network are divided into a central segment and a peripheral segment by using the segmentation dividing parameters obtained by training, and quantization is carried out by using corresponding quantization intervals respectively. The neural network model can be compressed by quantization, and the quantized neural network model has the advantages of being low in computational complexity, high in computational speed, more power-saving in training the model, low in energy consumption and small in model storage space.
The whole flow of the adaptive quantization method based on joint optimization is shown in fig. 2. The adaptive quantization method based on joint optimization is described below by taking activation (activation value) as an example. The weight is calculated in the same way.
Before deep neural network training begins, all layers, such as the convolutional layer and the linear layer of the fully connected layer, are counted, and the maximum absolute value of the activation (or weight) is Z. An initialized segmentation partitioning parameter is set for each layer of activation (or weight). Let the initial value of the segmentation dividing parameter of the activation (or weight) of the ith layer beFor example, may be set to 1/4 of the maximum absolute value Z.
In the t-th iteration, the segmentation division parameter value of the activation (or weight) of the i-th layer isApplication ofDividing the activation (or weight) of the i-th layer into two parts, dividing +.>The region is defined as the "center segment", the regions on both sides +.>And->Defined as "surrounding segments". The "center segment" uses a smaller quantization interval as
The "surrounding segments" use larger quantization intervals as
In the forward propagation process of the t-th iteration, a full-precision forward propagation result and a quantized low-precision forward propagation result are simultaneously calculated.
Quantize the activation (or weight) value of the "center segment" to
Quantifying the activation (or weight) value of "surrounding segments" as
Wherein the method comprises the steps ofIs the full precision activation (or weight) value of the ith layer in the t-th iteration,/->Is the number of low precision fixed points of the i-th layer after quantization. The convolution and full connection layers in the deep neural network are then calculated using the low precision fixed point numbers.
After the forward propagation is completed, calculating the quantization error of each layer of the deep neural network and the quantization error of the ith layerMultiplying an artificially set coefficient gamma with the loss function of the deep neural network itself +.>And adding as a final loss function. Then, the back propagation process is calculated, the gradient of the deep neural network parameter is calculated, and the segmentation parameter is calculated at the same time>Is a gradient of (a). Setting the calculated segmentation dividing parameter after back propagation>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating the formula to be
Where η is the learning rate. After that useThe t+1st iteration is calculated.
After multiple iterations, the deep neural network converges, and finally the segmentation division parameter value capable of guaranteeing the network precision is obtained. It is used for quantization when deep neural network reasoning.
The following is a system example corresponding to the above method example, and this embodiment mode may be implemented in cooperation with the above embodiment mode. The related technical details mentioned in the above embodiments are still valid in this embodiment, and in order to reduce repetition, they are not repeated here. Accordingly, the related technical details mentioned in the present embodiment can also be applied to the above-described embodiments.
The invention also provides a deep neural network self-adaptive quantization system based on joint optimization, which comprises the following steps:
initial initiationThe module is used for acquiring a deep neural network model to be trained as a target model, and counting the maximum absolute value Z of the activation or weight in all linear layers of the target model; initializing the segmentation parameters of the activation or weight settings of each linear layer according to the maximum absolute value Z
The iterative training module is used for activating an ith linear layer or segmenting and dividing weight parameters into values when the target model is subjected to the t-th training iterationTo->Dividing the activation or weight of the ith linear layer into two parts, dividing +.>Region as center section, regions on both sides +.>And->As a surrounding section; quantifying the activation or weight in the target model to a low accuracy, wherein the quantization interval of the activation or weight at the center segment is smaller than the quantization interval of the activation or weight at the surrounding segments during the quantization process;
the loss calculation module is used for simultaneously calculating a full-precision forward propagation result of the activation or weight and a quantized low-precision forward propagation result in the forward propagation process of the t-th training iteration of the target model; obtaining the quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization error of each layer of the target model, multiplying the sum by a preset coefficient, and then adding the sum with a loss function of the training result of the target model to obtain a final loss function;
the task execution module is used for judging whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, the current target model is saved as an execution model, and the current segmentation dividing parameters are savedDividing parameter using current segment->Quantifying the activation or weight of the execution model to obtain a final model, and executing corresponding tasks by the final model; otherwise, t is added with 1, and the iterative training module is called again.
The target model is an image classification depth neural network model, a target picture to be classified is input into the final model, and the final model performs a picture classification task to obtain the category of the target picture.
The deep neural network self-adaptive quantification system based on joint optimization, wherein
The quantization interval of the activation or weight at the center segment is:
the quantization interval of the activation or weight of the surrounding segments is:
the joint optimization-based deep neural network adaptive quantization system, wherein the activation or weight at the central segment is quantized as follows:
the activation or weight of the segment located around is quantized as:
wherein the method comprises the steps ofIs the full-precision activation or weighting value of the ith linear layer in the t-th iteration,/->Is the low-precision fixed point number of the ith linear layer after quantization;
quantization error for each linear layer of the object modelSumming, multiplying the sum by a preset coefficient gamma and then adding the sum to a loss function of the training result of the target model>Adding to obtain a final loss function; the back propagation process calculates the gradient of the deep neural network parameters while calculating the segmentation parameters +.>Is a gradient of (2); calculated segmentation parameters after back propagation +.>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating, wherein the updating formula is as follows:
where η is the learning rate, then usedThe t+1st iteration is calculated.
The invention also provides a storage medium for storing a program for executing any one of the deep neural network adaptive quantization methods.
The invention also provides a client which is used for any deep neural network self-adaptive quantization system.

Claims (10)

1. The method for adaptively quantifying the deep neural network based on the joint optimization is characterized by comprising the following steps of:
step 1, acquiring a deep neural network model to be trained as a target model, and counting the maximum absolute value Z of activation or weight in all linear layers of the target model; initializing the segmentation parameters of the activation or weight settings of each linear layer according to the maximum absolute value Z
Step 2, when the target model is in the t training iteration, the activation of the ith linear layer or the segmentation division parameter value of the weight isTo->Dividing the activation or weight of the ith linear layer into two parts, dividing +.> Region as center section, regions on both sides +.>And->As a surrounding section; quantifying the activation or weight in the target model to a low accuracy, wherein the quantization interval of the activation or weight at the center segment is smaller than the quantization interval of the activation or weight at the surrounding segments during the quantization process;
step 3, in the forward propagation process of the t training iteration of the target model, calculating a full-precision forward propagation result of the activation or weight and a quantized low-precision forward propagation result at the same time; obtaining the quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization error of each layer of the target model, multiplying the sum by a preset coefficient, and then adding the sum with a loss function of the training result of the target model to obtain a final loss function;
step 4, judging whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, storing the current target model as an execution model, and storing the current segmentation dividing parametersDividing parameter using current segment->Quantifying the activation or weight of the execution model to obtain a final model, and executing corresponding tasks by the final model; otherwise t is added with 1, and the step 2 is executed again.
2. The adaptive quantization method of deep neural network based on joint optimization as claimed in claim 1, wherein the target model is an image classification deep neural network model, a target picture to be classified is input into the final model, and the final model performs a picture classification task to obtain the category to which the target picture belongs.
3. The method for adaptively quantizing deep neural networks based on joint optimization according to claim 1,
the quantization interval of the activation or weight at the center segment is:
the quantization interval of the activation or weight of the surrounding segments is:
4. a joint optimization-based deep neural network adaptive quantization method according to claim 3, characterized in that the activation or weight at the central segment is quantized as:
the activation or weight of the segment located around is quantized as:
wherein the method comprises the steps ofIs the full-precision activation or weighting value of the ith linear layer in the t-th iteration,/->Is the low-precision fixed point number of the ith linear layer after quantization;
each line of the object modelQuantization error of sex layerSumming, multiplying the sum by a preset coefficient gamma and then adding the sum to a loss function of the training result of the target model>Adding to obtain a final loss function; the back propagation process calculates the gradient of the deep neural network parameters while calculating the segmentation parameters +.>Is a gradient of (2); calculated segmentation parameters after back propagation +.>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating, wherein the updating formula is as follows:
where η is the learning rate, then usedThe t+1st iteration is calculated.
5. A joint optimization-based deep neural network adaptive quantization system, comprising:
an initial module for acquiring a deep neural network model to be trained as a target model, and counting all linear layers of the target modelMaximum absolute value Z of the mid-activation or weight; initializing the segmentation parameters of the activation or weight settings of each linear layer according to the maximum absolute value Z
The iterative training module is used for activating an ith linear layer or segmenting and dividing weight parameters into values when the target model is subjected to the t-th training iterationTo->Dividing the activation or weight of the ith linear layer into two parts, dividing +.>Region as center section, regions on both sides +.>And->As a surrounding section; quantifying the activation or weight in the target model to a low accuracy, wherein the quantization interval of the activation or weight at the center segment is smaller than the quantization interval of the activation or weight at the surrounding segments during the quantization process;
the loss calculation module is used for simultaneously calculating a full-precision forward propagation result of the activation or weight and a quantized low-precision forward propagation result in the forward propagation process of the t-th training iteration of the target model; obtaining the quantization error of each layer of the target model according to the full-precision forward propagation result and the low-precision forward propagation result, summing the quantization error of each layer of the target model, multiplying the sum by a preset coefficient, and then adding the sum with a loss function of the training result of the target model to obtain a final loss function;
the task execution module is configured to execute the task,for judging whether the final loss function is smaller than a preset value or t reaches a preset iteration number, if yes, storing the current target model as an execution model, and storing the current segmentation dividing parametersDividing parameter using current segment->Quantifying the activation or weight of the execution model to obtain a final model, and executing corresponding tasks by the final model; otherwise, t is added with 1, and the iterative training module is called again.
6. The adaptive quantization system of deep neural network based on joint optimization of claim 5, wherein the target model is an image classification deep neural network model, a target picture to be classified is input into the final model, and the final model performs a picture classification task to obtain a category to which the target picture belongs.
7. The deep neural network adaptive quantization system based on joint optimization of claim 5,
the quantization interval of the activation or weight at the center segment is:
the quantization interval of the activation or weight of the surrounding segments is:
8. the joint optimization-based deep neural network adaptive quantization system of claim 7, wherein the activation or weighting at the center segment is quantized to:
the activation or weight of the segment located around is quantized as:
wherein the method comprises the steps ofIs the full-precision activation or weighting value of the ith linear layer in the t-th iteration,/->Is the low-precision fixed point number of the ith linear layer after quantization;
quantization error for each linear layer of the object modelSumming, multiplying the sum by a preset coefficient gamma and then adding the sum to a loss function of the training result of the target model>Adding to obtain a final loss function; the back propagation process calculates the gradient of the deep neural network parameters while calculating the segmentation parameters +.>Is a gradient of (2); calculated segmentation parameters after back propagation +.>Gradient of +.>Then after the t-th iteration is finished, dividing the parameters in sections +.>Updating, wherein the updating formula is as follows:
where η is the learning rate, then usedThe t+1st iteration is calculated.
9. A storage medium storing a program for executing the deep neural network adaptive quantization method according to any one of claims 1 to 4.
10. A client for the deep neural network adaptive quantization system of any one of claims 5 to 8.
CN202310943969.1A 2023-07-28 2023-07-28 Deep neural network self-adaptive quantization method and system based on joint optimization Pending CN117151185A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310943969.1A CN117151185A (en) 2023-07-28 2023-07-28 Deep neural network self-adaptive quantization method and system based on joint optimization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310943969.1A CN117151185A (en) 2023-07-28 2023-07-28 Deep neural network self-adaptive quantization method and system based on joint optimization

Publications (1)

Publication Number Publication Date
CN117151185A true CN117151185A (en) 2023-12-01

Family

ID=88883214

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310943969.1A Pending CN117151185A (en) 2023-07-28 2023-07-28 Deep neural network self-adaptive quantization method and system based on joint optimization

Country Status (1)

Country Link
CN (1) CN117151185A (en)

Similar Documents

Publication Publication Date Title
US20210256348A1 (en) Automated methods for conversions to a lower precision data format
CN109002889B (en) Adaptive iterative convolution neural network model compression method
CN108491926B (en) Low-bit efficient depth convolution neural network hardware accelerated design method, module and system based on logarithmic quantization
KR20190051755A (en) Method and apparatus for learning low-precision neural network
CN111612147A (en) Quantization method of deep convolutional network
CN110555450A (en) Face recognition neural network adjusting method and device
CN111489364B (en) Medical image segmentation method based on lightweight full convolution neural network
CN109978144B (en) Model compression method and system
WO2023011002A1 (en) Overflow-aware quantization model training method and apparatus, medium and terminal device
CN114418057A (en) Operation method of convolutional neural network and related equipment
CN114139683A (en) Neural network accelerator model quantization method
CN115545177A (en) Transformer hardware accelerator based on FPGA
CN116502691A (en) Deep convolutional neural network mixed precision quantization method applied to FPGA
TWI722491B (en) A separate quantization method for a 4-bit and 8-bit combination of a neural network
CN114462591A (en) Inference method for dynamic quantitative neural network
CN112613604A (en) Neural network quantification method and device
CN117151185A (en) Deep neural network self-adaptive quantization method and system based on joint optimization
CN115860062A (en) Neural network quantization method and device suitable for FPGA
CN111614358B (en) Feature extraction method, system, equipment and storage medium based on multichannel quantization
Hoang et al. FPGA oriented compression of DNN using layer-targeted weights and activations quantization
CN116227563A (en) Convolutional neural network compression and acceleration method based on data quantization
KR102657904B1 (en) Method and apparatus for multi-level stepwise quantization for neural network
CN114169513A (en) Neural network quantization method and device, storage medium and electronic equipment
CN111210009A (en) Information entropy-based multi-model adaptive deep neural network filter grafting method, device and system and storage medium
CN113762496B (en) Method for reducing low-bit convolutional neural network reasoning operation complexity

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination