WO2022000373A1 - 基于压缩感知的神经网络模型压缩方法、设备及存储介质 - Google Patents

基于压缩感知的神经网络模型压缩方法、设备及存储介质 Download PDF

Info

Publication number
WO2022000373A1
WO2022000373A1 PCT/CN2020/099752 CN2020099752W WO2022000373A1 WO 2022000373 A1 WO2022000373 A1 WO 2022000373A1 CN 2020099752 W CN2020099752 W CN 2020099752W WO 2022000373 A1 WO2022000373 A1 WO 2022000373A1
Authority
WO
WIPO (PCT)
Prior art keywords
training
original
transform domain
loss function
weight parameters
Prior art date
Application number
PCT/CN2020/099752
Other languages
English (en)
French (fr)
Inventor
高伟
郭洋
Original Assignee
北京大学深圳研究生院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京大学深圳研究生院 filed Critical 北京大学深圳研究生院
Priority to CN202080001140.4A priority Critical patent/CN112352249A/zh
Priority to PCT/CN2020/099752 priority patent/WO2022000373A1/zh
Publication of WO2022000373A1 publication Critical patent/WO2022000373A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means

Definitions

  • the present application relates to the technical field of machine learning, and in particular, to a compression sensing-based neural network model compression method, device, and computer-readable storage medium.
  • the main purpose of this application is to provide a neural network model compression method based on compressed sensing, which aims to solve the technical problem of low compression performance of the existing network compression methods.
  • the present application provides a compression sensing-based neural network model compression method
  • the compression sensing-based neural network model compression method includes:
  • the present application also provides a compressed sensing-based neural network model compression device, and the compressed sensing-based neural network model compression device includes:
  • the weight parameter training module is used to obtain the original weight parameters of the neural network model to be compressed, and based on the preset first loss function and the basis of the preset transform domain, iteratively train the original weight parameters in the original domain to obtain the training weight parameters, and an observation matrix corresponding to the training weight parameter, wherein the first loss function is jointly designed according to the neural network modeling idea and the compressed sensing modeling idea;
  • the compressed sensing reconstruction module is used to perform compressed sensing reconstruction according to the basis of the transform domain and the observation matrix, obtain a reconstructed sparse coefficient representing the sparsity of the training weight parameter in the transform domain, and determine the training weight parameter in the transform. fixed sparsity within the domain; and,
  • a sparse coefficient training module configured to integrate the fixed sparsity into the first loss function to obtain a second loss function, and perform iterative training on the reconstructed sparse coefficient and the training weight parameter based on the second loss function, Retraining weight parameters and target sparse coefficients representing the sparsity of the retraining weight parameters in the transform domain are obtained, and a compressed neural network model is mapped in the transform domain according to the target sparse coefficients.
  • the weight parameter training module includes:
  • the first iterative training unit is used to obtain the original weight parameters of the neural network model to be compressed, perform iterative training on the original weight parameters in the original domain based on the first loss function and the basis of the preset transform domain, and obtain the The first real-time accuracy rate corresponding to the current network performance during the iterative training process of the original weight parameters;
  • the first end determination unit is configured to determine that the first real-time test accuracy rate is greater than or equal to the preset first reference accuracy rate, end the current iterative training process, and obtain the training weight parameter and the corresponding training weight parameter. observation matrix.
  • the obtaining the original weight parameters of the neural network model to be compressed based on the first loss function and the basis of the preset transform domain, iteratively train the original weight parameters in the original domain, and obtain the During the iterative training process of the original weight parameters, the steps of the first real-time accuracy rate corresponding to the current network performance include:
  • the steps of determining that the first real-time test accuracy rate is greater than or equal to the preset first benchmark accuracy rate, ending the current iterative training process, and obtaining the training weight parameters and the observation matrix corresponding to the training weight parameters include:
  • the expression of the first loss function is:
  • ⁇ l(f(x,W),y) represents the expression of the conventional loss function
  • ⁇ and ⁇ represent the preset multiplication factor
  • W represents the original weight parameter
  • ⁇ represents the basis of the transform domain
  • s represents the For the original sparse coefficients
  • the subscript 2 represents using a 2-norm to constrain the relationship between the training weight parameter W and the original sparse coefficient s
  • the subscript 1 represents using a 1-norm to constrain the original sparse coefficient s.
  • the compressed sensing reconstruction module includes:
  • the sparse coefficient training module includes:
  • a second iterative training unit configured to integrate the fixed sparsity into the first loss function to obtain a second loss function, and based on the second loss function, iterate the reconstructed sparsity coefficient and the training weight parameter training, and obtain the second real-time accuracy rate corresponding to the current network performance in the iterative training process for the reconstructed sparse coefficient;
  • a second end determination unit configured to end the current iterative training process for the reconstructed sparse coefficients when it is detected that the second real-time test accuracy rate is greater than or equal to the preset second reference accuracy rate, and obtain retraining weight parameters, and a target sparse coefficient representing the sparsity of the retraining weight parameter in the transform domain, and a compressed neural network model is mapped in the transform domain according to the target sparse coefficient.
  • the steps of the second real-time accuracy include:
  • the second loss function stochastic gradient descent training is performed on the reconstructed sparse coefficient and the training weight parameter under the fixed sparsity, and in the iterative training process of the reconstructed sparse coefficient, the difference between the current network and the current network is obtained.
  • the second real-time accuracy rate corresponding to the performance.
  • the expression of the second loss function is:
  • the present application also provides a compressed sensing-based neural network model compression device
  • the compressed sensing-based neural network model compression device includes: a memory, a processor, and a device stored on the memory and available in the Computer-readable instructions running on the processor, when the computer-readable instructions are executed by the processor, implement the steps of the compression method for a neural network model based on compressed sensing as described above.
  • the present application also provides a computer-readable storage medium, where computer-readable instructions are stored on the computer-readable storage medium, and when the computer-readable instructions are executed by a processor, the above-mentioned based Steps of a compression method for neural network model compression for compressed sensing.
  • the present application provides a neural network model compression method, device and computer-readable storage medium based on compressed sensing.
  • the neural network model compression method based on compressed sensing obtains the original weight parameters of the neural network model to be compressed, and iterates the original weight parameters in the original domain based on the preset first loss function and the basis of the preset transform domain.
  • the first loss function is jointly designed according to the neural network modeling idea and the compressed sensing modeling idea; Perform compressed sensing reconstruction with the observation matrix to obtain a reconstructed sparsity coefficient representing the sparsity of the training weight parameter in the transform domain, and determine the fixed sparsity of the training weight parameter in the transform domain; integrating the fixed sparsity into The first loss function obtains a second loss function, and based on the second loss function, the reconstructed sparse coefficient and the training weight parameter are iteratively trained to obtain a retraining weight parameter, which is the same as that representing the retraining weight parameter.
  • the target sparse coefficient of the sparsity in the transform domain is transformed, and the compressed neural network model is mapped in the transform domain according to the target sparse coefficient.
  • the present application integrates standard compressed sensing constraint modeling into a conventional loss function, designs a first loss function based on this, and uses the first loss function to train the original weight parameters, so that the compressed sensing
  • the modeling process is integrated into the network training process; the reconstructed sparse coefficients and fixed sparse coefficients are obtained by compressive sensing reconstruction based on the basis of the transform domain and the observation matrix obtained by training the original weight parameters, so that the training weights can be fully mined in a transform domain.
  • the inherent sparsity of the parameters by using the inherent sparsity to train the reconstructed sparsity coefficients, the network training process and the network compression process are combined, and the sparsity of the weight parameters is fully trained and mined as much as possible, and finally the neural network model is fully implemented.
  • the purpose of compression is to solve the technical problem of low compression performance of the existing network compression method.
  • FIG. 1 is a schematic structural diagram of a neural network model compression device based on compressed sensing of the hardware operating environment involved in the solution of the embodiment of the present application;
  • FIG. 2 is a schematic flowchart of the first embodiment of the compression sensing-based neural network model compression method of the present application.
  • FIG. 1 is a schematic structural diagram of a compressive sensing-based neural network model compression device of the hardware operating environment involved in the solution of the embodiment of the present application.
  • the neural network model compression device based on compressed sensing in this embodiment of the present application may be a terminal device such as a server and a PC.
  • the compressed sensing-based neural network model compression device may include: a processor 1001 , such as a CPU, a communication bus 1002 , a user interface 1003 , a network interface 1004 , and a memory 1005 .
  • the communication bus 1002 is used to realize the connection and communication between these components.
  • the user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface.
  • the network interface 1004 may include a standard wired interface and a wireless interface (eg, a WI-FI interface).
  • the memory 1005 may be high-speed RAM memory, or may be non-volatile memory, such as disk memory.
  • the memory 1005 may also be a storage device independent of the aforementioned processor 1001 .
  • the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and computer-readable instructions.
  • the network interface 1004 is mainly used to connect to the background server and perform data communication with the background server;
  • the user interface 1003 is mainly used to connect to the client (client) and perform data communication with the client;
  • the processor 1001 can be used to invoke computer readable instructions stored in memory 1005 and perform the following operations:
  • the present application provides a neural network model compression method based on compressed sensing, that is, the standard compressed sensing constraint modeling is integrated into a conventional loss function, and a first loss function is designed based on this, using the first loss function.
  • the function trains the original weight parameters, so that the compressive sensing modeling process can be integrated into the network training process; through compressive sensing reconstruction based on the basis of the transform domain and the observation matrix obtained by training the original weight parameters, the reconstructed sparse coefficient and fixed sparse coefficients, so that the inherent sparsity of the training weight parameters can be fully mined in a transform domain; by using the inherent sparsity to train the reconstructed sparse coefficients, the network training process and the network compression process are combined to fully train the mining weights as much as possible.
  • the sparsity of the parameters finally achieves the purpose of fully compressing the neural network model, thereby solving the technical problem of low compression performance of the existing network compression methods.
  • FIG. 2 is a schematic flowchart of a first embodiment of a neural network model compression method based on compressed sensing.
  • the first embodiment of the present application provides a method for compressing a neural network model based on compressive sensing, and the method for compressing a neural network model based on compressive sensing includes the following steps:
  • Step S10 Obtain the original weight parameters of the neural network model to be compressed, and based on the preset first loss function and the basis of the preset transform domain, perform iterative training on the original weight parameters in the original domain to obtain the training weight parameters and The observation matrix corresponding to the training weight parameter, wherein the first loss function is jointly designed according to the neural network modeling idea and the compressed sensing modeling idea;
  • the original weight parameter is an untrained weight parameter of the neural network model to be compressed currently.
  • the training weight parameter is the weight parameter after the original weight parameter is trained in the original domain.
  • the preset first loss function is a loss function used to train the original weight parameters corresponding to the neural network model to be detected, which is jointly designed based on the conventional neural network modeling idea and the compressed sensing modeling idea.
  • compressed sensing also known as compressed sampling, sparse sampling, compressed sensing.
  • it uses random sampling to obtain discrete samples of the signal by exploiting the sparse characteristics of the signal under the condition of much smaller than the Nyquist sampling rate, and then perfectly reconstructs the signal through a nonlinear reconstruction algorithm.
  • the computer when the computer obtains the original weight parameter of the neural network model to be compressed, the original weight parameter is used as the input signal in the compressed sensing modeling process, and the real value of the training sample and the transformation domain space are also required to be input. basis, network learning rate, and multiplication factor, etc.
  • the original weight parameters can also be randomly initialized before being trained and created to induce the initialization parameters to be sparse coefficient representations of the original weight parameters in the transform domain during training. Based on the first loss function, the original weight parameters and the initialization parameters are synchronized and iteratively trained, and the test accuracy rate reflecting the current network performance is obtained in real time during the training process.
  • the computer When the computer detects that the current test accuracy rate reaches a certain standard, it can end the current iterative training process for the original weight parameters.
  • the criterion for determining the end of the training may be that the current test accuracy is not less than a certain preset standard value, or the error between the current test accuracy rate and a certain preset standard value is within a certain threshold, etc.
  • This embodiment No specific limitation is made.
  • the original weight parameters become the training weight parameters after parameter updating during the training process, and the initialization parameters at this time are also trained synchronously with the original weight parameters to be the original sparsity coefficients representing the sparsity of the original weight parameters in the transform domain, and can be obtained and trained.
  • the observation matrix corresponding to the weight parameter is not less than a certain preset standard value, or the error between the current test accuracy rate and a certain preset standard value is within a certain threshold, etc.
  • Step S20 performing compressed sensing reconstruction according to the basis of the transform domain and the observation matrix, obtaining a reconstructed sparse coefficient representing the sparsity of the training weight parameter in the transform domain, and determining the fixed sparseness of the training weight parameter in the transform domain.
  • the present application can fully exploit the inherent sparsity of training weight parameters in the transform domain by introducing the principle of compressed sensing. After iterative training of the original weight parameters, the compressed sensing process begins.
  • the parameters required in the compressed sensing reconstruction process include: the basis of the transform domain, the observation matrix and the sampling matrix. The role of the sampling matrix is to project the high-dimensional signal into a low-dimensional space.
  • the compressed sensing reconstruction is performed by using the basis, observation matrix and sampling matrix of the transform domain, and a flag bit is generated to fix the fixed sparsity of the training weight parameters in the transform domain.
  • the specific compressed sensing reconstruction method can be selected by the subspace pursuit (SP, subspace pursuit).
  • Step S30 integrating the fixed sparsity into the first loss function to obtain a second loss function, and performing iterative training on the reconstructed sparsity coefficient and the training weight parameter based on the second loss function to obtain a retraining weight parameters, and a target sparsity coefficient representing the sparsity of the retraining weight parameter in the transform domain, and a compressed neural network model is mapped in the transform domain according to the target sparsity coefficient.
  • the second loss function is a loss function used to train the reconstructed sparse coefficients obtained in the compression and reconstruction process.
  • the inherent sparsity of the training weight parameters deeply excavated by the compressed sensing reconstruction is integrated into the first loss function to obtain the second loss function required for training and reconstructing the sparse coefficients.
  • the present application can train the reconstructed sparse coefficients without changing the sparsity of the reconstructed sparse coefficients in the transform domain, so as to fully exploit the network weight parameters as much as possible. sparseness.
  • the retraining weight parameter is a weight parameter obtained by jointly training the training weight parameter and the reconstructed sparse coefficient.
  • the computer in the process of iteratively training the reconstructed sparse coefficients and the training weight parameters synchronously through the second loss function, the computer also obtains the test accuracy rate reflecting the current network performance in real time.
  • the computer detects that the current test accuracy rate reaches a certain standard, it can end the current iterative training process for reconstructing the sparse coefficients.
  • the criterion for determining the end of the training may be that the current test accuracy is not less than a certain preset standard value, or the error between the current test accuracy rate and a certain preset standard value is within a certain threshold, etc.
  • the target sparse coefficients can be obtained. Through this target sparse coefficient, the compressed neural network model can be mapped in the transform domain, that is, obtaining the target sparse coefficient is equivalent to completing the compression of the neural network model.
  • the original weight parameters of the neural network model to be compressed by acquiring the original weight parameters of the neural network model to be compressed, and based on the preset first loss function and the basis of the preset transform domain, the original weight parameters are iteratively trained in the original domain to obtain the training weight parameters, and the observation matrix corresponding to the training weight parameter, wherein the first loss function is jointly designed according to the neural network modeling idea and the compressed sensing modeling idea; according to the basis of the transform domain and the observation matrix performing compressed sensing reconstruction to obtain a reconstructed sparsity coefficient representing the sparsity of the training weight parameter in the transform domain, and determining the fixed sparsity of the training weight parameter in the transform domain; integrating the fixed sparsity into the first loss
  • the function obtains a second loss function, and based on the second loss function, iteratively trains the reconstructed sparse coefficient and the training weight parameter to obtain a retraining weight parameter, and a parameter representing the sparsity of the retraining weight parameter in the transform domain.
  • the present application integrates standard compressed sensing constraint modeling into a conventional loss function, designs a first loss function based on this, and uses the first loss function to train the original weight parameters, so that the compressed sensing
  • the modeling process is integrated into the network training process; the reconstructed sparse coefficients and fixed sparse coefficients are obtained by compressive sensing reconstruction based on the basis of the transform domain and the observation matrix obtained by training the original weight parameters, so that the training weights can be fully mined in a transform domain.
  • the inherent sparsity of the parameters by using the inherent sparsity to train the reconstructed sparsity coefficients, the network training process and the network compression process are combined, and the sparsity of the weight parameters is fully trained and mined as much as possible, and finally the neural network model is fully implemented.
  • the purpose of compression is to solve the technical problem of low compression performance of the existing network compression method.
  • step S10 includes:
  • the current iterative training process is ended, and the training weight parameter and the observation matrix corresponding to the training weight parameter are obtained.
  • the first real-time accuracy rate is the performance of the neural network on the test data set during the iterative training of the original weight parameters, which is a measure of the actual performance of the current network.
  • the preset first reference accuracy rate is a critical value used for judging whether the first real-time accuracy rate meets the iteration end criterion, and can be set as the test accuracy rate of the uncompressed neural network after the training ends.
  • the definition of the observation matrix is the observation value of the high-dimensional signal in the low-dimensional space.
  • the computer obtains the original weight parameters of the current neural network model to be compressed, and starts the iterative process of the original weight parameters when it is detected that the test accuracy rate reflecting the initial network actual performance is lower than the preset first reference accuracy rate.
  • the computer creates initialization parameters, and synchronously trains the original weight parameters and the initialization parameters through the first loss function.
  • the first loss function is L 1
  • the original weight parameter is W
  • the initialization parameter is s
  • the network learning rate is ⁇
  • the computer In the process of iteratively training the original weight parameters and the initialization parameters, the computer also obtains the first real-time accuracy rate in real time, and compares it with the preset first reference accuracy rate, and detects that the first real-time accuracy rate is not less than When the first reference accuracy rate is preset, it is determined that the training of the original weight parameters is currently completed, the current iterative training process is stopped, and the training weight parameters after training the original weight parameters and the observation matrix corresponding to the training weight parameters are obtained.
  • the steps of determining that the first real-time test accuracy rate is greater than or equal to the preset first benchmark accuracy rate, ending the current iterative training process, and obtaining the training weight parameters and the observation matrix corresponding to the training weight parameters include:
  • the standard orthogonal basis is used as the basis of the above-mentioned transform domain.
  • the above-mentioned iterative training process for the original weight parameters is specifically a stochastic gradient descent training process.
  • the computer obtains the original weight parameters, creates the initialization parameters, and then performs stochastic gradient descent training on the original weight parameters in the original domain according to the first loss function and the standard orthonormal basis, and induces training of the initialization parameters synchronously in the transform domain.
  • the computer obtains the first real-time accuracy rate reflecting the actual performance of the current network during the training process in real time, and when it is detected that it is not less than the first benchmark accuracy rate, it is determined that the training process for the original weight parameters can be ended, and the training weight parameters and observations are obtained. matrix.
  • the initialization parameters are synchronously trained as the original sparse coefficients corresponding to the original weight parameters in the transform domain.
  • the expression of the first loss function is:
  • ⁇ l(f(x,W),y) represents the expression of the conventional loss function
  • ⁇ and ⁇ represent the preset multiplication factor
  • W represents the original weight parameter
  • ⁇ represents the basis of the transform domain
  • s represents the For the original sparse coefficients
  • the subscript 2 represents using a 2-norm to constrain the relationship between the training weight parameter W and the original sparse coefficient s
  • the subscript 1 represents using a 1-norm to constrain the original sparse coefficient s.
  • the derivation process of the first loss function is described.
  • the original input signal is the K-sparseness characterization coefficient of the original signal.
  • the matrix ⁇ can be stretched into the space
  • ⁇ is a standard orthonormal basis
  • z ⁇ ⁇ ⁇ s.
  • the expression can be written in the form without constraints, such as formula 1:
  • the loss function can be defined as formula 2:
  • Equation 1 Standard compression modeling can be turned into Equation 3:
  • Equation 4 When ⁇ ⁇ ⁇ satisfies the finite isometric property (RIP, Restricted Isometric Property), the above compressed sensing problem can be solved.
  • the matrix [Phi] is initialized when the matrix is too standard distribution, ⁇ ⁇ also fixed to the case presented in Equation 3 ⁇ ⁇ , yields Equation 4:
  • formula 4 is integrated into the conventional neural network modeling process, and the expression of the new loss function L new can be obtained (formula 5):
  • the sparse representation is performed in the domain, and the original sparse coefficients are constrained by the 1-norm to make the original sparse coefficients more sparse during training.
  • step S20 includes:
  • the compressed sensing reconstruction method is specifically limited to the SP compression reconstruction method.
  • the computer uses the preset sampling matrix, the basis of the transform domain, and the observation matrix to perform SP compressed sensing reconstruction, obtains the reconstructed sparse coefficient, and determines the fixed sparsity of the training weight parameter in the transform domain.
  • the iteration end timing is determined by the comparison result between the first real-time test accuracy rate in the training process and the preset first benchmark accuracy rate, so that the iterative end judgment for the original weight parameters is simple and feasible; by constraining the standard compressed sensing
  • the modeling is integrated into the conventional loss function, and the first loss function is designed based on this, and the original weight parameters are trained by the first loss function, so that the compressed sensing modeling process can be integrated into the network training process; by using 2 -
  • the norm constrains the relationship between the training weight parameters and the original sparse coefficients to induce the original sparse coefficients of the training weight parameters in the transformed domain and the distribution of the values of the training weight parameters in the original domain to facilitate the sparse representation of the original sparse coefficients in the transformed domain , constrain the original sparse coefficients with a 1-norm to make the original sparse coefficients more sparse during training.
  • step S30 includes:
  • the second real-time test accuracy rate is greater than or equal to the preset second reference accuracy rate
  • end the current iterative training process for the reconstructed sparse coefficients obtain retraining weight parameters, and characterize the retraining weight parameters
  • the target sparse coefficient of sparsity in the transform domain, and the compressed neural network model is mapped in the transform domain according to the target sparse coefficient.
  • the second real-time accuracy rate is the performance of the neural network on the test data set during the iterative training of the reconstructed sparse coefficients, which is a measure of the actual performance of the current network.
  • the preset second reference accuracy rate is a critical value for judging whether the second real-time accuracy rate meets the iteration end criterion, and can be set as the test accuracy rate of the uncompressed neural network after the training ends.
  • the second reference accuracy rate may be set to be the same as the first reference accuracy rate, or may be set to be different, which is not specifically limited in this embodiment.
  • the second loss function is integrated with a fixed sparsity on the basis of the first loss function.
  • the computer obtains the reconstruction sparse coefficient and the training weight parameter, and when it is detected that the test accuracy rate reflecting the actual performance of the initial network is lower than the preset second reference accuracy rate, the reconstruction sparse coefficient and the training weight parameter are calculated according to the second loss function. Perform iterative training. Let the second loss function be L 2 , and the training weight parameter is The reconstructed sparse coefficient is The network learning rate is ⁇ , then the actual iterative formula can be expressed as:
  • the computer also acquires the second real-time accuracy rate in real time, and compares it with the preset second reference accuracy rate.
  • the computer may determine that the training of the reconstructed sparse coefficients has been completed, stop the current iterative training process, and obtain the trained target sparse coefficients of the reconstructed sparse coefficients.
  • the target sparse coefficients can well represent the weight parameters after training in the transform domain, and at the same time map the compressed neural network model in the transform domain according to the target sparse coefficients.
  • the reconstructed sparse coefficient is iteratively trained, and the current value of the reconstructed sparse coefficient during the iterative training process is obtained.
  • the steps of the second real-time accuracy rate corresponding to the network performance include:
  • the second loss function stochastic gradient descent training is performed on the reconstructed sparse coefficient and the training weight parameter under the fixed sparsity, and in the iterative training process of the reconstructed sparse coefficient, the difference between the current network and the current network is obtained.
  • the second real-time accuracy rate corresponding to the performance.
  • the above-mentioned iterative training process for reconstructing sparse coefficients and training weight parameters is specifically defined as a stochastic gradient descent training process.
  • the computer obtains the reconstructed sparse coefficients and training weight parameters, and performs iterative training on the reconstructed sparse coefficients and training weight parameters according to the second loss function when it is detected that the test accuracy rate reflecting the actual performance of the initial network is less than the preset second reference accuracy rate .
  • the computer also acquires the second real-time accuracy rate in real time, and compares it with the preset second reference accuracy rate.
  • the expression of the second loss function is:
  • the sparsity is fixed Specifically:
  • subscript 2 indicates that the 2-norm is used to constrain the relationship between the training weight parameters and the reconstructed sparse coefficients, so that the distribution of the values of the training weight parameters in the original domain is conducive to the reconstruction of the sparse coefficients in the transformed domain. Perform sparse representation.
  • the subscript 1 indicates that the reconstructed sparse coefficients are constrained by the 1-norm
  • the iterative end timing is determined by the comparison result between the second real-time test accuracy rate in the training process and the preset second reference accuracy rate, so that the iterative end judgment of the reconstructed sparse coefficient is simple and feasible; by integrating the fixed sparsity into As soon as the first loss function obtains the second loss function, the reconstructed sparse coefficients are trained according to the second loss function, so that the reconstructed sparse coefficients are trained without changing the sparsity of the reconstructed sparse coefficients in the transform domain, thereby realizing Fully exploit the sparsity of the network weight parameters as much as possible; constrain the relationship between the training weight parameters and the reconstructed sparse coefficients by using the 2-norm, so that the distribution of the values of the training weight parameters in the original domain is conducive to the reconstruction of the sparse coefficients Sparse representation in the transform domain.
  • the present application also provides a compression sensing-based neural network model compression device.
  • the compressed sensing-based neural network model compression apparatus includes a processor, a memory, and computer-readable instructions stored on the memory and executable on the processor, wherein the computer-readable instructions are processed by the When the device is executed, it implements the steps of the compressed sensing-based neural network model compression method described above.
  • the present application also provides a computer-readable storage medium.
  • Computer-readable instructions are stored on the computer-readable storage medium of the present application, and when the computer-readable instructions are executed by the processor, implement the steps of the compression sensing-based neural network model compression method described above.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

本申请公开了一种基于压缩感知的神经网络模型压缩方法、设备及计算机可读存储介质，该方法通过将标准的压缩感知约束建模融入到常规的损失函数中，并基于此设计出第一损失函数，利用第一损失函数对原始权重参数进行训练，使得能够将压缩感知建模过程融入到网络训练过程中；通过基于变换域的基以及对原始权重参数训练所得的观测矩阵进行压缩感知重建，得到重建稀疏系数与固定稀疏系数，使得能够在一个变换域内充分挖掘训练权重参数的固有稀疏度；通过利用固有稀疏度对重建稀疏系数进行训练，使得将网络训练过程与网络压缩过程联合起来，尽可能地充分训练挖掘权重参数的稀疏性，最终实现对神经网络模型进行充分压缩的目的。

Description

基于压缩感知的神经网络模型压缩方法、设备及存储介质 技术领域
本申请涉及机器学习技术领域，尤其涉及一种基于压缩感知的神经网络模型压缩方法、设备及计算机可读存储介质。
背景技术
随着机器学习技术的快速发展，深度神经网络在计算机视觉、自然语言处理等众多领域取得重大突破，但是这些网络在应用过程中往往耗费需要巨大的计算开销和参数存储需求，导致其无法部署于一些资源有限的设备上。如何在保证网络性能的前提下，有效减少参数存储需求和计算复杂度成为目前急需研究的课题。目前已有的网络压缩模型方案主要通过是在权重参数的原始时域空域内进行一些处理，但是这种着眼于分析原始时空域的网络压缩方式，并未充分挖掘出权重参数的固有稀疏度，因此对网络的压缩程度也就不够彻底，从而导致了现有的网络压缩方式的压缩性能低下的问题。
发明内容
本申请的主要目的在于提供一种基于压缩感知的神经网络模型压缩方法，旨在解决现有的网络压缩方式的压缩性能低下的技术问题。
为实现上述目的，本申请提供一种基于压缩感知的神经网络模型压缩方法，所述基于压缩感知的神经网络模型压缩方法包括：
获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所 述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
此外，为实现上述目的，本申请还提供一种基于压缩感知的神经网络模型压缩装置，所述基于压缩感知的神经网络模型压缩装置包括：
权重参数训练模块，用于获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
压缩感知重建模块，用于根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
稀疏系数训练模块，用于将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
可选地，所述权重参数训练模块包括：
第一迭代训练单元，用于获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；以及，
第一结束判定单元，用于确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵。
可选地，所述获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率的步骤包括：
获取所述原始权重参数，将标准正交基作为所述变换域的基，并创建初始化参数；以及，
根据所述第一损失函数与所述标准正交基，在原始域内对所述原始权重参数进行随机梯度下降训练，在变换域内同步诱导训练所述初始化参数，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
所述确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前随机梯度下降训练过程，将所述初始化参数训练为所述原始权重参数在变换域内对应的原始稀疏系数，并得到所述训练权重参数与所述观测矩阵。
可选地，所述第一损失函数的表达式为：
Σl(f(x,W),y)+λ||W-ψs|| 2+μ||s|| 1，
其中，Σl(f(x,W),y)表示常规损失函数的表达式，λ和μ表示预设乘积因子，W表示所述原始权重参数，ψ表示所述变换域的基，s表示所述原始稀疏系数，下标2表示采用2-范数约束所述训练权重参数W与所述原始稀疏系数s之间的关系，下标1表示采用1-范数约束所述原始稀疏系数s。
可选地，所述压缩感知重建模块包括：
使用预设采样矩阵、所述变换域的基与所述观测矩阵，进行基于子空间追踪方式的压缩感知重建，得到所述重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度。
可选地，所述稀疏系数训练模块包括：
第二迭代训练单元，用于将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率；以及，
第二结束判定单元，用于在检测到所述第二实时测试准确率大于或等于预设第二基准准确率时，结束当前对所述重建稀疏系数的迭代训练过程，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀 疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
可选地，所述基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率的步骤包括：
根据所述第二损失函数，在所述固定稀疏度下对所述重建稀疏系数与所述训练权重参数进行随机梯度下降训练，并获取对所述重建疏系数进行迭代训练过程中，与当前网络性能所对应的第二实时准确率。
可选地，所述第二损失函数的表达式为：
Figure PCTCN2020099752-appb-000001
其中，
Figure PCTCN2020099752-appb-000002
表示常规损失函数的表达式，λ和μ表示预设乘积因子，
Figure PCTCN2020099752-appb-000003
表示所述训练权重参数，ψ表示所述变换域的基，
Figure PCTCN2020099752-appb-000004
表示所述重建稀疏系数，下标2表示采用2-范数约束所述训练权重参数
Figure PCTCN2020099752-appb-000005
与所述重建稀疏系数
Figure PCTCN2020099752-appb-000006
之间的关系，下标1表示采用1-范数约束所述重建稀疏系数
Figure PCTCN2020099752-appb-000007
此外，为实现上述目的，本申请还提供一种基于压缩感知的神经网络模型压缩设备，所述基于压缩感知的神经网络模型压缩设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机可读指令，所述计算机可读指令被所述处理器执行时实现如上述的基于压缩感知的神经网络模型压缩方法的步骤。
此外，为实现上述目的，本申请还提供一种计算机可读存储介质，所述计算机可读存储介质上存储有计算机可读指令，所述计算机可读指令被处理器执行时实现如上述的基于压缩感知的神经网络模型压缩方法的步骤。
本申请提供一种基于压缩感知的神经网络模型压缩方法、设备及计算机可读存储介质。所述基于压缩感知的神经网络模型压缩方法通过获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失 函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。通过上述方式，本申请通过将标准的压缩感知约束建模融入到常规的损失函数中，并基于此设计出第一损失函数，利用第一损失函数对原始权重参数进行训练，使得能够将压缩感知建模过程融入到网络训练过程中；通过基于变换域的基以及对原始权重参数训练所得的观测矩阵进行压缩感知重建，得到重建稀疏系数与固定稀疏系数，使得能够在一个变换域内充分挖掘训练权重参数的固有稀疏度；通过利用固有稀疏度对重建稀疏系数进行训练，使得将网络训练过程与网络压缩过程联合起来，尽可能地充分训练挖掘权重参数的稀疏性，最终实现对神经网络模型进行充分压缩的目的，从而解决了现有的网络压缩方式的压缩性能低下的技术问题。
附图说明
图1是本申请实施例方案涉及的硬件运行环境的基于压缩感知的神经网络模型压缩设备结构示意图；
图2为本申请基于压缩感知的神经网络模型压缩方法第一实施例的流程示意图。
具体实施方式
应当理解，此处所描述的具体实施例仅仅用以解释本申请，并不用于限定本申请。
如图1所示，图1是本申请实施例方案涉及的硬件运行环境的基于压缩感知的神经网络模型压缩设备结构示意图。
本申请实施例基于压缩感知的神经网络模型压缩设备可以是服务器、PC等终端设备。
如图1所示，该基于压缩感知的神经网络模型压缩设备可以包括：处理器1001，例如CPU，通信总线1002，用户接口1003，网络接口1004，存储器1005。其中，通信总线1002用于实现这些组件之间的连接通信。用户接口1003可以包括显示屏(Display)、输入单元比如键盘(Keyboard)，可选的用户接口 1003还可以包括标准的有线接口、无线接口。网络接口1004可选的可以包括标准的有线接口、无线接口(如WI-FI接口)。存储器1005可以是高速RAM存储器，也可以是稳定的存储器(non-volatile memory)，例如磁盘存储器。存储器1005可选的还可以是独立于前述处理器1001的存储设备。
如图1所示，作为一种计算机存储介质的存储器1005中可以包括操作系统、网络通信模块、用户接口模块以及计算机可读指令。
在图1所示的终端中，网络接口1004主要用于连接后台服务器，与后台服务器进行数据通信；用户接口1003主要用于连接客户端(用户端)，与客户端进行数据通信；而处理器1001可以用于调用存储器1005中存储的计算机可读指令，并执行以下操作：
获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
基于上述硬件结构，提出本申请基于压缩感知的神经网络模型压缩方法的各个实施例。
随着机器学习技术的快速发展，深度神经网络在计算机视觉、自然语言处理等众多领域取得重大突破，但是这些网络在应用过程中往往耗费需要巨大的计算开销和参数存储需求，导致其无法部署于一些资源有限的设备上。如何在保证网络性能的前提下，有效减少参数存储需求和计算复杂度成为目前急需研究的课题。目前已有的网络压缩模型方案主要通过是在权重参数的原始时域空域内进行一些处理，但是这种着眼于分析原始时空域的网络压缩方式，并未充分挖掘出权重参数的固有稀疏度，因此对网络的压缩程度也就 不够彻底，从而导致了现有的网络压缩方式的压缩性能低下的问题。
为解决上述问题，本申请提供一种基于压缩感知的神经网络模型压缩方法，即将标准的压缩感知约束建模融入到常规的损失函数中，并基于此设计出第一损失函数，利用第一损失函数对原始权重参数进行训练，使得能够将压缩感知建模过程融入到网络训练过程中；通过基于变换域的基以及对原始权重参数训练所得的观测矩阵进行压缩感知重建，得到重建稀疏系数与固定稀疏系数，使得能够在一个变换域内充分挖掘训练权重参数的固有稀疏性；通过利用固有稀疏性对重建稀疏系数进行训练，使得将网络训练过程与网络压缩过程联合起来，尽可能地充分训练挖掘权重参数的稀疏性，最终实现对神经网络模型进行充分压缩的目的，从而解决了现有的网络压缩方式的压缩性能低下的技术问题。
参照图2，图2为基于压缩感知的神经网络模型压缩方法第一实施例的流程示意图。本申请第一实施例提供一种基于压缩感知的神经网络模型压缩方法，所述基于压缩感知的神经网络模型压缩方法包括以下步骤：
步骤S10，获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
在本实施例中，原始权重参数为当前待压缩神经网络模型的还未经训练的权重参数。训练权重参数为原始的权重参数在原始域内进行训练后的权重参数。预设第一损失函数为用于训练待检测的神经网络模型所对应的原始权重参数的损失函数，是基于常规的神经网络建模思想与压缩感知建模思想联合设计的。其中，压缩感知，也被称为压缩采样，稀疏采样，压缩传感。它作为一个新的采样理论，通过开发信号的稀疏特性，在远小于奈奎斯特采样率的条件下，用随机采样获取信号的离散样本，然后通过非线性重建算法完美的重建信号。
具体地，计算机在获取到当前待压缩的神经网络模型的原始权重参数时，将此原始权重参数作为压缩感知建模过程中的输入信号，同时还需输入训练样本的真实值、变换域空间的基、网络学习率以及乘积因子等。在对原始权重参数进行训练之前，还可对其进行随机初始化，并创建初始化参数，以在 训练过程中将初始化参数诱导为原始权重参数在变换域内的稀疏系数表征。基于第一损失函数对原始权重参数与初始化参数同步进行迭代训练，并在训练过程中实时获取反映当前网络性能的测试准确率。计算机在检测到当前的测试准确率达到一定标准时，即可结束当前对于原始权重参数的迭代训练过程。其中，训练结束判定标准可为当前的测试准确率不小于某一预设标准值，或是当前的测试准确率与某一预设标准值之间的误差在某一阈值内等，本实施例不做具体限定。而原始权重参数在训练过程中进行参数更新后成为训练权重参数，且此时的初始化参数也与原始权重参数同步训练为表征原始权重参数在变换域内稀疏度的原始稀疏系数，并可得到与训练权重参数对应的观测矩阵。
步骤S20，根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；
在本实施例中，本申请通过引入压缩感知原理，从而能够在变换域内充分挖掘训练权重参数的固有稀疏度。在对原始权重参数进行迭代训练后，就开始进行压缩感知处理。需要说明的是，在进行压缩感知重建过程所需要的参数包括：变换域的基、观测矩阵与采样矩阵。采样矩阵的作用是将高维信号投影到低维空间。
通过利用变换域的基、观测矩阵与采样矩阵进行压缩感知重建，并生成一个标志位来固定训练权重参数在变换域内的固定稀疏度。具体的压缩感知重建方式可选用子空间追踪(SP，subspace pursuit)方式。
步骤S30，将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
在本实施例中，第二损失函数为用于训练压缩重建过程所得的重建稀疏系数的损失函数。将由压缩感知重建所深度挖掘的训练权重参数的固有稀疏度整合进第一损失函数，得到训练重建稀疏系数所需的第二损失函数。通过使用第二损失函数同步训练重建稀疏系数与训练权重参数，使得本申请能够 在不改变变换域内重建稀疏系数的稀疏度的前提下，对重建稀疏系数进行训练，以尽可能充分挖掘网络权重参数的稀疏性。重训练权重参数为将训练权重参数与重建稀疏系数共同进行训练后所得到的权重参数。
具体地，计算机在通过第二损失函数对重建稀疏系数与训练权重参数同步进行迭代训练的过程中，还会实时获取反映当前网络性能的测试准确率。计算机在检测到当前的测试准确率达到一定标准时，即可结束当前对于重建稀疏系数的迭代训练过程。其中，训练结束判定标准可为当前的测试准确率不小于某一预设标准值，或是当前的测试准确率与某一预设标准值之间的误差在某一阈值内等，本实施例不做具体限定。计算机在结束对重建稀疏系数的迭代训练过程后，即可得到目标稀疏系数。通过这一目标稀疏系数，可在变换域内映射出压缩后的神经网络模型，也即是得到此目标稀疏系数相当于完成了对神经网络模型的压缩。
在本实施例中，通过获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。通过上述方式，本申请通过将标准的压缩感知约束建模融入到常规的损失函数中，并基于此设计出第一损失函数，利用第一损失函数对原始权重参数进行训练，使得能够将压缩感知建模过程融入到网络训练过程中；通过基于变换域的基以及对原始权重参数训练所得的观测矩阵进行压缩感知重建，得到重建稀疏系数与固定稀疏系数，使得能够在一个变换域内充分挖掘训练权重参数的固有稀疏度；通过利用固有稀疏度对重建稀疏系数进行训练，使得将网络训练过程与网络压缩过程联合起来，尽可能地充分训练挖掘权重参数的稀疏性，最终实现对神经网络模型进行充分压 缩的目的，从而解决了现有的网络压缩方式的压缩性能低下的技术问题。
进一步地，图中未示的，基于上述图2所示的第一实施例，提出本申请基于压缩感知的神经网络模型压缩方法的第二实施例。在本实施例中，步骤S10包括：
获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵。
在本实施例中，第一实时准确率为在对原始权重参数进行迭代训练的过程中神经网络在测试数据集上的表现，是对当前网络实际性能的衡量。预设第一基准准确率为用于判断第一实时准确率是否符合迭代结束标准的临界值，可设定为未压缩的神经网络在训练结束后的测试准确率。观测矩阵的定义是高维信号在低维空间内的观测值。具体地，计算机获取当前待压缩的神经网络模型的原始权重参数，并在检测到反映初始的网络实际性能的测试准确率小于预设第一基准准确率时，开始对原始权重参数的迭代过程。计算机创建初始化参数，通过第一损失函数同步训练原始权重参数与初始化参数。设第一损失函数为L 1，原始权重参数为W，初始化参数为s，网络学习率为η，则实际的迭代公式可表示为：
Figure PCTCN2020099752-appb-000008
计算机在对原始权重参数与初始化参数进行迭代训练的过程中，还实时获取第一实时准确率，并将其与预设第一基准准确率进行比较，并在检测到第一实时准确率不小于预设第一基准准确率时，判定当前已完成对原始权重参数的训练，停止当前迭代训练过程，得到原始权重参数经训练后的训练权重参数，以及与训练权重参数对应的观测矩阵。
进一步地，在本实施例中，所述获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当 前网络性能所对应的第一实时准确率的步骤包括：
获取所述原始权重参数，将标准正交基作为所述变换域的基，并创建初始化参数；
根据所述第一损失函数与所述标准正交基，在原始域内对所述原始权重参数进行随机梯度下降训练，在变换域内同步诱导训练所述初始化参数，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
所述确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前随机梯度下降训练过程，将所述初始化参数训练为所述原始权重参数在变换域内对应的原始稀疏系数，并得到所述训练权重参数与所述观测矩阵。
在本实施例中，为便于计算，将标准正交基作为上述变换域的基。限定了上述对于原始权重参数的迭代训练过程具体为随机梯度下降训练过程。具体地，计算机获取原始权重参数，并创建初始化参数，再根据第一损失函数与标准正交基，在原始域内对原始权重参数进行随机梯度下降训练，并在变换域内同步诱导训练初始化参数。计算机实时获取训练过程中反映当前网络实际性能的第一实时准确率，并在检测到其不小于第一基准准确率时，判定可结束对原始权重参数的训练过程，并得到训练权重参数与观测矩阵。此时初始化参数被同步训练为原始权重参数在变换域内所对应的原始稀疏系数。
进一步地，在本实施例中，所述第一损失函数的表达式为：
Σl(f(x,W),y)+λ||W-ψs|| 2+μ||s|| 1，
其中，Σl(f(x,W),y)表示常规损失函数的表达式，λ和μ表示预设乘积因子，W表示所述原始权重参数，ψ表示所述变换域的基，s表示所述原始稀疏系数，下标2表示采用2-范数约束所述训练权重参数W与所述原始稀疏系数s之间的关系，下标1表示采用1-范数约束所述原始稀疏系数s。
在本实施例中，对第一损失函数的推导过程进行说明。首先，假设
Figure PCTCN2020099752-appb-000009
是原始输入信号，
Figure PCTCN2020099752-appb-000010
是原始信号的K-稀疏度表征系数。而矩阵φ可以张成空间
Figure PCTCN2020099752-appb-000011
φ中的每一行是对于x的一个测量函数，可以得到：z＝φ Ωx。其 中，
Figure PCTCN2020099752-appb-000012
是采样矩阵，其是从矩阵φ中随机抽取M行所得，而z为观测矩阵。一般情况下原始信号在原始域内部稀疏，假设其在
Figure PCTCN2020099752-appb-000013
表征的空间内稀疏，则x可以表示为x＝ψs。当ψ是标准正交基时，可得s＝ψ Tx；z＝φ Ωψs。为了从观测矩阵z中恢复出未知的稀疏系数s，一般可以通过一定的范数约束实现，将表达式写为没有约束条件的形式如公式1：
Figure PCTCN2020099752-appb-000014
而由本领域常识可知，常规的神经网络中，损失函数可定义为公式2：
L＝Σl(f(x,W),y)
其中，x，y分别为再训练过程中样本的输入和真实值，f(x,W)为网络预测输出，l是对每一个训练样本的损失函数。为了将公式1中的压缩感知建模与公式2中的常规神经网络损失函数直接建立联系，本申请将网络权重参数W作为压缩感知建模过程中的输入信号，由此即可得到观测矩阵W～的表达式为：
Figure PCTCN2020099752-appb-000015
标准的压缩建模可以变成公式3：
Figure PCTCN2020099752-appb-000016
当{φ Ωψ}满足有限等距性质(RIP，Restricted Isometry Property)时，上述压缩感知问题才可解。将矩阵φ初始化为标准正太分布的矩阵时，φ Ω也就固定，此时即可将公式3中的φ Ω提出，得到公式4：
Figure PCTCN2020099752-appb-000017
在本申请中将公式4融入常规的神经网络建模过程中，即可得到新的损失函数L new的表达式(公式5)：
L new＝Σl(f(x,W),y)+λ||W-ψs|| p1+μ||s|| p2
在实际优化过程中，对于公式5，本申请选取p1＝2,p2＝1，即可得到第一损失函数。意为用2-范数约束训练权重参数与原始稀疏系数之间的关系，以诱导训练权重参数在变换域内的原始稀疏系数以及使训练权重参数在原始域内的值的分布利于原始稀疏系数在变换域内进行稀疏表征，用1-范数约束原始稀疏系数，以使得原始稀疏系数在训练过程中更稀疏。
进一步地，在本实施例中，步骤S20包括：
使用预设采样矩阵、所述变换域的基与所述观测矩阵，进行基于子空间追踪方式的压缩感知重建，得到所述重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度。
在本实施例中，将压缩感知重建的方式具体限定为SP压缩重建方式。具体地，计算机使用预设采样矩阵、变换域的基以及观测矩阵，进行SP压缩感知重建，得到重建稀疏系数，并确定训练权重参数在变换域内的固定稀疏度。
进一步地，通过训练过程中的第一实时测试准确率与预设第一基准准确率的比较结果判定迭代结束时机，使得对于原始权重参数的迭代结束判断简单易行；通过将标准的压缩感知约束建模融入到常规的损失函数中，并基于此设计出第一损失函数，利用第一损失函数对原始权重参数进行训练，使得能够将压缩感知建模过程融入到网络训练过程中；通过用2-范数约束训练权重参数与原始稀疏系数之间的关系，以诱导训练权重参数在变换域内的原始稀疏系数以及使训练权重参数在原始域内的值的分布利于原始稀疏系数在变换域内进行稀疏表征，用1-范数约束原始稀疏系数，以使得原始稀疏系数在训练过程中更稀疏。
进一步地，图中未示的，基于上述图2所示的第一实施例，提出本申请基于压缩感知的神经网络模型压缩方法的第三实施例。在本实施例中，步骤S30包括：
将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率；
在检测到所述第二实时测试准确率大于或等于预设第二基准准确率时，结束当前对所述重建稀疏系数的迭代训练过程，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
在本实施例中，第二实时准确率为在对重建稀疏系数进行迭代训练的过程中神经网络在测试数据集上的表现，是对当前网络实际性能的衡量。预设第二基准准确率为用于判断第二实时准确率是否符合迭代结束标准的临界值，可设定为未压缩的神经网络在训练结束后的测试准确率。第二基准准确率可设置与第一基准准确率相同，也可设置为不同，本实施例不做具体限定。第二损失函数是在第一损失函数的基础上由固定稀疏度整合而来。
具体地，计算机获取重建稀疏系数与训练权重参数，并在检测到反映初 始的网络实际性能的测试准确率小于预设第二基准准确率时，根据第二损失函数对重建稀疏系数与训练权重参数进行迭代训练。设第二损失函数为L 2，训练权重参数为
Figure PCTCN2020099752-appb-000018
重建稀疏系数为
Figure PCTCN2020099752-appb-000019
网络学习率为η，则实际的迭代公式可表示为：
Figure PCTCN2020099752-appb-000020
计算机在对训练权重参数与重建稀疏系数进行迭代训练的过程中，还实时获取第二实时准确率，并将其与预设第二基准准确率进行比较。在检测到第一实时准确率不小于预设第一基准准确率时，计算机可判定当前已完成对重建稀疏系数的训练，停止当前迭代训练过程，得到重建稀疏系数经训练后的目标稀疏系数。该目标稀疏系数可以在变换域内很好地表征训练后的权重参数，同时根据目标稀疏系数在变换域内映射出压缩后的神经网络模型。
进一步地，在本实施例中，所述基于所述第二损失函数与所述训练权重参数，对所述重建稀疏系数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率的步骤包括：
根据所述第二损失函数，在所述固定稀疏度下对所述重建稀疏系数与所述训练权重参数进行随机梯度下降训练，并获取对所述重建疏系数进行迭代训练过程中，与当前网络性能所对应的第二实时准确率。
在本实施例中，限定了上述对于重建稀疏系数与训练权重参数的迭代训练过程具体为随机梯度下降训练过程。计算机获取重建稀疏系数与训练权重参数，并在检测到反映初始的网络实际性能的测试准确率小于预设第二基准准确率时，根据第二损失函数对重建稀疏系数与训练权重参数进行迭代训练。计算机在对训练权重参数与重建稀疏系数进行迭代训练的过程中，还实时获取第二实时准确率，并将其与预设第二基准准确率进行比较。
进一步地，在本实施例中，所述第二损失函数的表达式为：
Figure PCTCN2020099752-appb-000021
其中，
Figure PCTCN2020099752-appb-000022
表示常规损失函数的表达式，λ和μ表示预设乘积因子，
Figure PCTCN2020099752-appb-000023
表示所述训练权重参数，ψ表示所述变换域的基，
Figure PCTCN2020099752-appb-000024
表示所述重建稀疏系数，下标2表示采用2-范数约束所述训练权重参数
Figure PCTCN2020099752-appb-000025
与所述重建稀疏系数
Figure PCTCN2020099752-appb-000026
之间的关系，下标1表示采用1-范数约束所述重建稀疏系数
Figure PCTCN2020099752-appb-000027
在本实施例中，固定稀疏度
Figure PCTCN2020099752-appb-000028
具体为：
Figure PCTCN2020099752-appb-000029
需要说明的是，下标2表示采用2-范数约束所述训练权重参数与所述重建稀疏系数之间的关系，以使训练权重参数在原始域内的值的分布利于重建稀疏系数在变换域内进行稀疏表征。下标1表示采用1-范数约束所述重建稀疏系数
Figure PCTCN2020099752-appb-000030
进一步地，通过训练过程中的第二实时测试准确率与预设第二基准准确率的比较结果判定迭代结束时机，使得对于重建稀疏系数的迭代结束判断简单易行；通过将固定稀疏度整合进第一损失函数一得到第二损失函数，并根据第二损失函数对重建稀疏系数进行训练，使得在不改变变换域内的重建稀疏系数的稀疏度前提下，对重建稀疏系数进行训练，进而实现尽可能充分挖掘网络权重参数的稀疏性；通过采用2-范数约束所述训练权重参数与所述重建稀疏系数之间的关系，以使训练权重参数在原始域内的值的分布利于重建稀疏系数在变换域内进行稀疏表征。
本申请还提供一种基于压缩感知的神经网络模型压缩设备。
所述种基于压缩感知的神经网络模型压缩设备包括处理器、存储器及存储在所述存储器上并可在所述处理器上运行的计算机可读指令，其中所述计算机可读指令被所述处理器执行时，实现如上所述的基于压缩感知的神经网络模型压缩方法的步骤。
其中，所述计算机可读指令被执行时所实现的方法可参照本申请基于压缩感知的神经网络模型压缩方法的各个实施例，此处不再赘述。
本申请还提供一种计算机可读存储介质。
本申请计算机可读存储介质上存储有计算机可读指令，所述计算机可读指令被处理器执行时实现如上所述的基于压缩感知的神经网络模型压缩方法的步骤。
其中，所述计算机可读指令被执行时所实现的方法可参照本申请基于压缩感知的神经网络模型压缩方法各个实施例，此处不再赘述。

Claims (20)

  1. 一种基于压缩感知的神经网络模型压缩方法，其中，所述基于压缩感知的神经网络模型压缩方法包括：
    获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
    根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
    将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
  2. 如权利要求1所述的基于压缩感知的神经网络模型压缩方法，其中，所述获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；以及，
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵。
  3. 如权利要求2所述的基于压缩感知的神经网络模型压缩方法，其中，所述获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预 设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率的步骤包括：
    获取所述原始权重参数，将标准正交基作为所述变换域的基，并创建初始化参数；以及，
    根据所述第一损失函数与所述标准正交基，在原始域内对所述原始权重参数进行随机梯度下降训练，在变换域内同步诱导训练所述初始化参数，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
    所述确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前随机梯度下降训练过程，将所述初始化参数训练为所述原始权重参数在变换域内对应的原始稀疏系数，并得到所述训练权重参数与所述观测矩阵。
  4. 如权利要求3所述的基于压缩感知的神经网络模型压缩方法，其中，所述第一损失函数的表达式为：
    ∑l(f(x,W),y)+λ||W-ψs|| 2+μ||s|| 1，
    其中，∑l(f(x,W),y)表示常规损失函数的表达式，λ和μ表示预设乘积因子，W表示所述原始权重参数，ψ表示所述变换域的基，s表示所述原始稀疏系数，下标2表示采用2-范数约束所述训练权重参数W与所述原始稀疏系数s之间的关系，下标1表示采用1-范数约束所述原始稀疏系数s。
  5. 如权利要求1所述的基于压缩感知的神经网络模型压缩方法，其中，所述根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度的步骤包括：
    使用预设采样矩阵、所述变换域的基与所述观测矩阵，进行基于子空间追踪方式的压缩感知重建，得到所述重建稀疏系数，并确定所述训练权重参 数在变换域内的固定稀疏度。
  6. 如权利要求1所述的基于压缩感知的神经网络模型压缩方法，其中，所述将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型的步骤包括：
    将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率；以及，
    在检测到所述第二实时测试准确率大于或等于预设第二基准准确率时，结束当前对所述重建稀疏系数的迭代训练过程，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
  7. 如权利要求6所述的基于压缩感知的神经网络模型压缩方法，其中，所述基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率的步骤包括：
    根据所述第二损失函数，在所述固定稀疏度下对所述重建稀疏系数与所述训练权重参数进行随机梯度下降训练，并获取对所述重建疏系数进行迭代训练过程中，与当前网络性能所对应的第二实时准确率。
  8. 如权利要求1所述的基于压缩感知的神经网络模型压缩方法，其中，所述第二损失函数的表达式为：
    Figure PCTCN2020099752-appb-100001
    其中，
    Figure PCTCN2020099752-appb-100002
    表示常规损失函数的表达式，λ和μ表示预设乘积因子，
    Figure PCTCN2020099752-appb-100003
    表示所述训练权重参数，ψ表示所述变换域的基，
    Figure PCTCN2020099752-appb-100004
    表示所述重建稀疏 系数，下标2表示采用2-范数约束所述训练权重参数
    Figure PCTCN2020099752-appb-100005
    与所述重建稀疏系数
    Figure PCTCN2020099752-appb-100006
    之间的关系，下标1表示采用1-范数约束所述重建稀疏系数
    Figure PCTCN2020099752-appb-100007
  9. 一种基于压缩感知的神经网络模型压缩设备，其中，所述基于压缩感知的神经网络模型压缩设备包括：存储器、处理器及存储在所述存储器上并可在所述处理器上运行的计算机可读指令，所述计算机可读指令被所述处理器执行时实现如下步骤：
    获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
    根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
    将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
  10. 如权利要求9所述的基于压缩感知的神经网络模型压缩设备，其中，所述获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；以及，
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵。
  11. 如权利要求10所述的基于压缩感知的神经网络模型压缩设备，其中，所述获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率的步骤包括：
    获取所述原始权重参数，将标准正交基作为所述变换域的基，并创建初始化参数；以及，
    根据所述第一损失函数与所述标准正交基，在原始域内对所述原始权重参数进行随机梯度下降训练，在变换域内同步诱导训练所述初始化参数，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
    所述确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前随机梯度下降训练过程，将所述初始化参数训练为所述原始权重参数在变换域内对应的原始稀疏系数，并得到所述训练权重参数与所述观测矩阵。
  12. 如权利要求11所述的基于压缩感知的神经网络模型压缩设备，其中，所述第一损失函数的表达式为：
    ∑l(f(x,W),y)+λ||W-ψs|| 2+μ||s|| 1，
    其中，∑l(f(x,W),y)表示常规损失函数的表达式，λ和μ表示预设乘积因子，W表示所述原始权重参数，ψ表示所述变换域的基，s表示所述原始稀疏系数，下标2表示采用2-范数约束所述训练权重参数W与所述原始稀疏系数s之间的关系，下标1表示采用1-范数约束所述原始稀疏系数s。
  13. 如权利要求9所述的基于压缩感知的神经网络模型压缩设备，其中，所述根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数 在变换域内的固定稀疏度的步骤包括：
    使用预设采样矩阵、所述变换域的基与所述观测矩阵，进行基于子空间追踪方式的压缩感知重建，得到所述重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度。
  14. 如权利要求9所述的基于压缩感知的神经网络模型压缩设备，其中，所述将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型的步骤包括：
    将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所述第二损失函数，对所述重建稀疏系数与所述训练权重参数进行迭代训练，并获取对所述重建稀疏系数进行迭代训练过程中与当前网络性能所对应的第二实时准确率；以及，
    在检测到所述第二实时测试准确率大于或等于预设第二基准准确率时，结束当前对所述重建稀疏系数的迭代训练过程，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
  15. 一种计算机可读存储介质，其中，所述计算机可读存储介质上存储有计算机可读指令，所述计算机可读指令被处理器执行时实现如下步骤：
    获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵，其中，所述第一损失函数根据神经网络建模思想与压缩感知建模思想联合设计所得；
    根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度；以及，
    将所述固定稀疏度整合至所述第一损失函数得到第二损失函数，基于所 述第二损失函数对所述重建稀疏系数与所述训练权重参数进行迭代训练，得到重训练权重参数，与表征所述重训练权重参数在变换域内稀疏度的目标稀疏系数，并根据所述目标稀疏系数在变换域内映射出压缩后的神经网络模型。
  16. 如权利要求15所述的计算机可读存储介质，其中，所述获取待压缩神经网络模型的原始权重参数，并基于预设第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，得到训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；以及，
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵。
  17. 如权利要求16所述的计算机可读存储介质，其中，所述获取待压缩神经网络模型的原始权重参数，基于所述第一损失函数与预设变换域的基，在原始域内对所述原始权重参数进行迭代训练，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率的步骤包括：
    获取所述原始权重参数，将标准正交基作为所述变换域的基，并创建初始化参数；以及，
    根据所述第一损失函数与所述标准正交基，在原始域内对所述原始权重参数进行随机梯度下降训练，在变换域内同步诱导训练所述初始化参数，并获取对所述原始权重参数进行迭代训练过程中，与当前网络性能所对应的第一实时准确率；
    所述确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当前迭代训练过程，得到所述训练权重参数、以及与所述训练权重参数对应的观测矩阵的步骤包括：
    确定所述第一实时测试准确率大于或等于预设第一基准准确率，结束当 前随机梯度下降训练过程，将所述初始化参数训练为所述原始权重参数在变换域内对应的原始稀疏系数，并得到所述训练权重参数与所述观测矩阵。
  18. 如权利要求17所述的计算机可读存储介质，其中，所述第一损失函数的表达式为：
    ∑l(f(x,W),y)+λ||W-ψs|| 2+μ||s|| 1，
    其中，∑l(f(x,W),y)表示常规损失函数的表达式，λ和μ表示预设乘积因子，W表示所述原始权重参数，ψ表示所述变换域的基，s表示所述原始稀疏系数，下标2表示采用2-范数约束所述训练权重参数W与所述原始稀疏系数s之间的关系，下标1表示采用1-范数约束所述原始稀疏系数s。
  19. 如权利要求15所述的计算机可读存储介质，其中，所述根据所述变换域的基与所述观测矩阵进行压缩感知重建，得到表征所述训练权重参数在变换域内稀疏度的重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度的步骤包括：
    使用预设采样矩阵、所述变换域的基与所述观测矩阵，进行基于子空间追踪方式的压缩感知重建，得到所述重建稀疏系数，并确定所述训练权重参数在变换域内的固定稀疏度。
  20. 如权利要求15所述的计算机可读存储介质，其中，所述第二损失函数的表达式为：
    Figure PCTCN2020099752-appb-100008
    其中，
    Figure PCTCN2020099752-appb-100009
    表示常规损失函数的表达式，λ和μ表示预设乘积因子，
    Figure PCTCN2020099752-appb-100010
    表示所述训练权重参数，ψ表示所述变换域的基，
    Figure PCTCN2020099752-appb-100011
    表示所述重建稀疏系数，下标2表示采用2-范数约束所述训练权重参数
    Figure PCTCN2020099752-appb-100012
    与所述重建稀疏系数
    Figure PCTCN2020099752-appb-100013
    之间的关系，下标1表示采用1-范数约束所述重建稀疏系数
    Figure PCTCN2020099752-appb-100014
PCT/CN2020/099752 2020-07-01 2020-07-01 基于压缩感知的神经网络模型压缩方法、设备及存储介质 WO2022000373A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN202080001140.4A CN112352249A (zh) 2020-07-01 2020-07-01 基于压缩感知的神经网络模型压缩方法、设备及存储介质
PCT/CN2020/099752 WO2022000373A1 (zh) 2020-07-01 2020-07-01 基于压缩感知的神经网络模型压缩方法、设备及存储介质

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/099752 WO2022000373A1 (zh) 2020-07-01 2020-07-01 基于压缩感知的神经网络模型压缩方法、设备及存储介质

Publications (1)

Publication Number Publication Date
WO2022000373A1 true WO2022000373A1 (zh) 2022-01-06

Family

ID=74427531

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/099752 WO2022000373A1 (zh) 2020-07-01 2020-07-01 基于压缩感知的神经网络模型压缩方法、设备及存储介质

Country Status (2)

Country Link
CN (1) CN112352249A (zh)
WO (1) WO2022000373A1 (zh)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023089063A1 (en) 2021-11-18 2023-05-25 Rockley Photonics Limited Architecture of a photonic integrated circuit (pic) and method for operating the same as well as an optical coupler

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113379867B (zh) * 2021-07-05 2023-09-12 北京大学深圳研究生院 一种基于联合优化采样矩阵的核磁共振图像重建方法
CN113746870B (zh) * 2021-11-05 2022-02-08 山东万网智能科技有限公司 一种物联网设备数据智能传输方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190124346A1 (en) * 2017-10-19 2019-04-25 Arizona Board Of Regents On Behalf Of Arizona State University Real time end-to-end learning system for a high frame rate video compressive sensing network
CN109978142A (zh) * 2019-03-29 2019-07-05 腾讯科技（深圳）有限公司 神经网络模型的压缩方法和装置
CN111106836A (zh) * 2019-10-30 2020-05-05 北京歌尔泰克科技有限公司 图像重构方法和装置
CN111105007A (zh) * 2018-10-26 2020-05-05 中国科学院半导体研究所 用于目标检测的深度卷积神经网络的压缩加速方法

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190124346A1 (en) * 2017-10-19 2019-04-25 Arizona Board Of Regents On Behalf Of Arizona State University Real time end-to-end learning system for a high frame rate video compressive sensing network
CN111105007A (zh) * 2018-10-26 2020-05-05 中国科学院半导体研究所 用于目标检测的深度卷积神经网络的压缩加速方法
CN109978142A (zh) * 2019-03-29 2019-07-05 腾讯科技（深圳）有限公司 神经网络模型的压缩方法和装置
CN111106836A (zh) * 2019-10-30 2020-05-05 北京歌尔泰克科技有限公司 图像重构方法和装置

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2023089063A1 (en) 2021-11-18 2023-05-25 Rockley Photonics Limited Architecture of a photonic integrated circuit (pic) and method for operating the same as well as an optical coupler

Also Published As

Publication number Publication date
CN112352249A (zh) 2021-02-09

Similar Documents

Publication Publication Date Title
WO2022000373A1 (zh) 基于压缩感知的神经网络模型压缩方法、设备及存储介质
WO2020233130A1 (zh) 一种深度神经网络压缩方法及相关设备
CN110880036B (zh) 神经网络压缩方法、装置、计算机设备及存储介质
CN105827250B (zh) 一种基于自适应字典学习的电能质量数据压缩重构方法
Yang et al. Output-only modal identification by compressed sensing: Non-uniform low-rate random sampling
Erichson et al. Randomized CP tensor decomposition
Doostan et al. A non-adapted sparse approximation of PDEs with stochastic inputs
CN103595414B (zh) 一种稀疏采样与信号压缩感知重构方法
De Vito et al. Learning sets with separating kernels
CN107506822B (zh) 一种基于空间融合池化的深度神经网络方法
JP5509488B2 (ja) 形状を認識する方法および形状を認識する方法を実施するシステム
US11729286B2 (en) Feature-based network embedding
Anaraki et al. Compressive k-svd
US20200265321A1 (en) Embedding constrained and unconstrained optimization programs as neural network layers
Székely et al. Extraction and predictability of coherent intraseasonal signals in infrared brightness temperature data
JP6232522B2 (ja) 計算機及びグラフデータ生成方法
CN113674172A (zh) 一种图像处理方法、系统、装置及存储介质
CN111402922B (zh) 基于小样本的音频信号分类方法、装置、设备及存储介质
Li et al. On joint optimization of sensing matrix and sparsifying dictionary for robust compressed sensing systems
WO2022193379A1 (zh) 图像重建模型生成及装置、图像重建方法及装置、设备、介质
WO2021253671A1 (zh) 磁共振电影成像方法、装置、成像设备及存储介质
CN114612671A (zh) 一种多视图子空间聚类方法、装置、设备及存储介质
Chen et al. Physics-enhanced pca for data compression in edge devices
CN109584320A (zh) 一种低相干观测矩阵构造方法
WO2021159772A1 (zh) 语音增强方法及装置、电子设备和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20943581

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20943581

Country of ref document: EP

Kind code of ref document: A1