CN115423087A - Pruning method and device for impulse neural network and electronic equipment - Google Patents
Pruning method and device for impulse neural network and electronic equipment Download PDFInfo
- Publication number
- CN115423087A CN115423087A CN202210867737.8A CN202210867737A CN115423087A CN 115423087 A CN115423087 A CN 115423087A CN 202210867737 A CN202210867737 A CN 202210867737A CN 115423087 A CN115423087 A CN 115423087A
- Authority
- CN
- China
- Prior art keywords
- neural network
- function
- vector
- parameter vector
- pruning
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/77—Processing image or video features in feature spaces; using data integration or data reduction, e.g. principal component analysis [PCA] or independent component analysis [ICA] or self-organising maps [SOM]; Blind source separation
- G06V10/774—Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/40—Scenes; Scene-specific elements in video content
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Multimedia (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Molecular Biology (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a pruning method and device for a pulse neural network and electronic equipment. Wherein, the method comprises the following steps: initializing an initial vector formed by each connection weight in a synapse connection layer to obtain a weight vector; when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to a hidden parameter vector through a predefined derivative function; performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function; based on the target threshold, mapping the hidden parameter vector to the weight vector again by using a soft threshold function; and when the number of pruning training rounds of the pulse neural network model reaches the preset number of rounds, obtaining the trained pulse neural network model. The invention solves the technical problem that the pulse neural network cannot be effectively deployed on the neural morphology calculation chip in the related technology.
Description
Technical Field
The invention relates to the technical field of brain-like intelligence, in particular to a pruning method and device of a pulse neural network and electronic equipment.
Background
Spiking Neural Networks (SNNs) are important learning models in the field of brain-like intelligence, and more closely simulate the computational patterns in the brain than traditional artificial intelligence networks. Neurons in a spiking neural network fire discrete events, also referred to as "spikes," and communicate with other neurons through synaptic connections between neurons. The human brain using pulse communication can often consume only about 20 watts of power to complete a large number of complex tasks including face recognition, and has high energy efficiency. Similarly, a spiking neural network can run on a neuromorphic computing chip and has far superior energy efficiency when inferred over traditional general purpose computing devices such as GPUs and the like. At present, the impulse neural network can obtain certain performance on tasks such as space-time data classification, detection, segmentation, tracking and the like.
In order to deploy a practical impulse neural network on the above-described neuromorphic computational chip, impulse neural networks will typically be of a relatively large scale. This type of neural network generally has a deeper network structure, also referred to as a deep impulse neural network. A deep impulse neural network such as Spiking ResNet or Spiking GoogleNet tends to have 50 or more network layers, which contain a large number of synapses. However, the computational resources on the neuromorphic computational chip are often very limited, for example, a 48-node SpiNNaker computational board supports at most 25 ten thousand neurons and 8 ten million synapses, which makes it difficult to deploy even a single-depth impulse neural network. Pruning of the deep spiking neural network is therefore crucial to the practical deployment of the spiking neural network.
In view of the above problems, no effective solution has been proposed.
Disclosure of Invention
The embodiment of the invention provides a pruning method and device of a pulse neural network and electronic equipment, which are used for at least solving the technical problem that the pulse neural network cannot be effectively deployed on a nerve form calculation chip in the related technology.
According to an aspect of an embodiment of the present invention, there is provided a pruning method for an impulse neural network, including: initializing an initial vector composed of each connection weight in a synaptic connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained; substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information; when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function; performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function; based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function; and when the number of pruning training rounds of the pulse neural network model reaches a preset number, deleting the hidden parameter vector to obtain the pulse neural network model with good pruning training.
According to another aspect of the embodiments of the present invention, there is also provided a pruning device for a spiking neural network, including: the initialization unit is used for carrying out initialization operation on an initial vector composed of each connection weight in a synaptic connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained; the first calculation unit is used for substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; the second calculation unit is used for inputting training data into the impulse neural network model and calculating a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information; the third calculation unit is used for calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function when the pulse neural network is subjected to pruning training based on a back propagation algorithm; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function; the gradient updating unit is used for performing gradient descending updating on the gradient of the hidden parameter vector and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function; a mapping unit, configured to map the hidden parameter vector to the weight vector again by using the soft threshold function based on the target threshold; and the determining unit is used for deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset number of rounds, so as to obtain the pulse neural network model which is pruned and trained.
According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the pruning method for the spiking neural network through the computer program.
According to a further aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the pruning method for the impulse neural network described above when running.
In the embodiment of the invention, an initial vector composed of each connection weight in a synaptic connection layer is adopted to carry out initialization operation, so as to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained; substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information; when the pulse neural network is subjected to pruning training based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function; gradient descending updating is carried out on the gradient of the hidden parameter vector, and a preset increasing function is utilized to calculate a target threshold used when the soft threshold function carries out gradient updating again; based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function; in the method, firstly, the hidden parameters with the same size are used for coding the weights of the pulse neural network by using a soft threshold function, so that the possibility that the weights enter a pruning state is preliminarily obtained. The gradient of the soft threshold function is modified during training so that the pruned weights can obtain a non-zero gradient to avoid the disappearance of the gradient and the consequent degradation of the network expression capability. The speed of network weight sparseness in different training stages is controlled through a threshold value increasing function, and balance can be obtained between two targets of learning and sparseness. The finally obtained network can directly abandon hidden parameters, uses sparse weight to carry out reasoning to save hardware computing resources, is suitable for various pulse neural networks based on gradient training, and further solves the technical problem that the pulse neural networks cannot be effectively deployed on a neural morphology computing chip in the related technology.
Drawings
The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention and do not constitute a limitation of the invention. In the drawings:
FIG. 1 is a schematic diagram of an application environment of an alternative pruning method for an impulse neural network according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of an application environment of an alternative pruning method for a spiking neural network according to an embodiment of the present invention;
FIG. 3 is a flow diagram illustrating an alternative method of pruning an impulse neural network in accordance with an embodiment of the present invention;
FIG. 4 is a schematic diagram of a threshold increasing function according to an alternative pruning method of a spiking neural network in the related art;
FIG. 5 is a schematic flow diagram of an alternative method of pruning an impulse neural network, according to an embodiment of the present invention;
FIG. 6 is a sparsity and precision comparison diagram of an alternative pruning method for a spiking neural network according to an embodiment of the present invention;
FIG. 7 is a schematic structural diagram of an alternative pruning device for a spiking neural network according to an embodiment of the present invention;
fig. 8 is a schematic structural diagram of an alternative electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Moreover, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
According to an aspect of the embodiments of the present invention, there is provided a pruning method for an impulse neural network, which may be, but is not limited to, applied in the application environment shown in fig. 1 as an optional implementation manner. The application environment comprises: the terminal equipment 102, the network 104 and the server 106 are used for human-computer interaction with the user. The user 108 and the terminal device 102 can perform human-computer interaction, and a pruning application of the impulse neural network is operated in the terminal device 102. The terminal device 102 includes a human-machine interaction screen 1022, a processor 1024, and a memory 1026. The human-computer interaction screen 1022 is used to display different types of visual information; the processor 1024 is configured to obtain the weight vector and the visual information. The memory 1026 is used to store the weight vector and the visual information.
In addition, the server 106 includes a database 1062 and a processing engine 1064, where the database 1062 is used for storing the weight vector to be displayed and the visual information. The processing engine 1064 is configured to perform initialization operation on an initial vector composed of each connection weight in the synaptic connection layer, so as to obtain a hidden parameter vector having the same elements as those of the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained; substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; wherein the training data comprises visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information; when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function; performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function; based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function; deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset round number, so as to obtain a trained pulse neural network model; and inputting the target visual information into the pulse neural network model trained by pruning, and acquiring a visual information processing result corresponding to the current task type. The visual information processing result is displayed on the client side of the terminal device 102.
In one or more embodiments, the pruning method of the impulse neural network described above in the present application may be applied to the application environment shown in fig. 2. As shown in fig. 2, a user 202 may interact with a user device 204. The user equipment 204 includes a memory 206 and a processor 208. The user device 204 in this embodiment may refer to, but is not limited to, performing the operation performed by the terminal device 102 described above to obtain the visual information processing result.
Optionally, the terminal device 102 and the user device 204 include, but are not limited to, a mobile phone, a set-top box, a television, a tablet computer, a notebook computer, a PC, a vehicle-mounted electronic device, a wearable device, and the like, and the network 104 may include, but is not limited to, a wireless network or a wired network. Wherein, this wireless network includes: WIFI and other networks that enable wireless communication. Such wired networks may include, but are not limited to: wide area networks, metropolitan area networks, and local area networks. The server 106 may include, but is not limited to, any hardware device capable of performing computations. The server may be a single server, a server cluster composed of a plurality of servers, or a cloud server. The above is merely an example, and this is not limited in this embodiment.
In order to solve the above technical problem, as an alternative implementation manner, as shown in fig. 3, an embodiment of the present invention provides a pruning method for a spiking neural network, including the following steps:
s302, initializing an initial vector composed of each connection weight in a synaptic connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
s304, substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector;
s306, inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; wherein the training data comprises visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
s308, when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
s310, performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function;
s312, based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function;
and S314, deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset number, so as to obtain the pulse neural network model with the pruned training.
In the embodiment of the invention, an initial vector composed of each connection weight in a synaptic connection layer is adopted to carry out initialization operation, so as to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained; substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; wherein the training data comprises visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information; when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function; gradient descending updating is carried out on the gradient of the hidden parameter vector, and a preset increasing function is utilized to calculate a target threshold used when the soft threshold function carries out gradient updating again; based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function; in the method, firstly, the hidden parameters with the same size are used for coding the weights of the pulse neural network by using a soft threshold function, so that the possibility that the weights enter a pruning state is preliminarily obtained. The gradient of the soft threshold function is modified during training so that the pruned weights can obtain a non-zero gradient to avoid the disappearance of the gradient and the consequent degradation of the network expression capability. The speed of network weight sparseness at different stages of training is controlled through a threshold value increasing function, and balance can be obtained between two targets of learning and sparseness. The finally obtained network can directly abandon hidden parameters, uses sparse weight to carry out reasoning to save hardware computing resources, is suitable for various pulse neural networks based on gradient training, and further solves the technical problem that the pulse neural networks cannot be effectively deployed on a neural morphology computing chip in the related technology.
In one or more embodiments, the pruning method of the spiking neural network further includes:
and inputting target visual information into the pruned and trained pulse neural network model, and acquiring a visual information processing result corresponding to the current task type.
In one or more embodiments, the target visual information includes at least one of an image, a video and a pulse signal, and the inputting the target visual information into the trained pulse neural network model to obtain a visual information processing result corresponding to the current task type includes at least one of:
and inputting the visual information into a pulse neural network model which is trained by pruning to carry out classification operation, and obtaining a classification processing result of the target visual information.
In particular, the pulse signal includes, but is not limited to, a signal obtained by sampling by a neuromorphic sensor. For example, the image set or the video set is input into a pulse neural network model trained by pruning to perform classification operation, and the image set or the video set is classified to obtain images and videos of different categories.
And inputting the visual information into the pulse neural network model which is trained by pruning to carry out target detection operation, and obtaining a detection result of a target object in the visual information.
Specifically, for example, the image set or the video set is input into a pruned trained impulse neural network model to perform a target detection operation, so that the attribute information of any target object in the image or the video can be acquired in real time.
And inputting the visual information into the pulse neural network model which is trained by pruning to perform target tracking operation, and acquiring a tracking processing result of a target object in the visual information.
Specifically, for example, the image set or the video set is input into the pruned and trained impulse neural network model for target tracking operation, so that a rectangular frame of an area where any target object in the image or the video is located can be obtained in real time.
In one or more embodiments, the synaptic connection layer comprises a fully-connected layer and a convolutional layer; the initializing operation of an initial vector composed of each connection weight in the synaptic connection layer to obtain a hidden parameter vector with the same number of elements as the initial vector comprises the following steps:
fusing trainable connection weights in the fully-connected layer and the convolutional layer into an initial vector w;
replacing w with a hidden parameter vector theta with the same number of elements as the w;
the substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector includes:
substituting each element in the theta into a soft threshold function S d Obtain a weight vector S d (θ); wherein, theta is a hidden parameter vector, and d is a threshold parameter;
S d (x)＝sign(x)·(|x|-d) + d is more than or equal to 0, sign (x) is a sign function;
in one or more embodiments, the predefined derivative function comprises: s d The derivative of (theta) with respect to theta is constantly equal to 1.
In one or more embodiments, in training the training process of pruning training the spiking neural network, the d is obtained by at least one of formulas (1), (2), and (3):
d＝kx,x∈[0,1],k>0； (1)
wherein x is the training progress of pruning,
wherein N is the current gradient updating step number of the hidden parameter vector, N is the cumulative step number of gradient updating in the pruning training process, and k is the final value of d when the pruning training is finished.
In one or more embodiments, the training rounds of pruning the spiking neural network model to reach a preset round number includes: and executing the gradient updating times of the hidden parameter vector to reach a preset round number.
Based on the foregoing embodiment, as shown in fig. 4, in an application embodiment, the pruning method for the spiking neural network further includes:
in the first step, a soft threshold function is defined as S d (θ)＝sign(θ)·(|θ|-d) + D is more than or equal to 0, wherein (·) + Representing element-by-element functions
And the parameter d ∈ R + And d is the threshold of the soft threshold function. Directly assigning the initialized network weight vector w to a hidden parameter vector theta ← w = S with the same size as the initialized network weight vector w 0 (θ)。
Secondly, in the impulse neural network, a loss function is obtained according to input training data X and data labels Y
Value of (A)
And obtaining the gradient of the loss function to the weight vector according to an algorithm based on back propagation
Here, training data such as images or video or pulse signals (e.g. acquired by neuromorphic sensors)The sampled signal) and the like. Loss function
Depending on the specific task (e.g., classification, detection, tracking of visual information).
Third, the gradient of the weight vector according to the loss function
And the modified derivative:
chain rule based on derivation
The gradient of the loss function to the hidden parameter vector can be obtained
Fourthly, after the hidden parameter theta is updated by the gradient, a predefined threshold value increasing function is used
The threshold at the nth update is obtained. Where N denotes the number of updates in the overall training process, thus
Represents the current algorithm execution progress, and k represents a preset threshold value when the algorithm is finished.
Fifthly, the hidden parameters of the function of the soft threshold value after the threshold value is increased are reused to obtain the network weight before the next updating
w＝S d (θ)
Repeating the above processes after the second step for N times, wherein the final model does not need to store hidden parameters during reasoning, but directly uses the sparse weight for computational reasoning.
In a specific implementation, the available function f is defined in the interval [0,1] and monotonically increases, and f (0) =0, f (1) =1, and there are various choices for the specific function f, and all three functions shown in fig. 5 satisfy the requirement, which may be taken to include, but is not limited to, any of the functions shown in fig. 5:
for example, the direct proportional function: d = kx, x ∈ [0,1], k >0;
sine function rise segment:
combination of sinusoidal and proportional functions:
in a specific implementation, the back propagation-based impulse neural network training algorithm refers to a series of methods for calculating the gradient of the output impulse to the network weight by a direct or indirect method, such as calculating the gradient by the relation between the impulse time and the input connection weight, smoothing the step function in the impulse distribution by a gradient substitution method to obtain a smoothed gradient, and obtaining an approximately optimal weight decreasing direction by a Widrow-Hoff algorithm or a variant thereof.
In order to verify the effect of the invention, pruning experiments are carried out on the currently popular test network SEW ResNet-18 and the large image classification data set ILSVRC2012 in the field. Pruning is performed under different parameters k by using the three threshold increasing functions, and the obtained sparsity and precision comparison is shown in fig. 6, wherein the dotted lines represent the classification precision of the dense network. It can be seen that the selection of different threshold increasing functions has a great influence on the sparsity of the network, and the proper selection can make the network still obtain a low classification precision loss at sparsity of more than 90%.
The technical scheme provided in the embodiment of the application has the following beneficial technical effects:
firstly, the weights of the impulse neural network are encoded by using hidden parameters with the same size and using a soft threshold function, so that the possibility that the weights enter a pruning state is preliminarily obtained. The gradient of the soft threshold function is modified during training so that the pruned weights can obtain a non-zero gradient to avoid the disappearance of the gradient and the consequent degradation of the network expression capability. The speed of network weight sparseness in different training stages is controlled through a threshold value increasing function, and balance can be obtained between two targets of learning and sparseness. The finally obtained network can directly abandon hidden parameters, use sparse weight to carry out reasoning to save hardware computing resources, and is suitable for various pulse neural networks based on gradient training.
It should be noted that for simplicity of description, the above-mentioned method embodiments are shown as a series of combinations of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.
According to another aspect of the embodiments of the present invention, there is also provided a pruning apparatus for an impulse neural network for implementing the pruning method for an impulse neural network described above. As shown in fig. 7, the apparatus includes:
an initialization unit 702, configured to perform initialization operation on an initial vector composed of each connection weight in a synapse connection layer, so as to obtain a hidden parameter vector having the same elements as those of the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
a first calculating unit 704, configured to substitute each element of the hidden parameter vector into a soft threshold function to obtain a weight vector;
the second calculating unit 706 is configured to input training data into the impulse neural network model, and calculate a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
a third calculating unit 708, configured to calculate a gradient of each loss function value to the hidden parameter vector through a predefined derivative function when the back propagation-based algorithm performs pruning training on the impulse neural network; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
a gradient updating unit 710, configured to perform gradient descent updating on the gradient of the hidden parameter vector, and calculate, by using a preset increasing function, a target threshold used when the soft threshold function performs gradient updating again;
a mapping unit 712, configured to map the hidden parameter vector to the weight vector again by using the soft threshold function based on the target threshold;
and the determining unit 714 is configured to delete the hidden parameter vector when the number of pruning training rounds of the spiking neural network model reaches a preset number of rounds, so as to obtain the spiking trained spiking neural network model.
In the embodiment of the invention, the method comprises the steps of acquiring the distinguishing identification information corresponding to the page to be displayed of the current client; acquiring a popup configuration resource file matched with the distinguishing identification information from a popup resource database; rendering the popup to be displayed according to the popup configuration resource file to obtain a target popup; in the method, a popup configuration resource file matched with the distinguishing identification information is obtained from a popup resource database; rendering the popup to be displayed according to the popup configuration resource file to obtain the target popup, and not separately developing the popup with different display styles, so that the development efficiency of the web application is improved, the application maintenance cost is reduced, and the technical problems of low web application development efficiency and high maintenance cost in the related technology are solved.
In one or more embodiments, the pruning device of the spiking neural network further includes:
and the first acquisition unit is used for inputting the target visual information into the pruned and trained pulse neural network model and acquiring a visual information processing result corresponding to the current task type.
In one or more embodiments, the target visual information includes at least one of an image, a video, and a pulse signal, and the first acquisition unit includes at least one of:
the first acquisition module is used for inputting the visual information into a pulse neural network model which is trained by pruning to carry out classification operation, and acquiring a classification processing result of the target visual information;
the second acquisition module is used for inputting the visual information into the pruned and trained pulse neural network model to perform target detection operation, and acquiring a detection result of a target object in the visual information;
and the third acquisition module is used for inputting the visual information into the pruned and trained pulse neural network model to perform target tracking operation, and acquiring a tracking processing result of a target object in the visual information.
In one or more embodiments, the synaptic connection layer comprises a fully-connected layer and a convolutional layer; the initialization unit 702 includes:
a fusion module for fusing the trainable connection weights in the fully-connected layer and the convolutional layer into an initial vector w;
a replacing module, configured to replace w with a hidden parameter vector θ having the same number as the elements of w;
the first computing unit 704 includes:
a first calculation module for substituting each element in the theta into a soft threshold function S d Obtain the weight vector S d (θ); wherein, theta is a hidden parameter vector, and d is a threshold parameter;
S d (x)＝sign(x)·(|x|-d) + d is more than or equal to 0, sign (x) is a sign function;
in one or more embodiments, the predefined derivative function comprises: s d (x) The derivative to x is constantly equal to 1.
In one or more embodiments, the pruning device of the spiking neural network further includes:
a second obtaining unit, configured to obtain, during a training process of pruning the spiking neural network to obtain d, at least one of equations (1), (2), and (3):
d＝kx,x∈[0,1],k>0；(1)
wherein x is the training progress of pruning,
wherein N is the current gradient updating step number of the hidden parameter vector, N is the cumulative step number of gradient updating in the pruning training process, and k is the final value of d when the pruning training is finished.
In one or more embodiments, the training rounds of pruning the spiking neural network model to reach a preset round number includes: and executing the gradient updating times of the hidden parameter vector to reach a preset round number.
According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the pruning method of the spiking neural network, where the electronic device may be a terminal device or a server shown in fig. 1. The present embodiment takes the electronic device as a server as an example for explanation. As shown in fig. 8, the electronic device comprises a memory 802 and a processor 804, the memory 802 having a computer program stored therein, the processor 804 being arranged to perform the steps of any of the method embodiments described above by means of the computer program.
Optionally, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of a computer network.
Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:
s1, initializing an initial vector composed of each connection weight in a synapse connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
s2, substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector;
s3, inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; wherein the training data comprises visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
s4, when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
s5, performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function;
s6, based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function;
and S7, deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset number, so as to obtain the pulse neural network model which is pruned and trained.
Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 8 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 8 is a diagram illustrating a structure of the electronic apparatus. For example, the electronics may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 8, or have a different configuration than shown in FIG. 8.
The memory 802 may be configured to store software programs and modules, such as program instructions/modules corresponding to the method and apparatus for pruning an impulse neural network in the embodiment of the present application, and the processor 804 executes various functional applications and data processing by running the software programs and modules stored in the memory 802, so as to implement the method for pruning an impulse neural network. The memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 802 can further include memory located remotely from the processor 804, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 802 may be, but not limited to, specifically configured to store weight vectors, visual information. As an example, as shown in fig. 8, the memory 802 may include, but is not limited to, an initialization unit 702, a first calculation unit 704, a second calculation unit 706, a third calculation unit 708, a gradient update unit 710, a mapping unit 712, and a determination unit 714 in the pruning device including the impulse neural network. In addition, other module units in the pruning device of the spiking neural network may also be included, but are not limited to these, and are not described in detail in this example.
Optionally, the transmission device 806 is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 806 includes a Network adapter (NIC) that can be connected to a router via a Network cable and can communicate with the internet or a local area Network. In one example, the transmission device 806 is a Radio Frequency (RF) module, which is used to communicate with the internet via wireless.
In addition, the electronic device further includes: a display 808 for displaying the weight vectors, visual information; and a connection bus 810 for connecting the respective module parts in the above-described electronic apparatus.
In other embodiments, the terminal device or the server may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. The nodes may form a Peer-To-Peer (P2P) network, and any type of computing device, such as a server, a terminal, and other electronic devices, may become a node in the blockchain system by joining the Peer-To-Peer network.
In one or more embodiments, the present application also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to cause the computer device to execute the pruning method of the spiking neural network. Wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.
Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:
s1, initializing an initial vector composed of each connection weight in a synaptic connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
s2, substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector;
s3, inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
s4, when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
s5, performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function;
s6, based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function;
and S7, deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset number, so as to obtain the pulse neural network model which is pruned and trained.
Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, read-Only memories (ROMs), random Access Memories (RAMs), magnetic or optical disks, and the like.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be essentially or partially contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, and the like) to execute all or part of the steps of the methods of the embodiments of the present invention.
In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, a division of a unit is merely a division of a logic function, and an actual implementation may have another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.
Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.
In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit may be implemented in the form of hardware, or may also be implemented in the form of a software functional unit.
The above is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, a plurality of modifications and embellishments can be made without departing from the principle of the present invention, and these modifications and embellishments should also be regarded as the protection scope of the present invention.
Claims (10)
1. A pruning method of an impulse neural network is characterized by comprising the following steps:
initializing an initial vector composed of each connection weight in a synapse connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector; inputting training data into the impulse neural network model, and calculating a loss function value corresponding to each task type; the training data comprise visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
when the pulse neural network is pruned and trained based on a back propagation algorithm, calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
performing gradient descending update on the gradient of the hidden parameter vector, and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function;
based on the target threshold, mapping the hidden parameter vector to the weight vector again by using the soft threshold function;
and when the number of pruning training rounds of the pulse neural network model reaches a preset number, deleting the hidden parameter vector to obtain the pulse neural network model with the good pruning training.
2. The method of claim 1, further comprising:
and inputting the target visual information into the pulse neural network model trained by pruning, and acquiring a visual information processing result corresponding to the current task type.
3. The method of claim 2, wherein the target visual information comprises at least one of an image, a video and a pulse signal, and the inputting the target visual information into the trained spiking neural network model to obtain the visual information processing result corresponding to the current task type comprises at least one of:
inputting the visual information into a pulse neural network model which is pruned and trained to perform classification operation, and acquiring a classification processing result of the target visual information;
inputting the visual information into the pruned and trained pulse neural network model to perform target detection operation, and acquiring a detection result of a target object in the visual information;
and inputting the visual information into the pruned and trained pulse neural network model to perform target tracking operation, and acquiring a tracking processing result of a target object in the visual information.
4. The method of claim 1, wherein the synaptic connection layer comprises a fully connected layer and a convolutional layer; the initializing operation of an initial vector composed of each connection weight in the synaptic connection layer to obtain a hidden parameter vector with the same number of elements as the initial vector comprises the following steps:
combining trainable connection weights in the full connection layer and convolution layer into an initial vector w;
replacing w with a hidden parameter vector theta with the same number of elements as the w;
the substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector includes:
substituting each element in the theta into a soft threshold function S d Obtain a weight vector S d (θ); where θ is the hidden parameter vector, d is the threshold parameterAnd (4) counting.
5. The method of claim 4, wherein the predefined derivative function comprises:
S d the derivative of (theta) with respect to theta is constantly equal to 1.
6. The method of claim 4, wherein the d is obtained by at least one of equations (1), (2), and (3) during training for training the spiking neural network for pruning:
d＝kx,x∈[0,1],k>0；(1)
wherein x is the training progress of pruning,
wherein N is the current gradient updating step number of the hidden parameter vector, N is the cumulative step number of gradient updating in the pruning training process, and k is the final value of d when the pruning training is finished.
7. The method of claim 1, wherein the training rounds of pruning of the spiking neural network model to a preset number of rounds comprises:
and executing the gradient updating times of the hidden parameter vector to reach a preset round number.
8. A pruning device of an impulse neural network is characterized by comprising the following components:
the initialization unit is used for carrying out initialization operation on an initial vector formed by each connection weight in a synapse connection layer to obtain a hidden parameter vector with the same elements as the initial vector; the synaptic connection layer is positioned in a spiking neural network model to be pruned, optimized and retrained;
the first calculation unit is used for substituting each element of the hidden parameter vector into a soft threshold function to obtain a weight vector;
the second calculation unit is used for inputting training data into the pulse neural network model and calculating a loss function value corresponding to each task type; wherein the training data comprises visual information, and the task type comprises classification, detection and tracking of a target object contained in the visual information;
the third calculation unit is used for calculating the gradient of each loss function value to the hidden parameter vector through a predefined derivative function when the pulse neural network is subjected to pruning training based on a back propagation algorithm; wherein the predefined derivative function is a derivative function obtained by modifying a derivative function of the soft threshold function;
the gradient updating unit is used for performing gradient descending updating on the gradient of the hidden parameter vector and calculating a target threshold used when the soft threshold function performs gradient updating again by using a preset increasing function;
a mapping unit, configured to map the hidden parameter vector to the weight vector again by using the soft threshold function based on the target threshold;
and the determining unit is used for deleting the hidden parameter vector when the number of pruning training rounds of the pulse neural network model reaches a preset number of rounds, so as to obtain the pulse neural network model which is pruned and trained.
9. A computer-readable storage medium, comprising a stored program, wherein the program when executed performs the method of any one of claims 1 to 7.
10. An electronic device comprising a memory and a processor, characterized in that the memory has stored therein a computer program, the processor being arranged to execute the method of any of claims 1 to 7 by means of the computer program.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210867737.8A CN115423087A (en) | 2022-07-22 | 2022-07-22 | Pruning method and device for impulse neural network and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210867737.8A CN115423087A (en) | 2022-07-22 | 2022-07-22 | Pruning method and device for impulse neural network and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115423087A true CN115423087A (en) | 2022-12-02 |
Family
ID=84196503
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210867737.8A Pending CN115423087A (en) | 2022-07-22 | 2022-07-22 | Pruning method and device for impulse neural network and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115423087A (en) |
-
2022
- 2022-07-22 CN CN202210867737.8A patent/CN115423087A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Jaafra et al. | Reinforcement learning for neural architecture search: A review | |
| US11875268B2 (en) | Object recognition with reduced neural network weight precision | |
| US9111375B2 (en) | Evaluation of three-dimensional scenes using two-dimensional representations | |
| CN109816009A (en) | Multi-tag image classification method, device and equipment based on picture scroll product | |
| CN113159283B (en) | Model training method based on federal transfer learning and computing node | |
| US20220215259A1 (en) | Neural network training method, data processing method, and related apparatus | |
| CN111176758B (en) | Configuration parameter recommendation method and device, terminal and storage medium | |
| CN111695596A (en) | Neural network for image processing and related equipment | |
| JP7013489B2 (en) | Learning device, live-action image classification device generation system, live-action image classification device generation device, learning method and program | |
| CN111222046B (en) | Service configuration method, client for service configuration, equipment and electronic equipment | |
| CN113435520A (en) | Neural network training method, device, equipment and computer readable storage medium | |
| CN111833360B (en) | Image processing method, device, equipment and computer readable storage medium | |
| CN110765960B (en) | Pedestrian re-identification method for adaptive multi-task deep learning | |
| CN112381707B (en) | Image generation method, device, equipment and storage medium | |
| CN106778910A (en) | Deep learning system and method based on local training | |
| CN112906888A (en) | Task execution method and device, electronic equipment and storage medium | |
| CN110378250A (en) | Training method, device and the terminal device of neural network for scene cognition | |
| CN115238909A (en) | Data value evaluation method based on federal learning and related equipment thereof | |
| CN111159279B (en) | Model visualization method, device and storage medium | |
| Manna et al. | Simple and complex spiking neurons: perspectives and analysis in a simple STDP scenario | |
| CN113987236A (en) | Unsupervised training method and unsupervised training device for visual retrieval model based on graph convolution network | |
| Hafiz et al. | Classification of skeletal wireframe representation of hand gesture using complex-valued neural network | |
| CN113537267A (en) | Method and device for generating countermeasure sample, storage medium and electronic equipment | |
| CN112686381A (en) | Neural network model, method, electronic device, and readable medium | |
| CN115423087A (en) | Pruning method and device for impulse neural network and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |