CN108717570A - A kind of impulsive neural networks parameter quantification method - Google Patents

Abstract

The present invention relates to nerual network technique field more particularly to a kind of impulsive neural networks parameter quantification methods.The present invention method by map offline or on-line training obtain training complete original pulse neural network, the parameters such as impulsive neural networks weights, threshold value, leakage constant, set voltage, refractory period, the synaptic delay completed to training quantify, and all layers of neural network can share same group of quantization parameter or respectively one group of quantization parameter.Impulsive neural networks after parameter quantization only need a small amount of parameter that high-precision pulse neural network function can be realized.This method is high-precision simultaneously in holding, and effectively save impulsive neural networks parameter storage space improves arithmetic speed, reduces operation power consumption.

Description

A Quantification Method of Spiking Neural Network Parameters

technical field

The invention relates to the technical field of neural networks, in particular to a parameter quantification method of a pulse neural network.

Background technique

Spiking neural network (SNN for short) is called the third-generation neural network, which is closer to the way the human brain processes information and is the future development direction of neural network technology. SNNs receive information based on pulse sequences. There are many encoding methods that can interpret the pulse sequence as an actual number. Commonly used encoding methods include pulse encoding and frequency encoding. The communication between neurons is also carried out through pulses. When the membrane potential of a neuron is greater than its threshold, it will generate a pulse signal and transmit it to other neurons to increase or decrease its membrane potential. The hardware platform of SNN is called neuromorphic chip or brain-like chip, which completely subverts the traditional von Neumann architecture. This kind of chip has the characteristics of low power consumption and less resource consumption. The performance is much better than traditional chips. There are two main ways to train SNN. One is to map the trained parameters to SNN by training the corresponding artificial neural network (ANN for short) under specific conditions, but it is often necessary to pass a large number of parameters during the mapping process. ; The other is to directly conduct online learning of SNN, which is also accompanied by a large number of parameters. If traditional memory (such as SRAM, DRAM, etc.) is used to store parameters, a huge storage space is required. If new devices such as memristors are used to store parameters, it is difficult to accurately and stably realize many parameters; at the same time, the huge amount of parameters will reduce the calculation speed , Increase computing power consumption. There is currently no method that can compress the large number of parameters in SNNs.

Contents of the invention

In order to solve the problems in the prior art, the present invention provides a method that can reduce the storage space of SNN parameters.

Technical scheme of the present invention is as follows:

A kind of spiking neural network parameter quantification method, is characterized in that, comprises the following steps:

Get the original SNN that has been trained. The neuron in the SNN is a pulse neuron (such as a LIF neuron), which has the function of integrating and accumulating input pulses and the function of emitting pulses. The SNN uses pulse sequences as input. The main parameters of the SNN include weights, thresholds, leakage constants, set voltages, Refractory period, synaptic delay, etc. The trained SNN has high-precision classification, recognition and other functions. There are two main ways to obtain the trained neural network: one is to obtain the trained SNN through offline mapping, and train the ANN (including MLP, CNN, RNN, LSTM, etc.) After completing the training process of the ANN with index requirements (such as classification, recognition accuracy, etc.), the parameters of the trained ANN are mapped to the SNN with the same topology, and the input of the ANN is coded by pulse sequence (such as the pulse sequence of Poisson distribution ) as the input of the SNN to obtain the trained SNN; one is to obtain the trained SNN through online training, establish a self-organizing SNN or a SNN with other structures, and use learning rules such as synaptic pulse timing-dependent plasticity (STDP) , use pulse sequences (such as Poisson distribution pulse sequences, time-coded pulse sequences, etc.) to train SNN through online learning, and adjust SNN weights, thresholds, leakage constants, set voltages, refractory periods, and synaptic delays during training. After obtaining the SNN that meets the index requirements (such as classification, recognition accuracy, etc.), the training process is completed. After the training, the parameters such as the weight, threshold, leakage constant, set voltage, refractory period, and synaptic delay of the SNN are fixed, so as to obtain The trained SNN;

Select one or more parameters to be quantized. Parameters that can be quantified include weights, thresholds, leakage constants, set voltages, refractory periods, synaptic delays, and more.

Separately count the distribution of parameters of a certain layer or several layers or all layers;

Try interval-dividing the parameters. Select the parameter interval division method and the number of intervals. The interval division method can be divided into equal division method, non-uniform division method, confidence interval division method, etc. The method and number of division intervals can be based on the specific neural network structure, task type and index requirements. Try and adjust through parameter adjustment experience;

Quantize parameters in terms of intervals. Traverse all the parameters in the interval, and quantize all the parameters distributed in the same interval to the same value (that is, a quantization value). The size and sign of the quantization value are based on the specific neural network structure, task type, and index requirements. Adapt experience to try and adjust;

Replace the corresponding parameters in the original SNN with the quantized parameters to obtain the parameter quantized SNN;

The input of the original SNN is used as the input, and the parameter quantized SNN is tested. If the test result meets the index requirements, it ends. Otherwise, it returns to reselect the parameter interval division method and the number of intervals, and performs interval division and subsequent processes on the parameters.

The beneficial effects of the present invention are that the method of the present invention can transform ANN into SNN, and realize parameter quantization of SNN. The quantization method is simple and flexible in operation, can realize quantization in various forms, and has almost no influence on the performance of neural network , which can save storage resources and improve computing speed. Especially when it is necessary to implement SNN hardware, this method can reduce the consumption of RAM and other on-chip resources and computational complexity, and improve the hardware computational speed and performance.

Description of drawings

Fig. 1 is the schematic diagram of a kind of SNN parameter quantization method provided by the example of the present invention;

Fig. 2 is a schematic diagram of MLP, one of the ANN examples in Fig. 1;

Fig. 3 is a CNN schematic diagram of one of the ANN examples in Fig. 1;

Fig. 4 is a schematic diagram of RNN, one of the ANN examples in Fig. 1;

Fig. 5 is a schematic diagram of LSTM, one of the ANN examples in Fig. 1;

Fig. 6 is a schematic diagram of an ad hoc network of one of the SNN examples in Fig. 1;

Figure 7 is a weight distribution diagram and interval division of one of the quantization parameter examples in Figure 1;

Figure 8 is a threshold distribution diagram and interval division of one of the quantization parameter examples in Figure 2;

Fig. 9 is a leakage constant distribution diagram and interval division of one of the quantitative parameter examples in Fig. 2;

FIG. 10 is a set voltage distribution diagram and interval division of one of the quantization parameter examples in FIG. 2;

Fig. 11 is one of quantitative parameter examples among Fig. 2 refractory period distribution figure and interval division situation;

FIG. 12 is a diagram of synaptic delay distribution and division of intervals, one of the examples of quantization parameters in FIG. 2 .

Fig. 13 is a schematic diagram of an example method for implementing parameter quantization by the SNN in Fig. 1 .

Detailed ways

The present invention will be described in detail below in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when detailed descriptions of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

As shown in Figure 1, a SNN parameter quantization method includes the following steps:

S1: Obtain the original SNN that has been trained.

The neuron in the SNN is a pulse neuron (such as a LIF neuron), which has the function of integrating and accumulating input pulses and the function of emitting pulses. The SNN uses pulse sequences as input. The main parameters of the SNN include weights, thresholds, leakage constants, set voltages, Refractory period, synaptic delay, etc. The trained SNN has high-precision classification, recognition and other functions. There are two main methods to obtain the trained neural network: one is to obtain the trained SNN through offline mapping, and to train MLP (see Figure 2), CNN (see Figure 3), RNN through stochastic gradient descent commonly used in training ANN. (See Figure 4), LSTM (see Figure 5) and other ANNs, after obtaining the ANN that meets the index requirements (such as classification, recognition accuracy, etc.), the training process is completed, and then the parameters of the trained ANN are mapped to the SNN with the same topology , the input of ANN is coded by pulse sequence (for example, the pulse sequence of Poisson distribution) as the input of SNN, so as to obtain the trained SNN; one is to obtain the trained SNN through online training, and establish a self-organizing SNN (see Figure 6) or SNN with other structures, using learning rules such as synaptic spike timing-dependent plasticity (STDP), using pulse sequences (such as Poisson distribution pulse sequences, time-coded pulse sequences, etc.) to train SNN through online learning, and adjust during training SNN weights, thresholds, leakage constants, set voltages, refractory periods, synaptic delays and other parameters, after obtaining the SNN that meets the index requirements (such as classification, recognition accuracy, etc.), the training process is completed, and the weight of the SNN is fixed after the training is over. , threshold, leakage constant, setting voltage, refractory period, synaptic delay and other parameters, so as to obtain the trained SNN.

S2: Select one or more parameters to be quantized.

You can quantize one parameter at a time, or you can quantize multiple parameters at a time. Parameters that can be quantified include weights, thresholds, leakage constants, set voltages, refractory periods, synaptic delays, etc.

S3: Statistically calculate the parameter distribution of a certain layer or several layers or all layers.

For a certain parameter that needs to be quantified, the parameters of a certain layer or a few layers or all layers of the SNN are counted and the parameter distribution diagram is drawn.

S4: Try to divide the parameters into intervals.

Select the parameter interval division method and the number of intervals. The interval division method can be divided into equal division method, non-uniform division method, confidence interval division method, etc. The method and number of division intervals can be based on the specific neural network structure, task type and index requirements. Experiment and adjust with parameter tuning experience. On the basis of the parameter distribution diagram described in S3, the weight (see Figure 7), threshold (see Figure 8), leakage constant (see Figure 9), set voltage (see Figure 10), refractory period (Refer to FIG. 11 ), synaptic delay (refer to FIG. 12 ) and other parameters after the interval division results are given in the accompanying drawings.

S5: Quantify the parameters according to the interval.

Traverse all the parameters in the interval, and quantize all the parameters distributed in the same interval to the same value (that is, a quantization value). The size and sign of the quantization value are based on the specific neural network structure, task type, and index requirements. Adjust experience to try and adjust.

S6: Obtain parameter quantized SNN.

Replace the corresponding parameters in the original SNN with the quantized parameters to obtain the parameter quantized SNN.

S7: Test parameter quantization SNN.

The input of the original SNN is used as the input, and the parameter quantized SNN is tested. If the test result meets the index requirements, it ends. Otherwise, it returns to reselect the parameter interval division method and the number of intervals, and performs interval division and subsequent processes on the parameters.

Referring to Figure 13, the quantification method is further explained by using MLP to realize the MNIST handwritten digit recognition task as an example, including the following steps:

S8: Set the correct rate of target recognition.

That is to say, the index requirements described in SI set the target recognition accuracy rate of the neural network system on the MNIST test set.

S9: Use the BP algorithm to train the MLP and directly map the trained weights to the SNN.

That is, the offline training method described in S1. In this example, some specific conditions need to be met when training the MLP: (1) All units of the MLP should use the Relu function as the activation function; (2) During the training process, the neuron The bias is fixed at 0.

S10: Setting a threshold and a maximum frequency of the SNN.

This is a characteristic parameter of SNN. The input of the SNN needs to be changed to pulse form, so the input picture needs to be encoded. In this example, each pixel of the picture is encoded using the Poisson distribution frequency coding method. The frequency of the pulse is positively correlated with the size of the input pixel. The maximum frequency of pulses and the threshold of LIF neurons are tried and adjusted through parameter adjustment experience according to the size of the mapping parameters and the recognition rate of subsequent feedback.

S11: Test the SNN recognition rate.

If the accuracy rate of target recognition in S8 is met, proceed to the next step, otherwise return to S8 to retrain the MLP and follow-up process.

S12: Obtain weight distribution maps of all layers and equally divide the weight intervals.

That is, as described in S3, the statistics of the distribution of parameters in all layers and as described in S4, attempt to divide the parameters into intervals. In this example, only the weights are counted and divided into intervals, and the equal division method is tried for division, and the number of intervals is 4.

S13: Quantize the weight according to the interval (try to use the midpoint of the interval for quantization).

That is, the parameters are quantized according to intervals as described in S5. In this example try quantization using the midpoint of the interval.

S14: Traversing all values of the SNN to find the maximum value wmax and the minimum value wmin, taking the average value of wmax and wmin and recording it as w0.

Steps for quantization using interval midpoints.

S15: take the average value of wmin and w0 and record it as w-1; take the average value of wmax and w0 and record it as w1.

Steps for quantization using interval midpoints.

S16: Traversing all the values again, if the weight is between wmin and w-1, set the weight equal to the average x1 of the two; if it is between w-1 and w0, set the weight equal to both The average value of x2, similarly get x3 and x4.

Steps for quantization using interval midpoints.

S17: Test the SNN recognition rate.

That is, the test parameters described in S7 quantify the SNN. If the performance index is satisfied, then end, otherwise return to S12 to reselect the interval division method and subsequent process.

Claims (4)

1. A pulse neural network parameter quantification method is characterized by comprising the following steps:

s1, acquiring a trained original impulse neural network, wherein parameters of the original impulse neural network comprise weight, threshold, leakage constant, set voltage, refractory period and synaptic delay;

s2, selecting one or more parameters needing quantization;

s3, counting the distribution of the selected parameters in the neural network;

s4, carrying out interval division on the selected parameters;

s5, quantizing the parameters according to the intervals, namely quantizing the parameters distributed in the same interval into the same value;

s6, obtaining a parameter quantization pulse neural network, namely replacing original parameters which only correspond to the original pulse neural network by the quantization values obtained in the step 5;

s7, testing the parameter quantization pulse neural network obtained in the step S6: and (4) testing the parameter quantization neural network by taking the input of the original pulse neural network as input, finishing if the test result meets the requirement of a preset index, and returning to the step S3 if the test result does not meet the requirement of the preset index.

2. The method according to claim 1, wherein the specific method for obtaining the trained raw spiking neural network in step S1 is as follows:

training a corresponding artificial neural network, wherein the artificial neural network is one of a multilayer perceptron, a convolutional neural network, a cyclic neural network and a long-term and short-term memory network, mapping training parameters to a pulse neural network with the same topological structure, and coding input data by adopting a pulse sequence to obtain an original pulse neural network;

or,

establishing a pulse neural network, adopting a pulse sequence as input, training the pulse neural network on line by utilizing a learning mechanism that synaptic pulse time sequence depends on plasticity, and fixing parameters of the pulse neural network after training is finished so as to obtain an original pulse neural network after training is finished.

3. The method for quantifying parameters of an impulse neural network according to claim 2, wherein the specific method in step S3 is as follows:

counting the parameter distribution condition of the selected parameters in a certain layer of network;

or,

counting the respective conditions of the parameters of the selected parameters in a certain layer of network;

still alternatively, the first and second substrates may be,

and counting the respective conditions of the parameters of the selected parameters in each layer network.

4. The method according to claim 3, wherein the specific method in step S4 is as follows:

and dividing the parameter intervals by adopting one of an averaging method, a non-averaging method and a confidence interval dividing method, and obtaining the number of the intervals.