CN115100458A - Image classification method and related device - Google Patents
Image classification method and related device Download PDFInfo
- Publication number
- CN115100458A CN115100458A CN202210623187.5A CN202210623187A CN115100458A CN 115100458 A CN115100458 A CN 115100458A CN 202210623187 A CN202210623187 A CN 202210623187A CN 115100458 A CN115100458 A CN 115100458A
- Authority
- CN
- China
- Prior art keywords
- membrane potential
- image
- memory
- layer
- neuron
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 66
- 239000012528 membrane Substances 0.000 claims abstract description 122
- 230000015654 memory Effects 0.000 claims abstract description 105
- 210000002569 neuron Anatomy 0.000 claims abstract description 90
- 238000012545 processing Methods 0.000 claims abstract description 51
- 238000013145 classification model Methods 0.000 claims abstract description 32
- 210000000225 synapse Anatomy 0.000 claims abstract description 22
- 230000000946 synaptic effect Effects 0.000 claims abstract description 21
- 230000008569 process Effects 0.000 claims abstract description 20
- 238000011176 pooling Methods 0.000 claims description 46
- 230000005764 inhibitory process Effects 0.000 claims description 27
- 238000012549 training Methods 0.000 claims description 22
- 238000001914 filtration Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 6
- 238000004364 calculation method Methods 0.000 abstract description 10
- 239000010410 layer Substances 0.000 description 155
- 238000013528 artificial neural network Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 9
- 230000001629 suppression Effects 0.000 description 8
- 238000004422 calculation algorithm Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000007274 generation of a signal involved in cell-cell signaling Effects 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000002346 layers by function Substances 0.000 description 2
- 230000005055 memory storage Effects 0.000 description 2
- 230000001537 neural effect Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000001242 postsynaptic effect Effects 0.000 description 2
- 230000036279 refractory period Effects 0.000 description 2
- 238000012421 spiking Methods 0.000 description 2
- 230000001133 acceleration Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000003139 buffering effect Effects 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000002401 inhibitory effect Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000006386 memory function Effects 0.000 description 1
- 210000000653 nervous system Anatomy 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Computational Linguistics (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Neurology (AREA)
- Image Analysis (AREA)
Abstract
The invention provides an image classification method and a related device, wherein the image classification method comprises the following steps: acquiring an image to be classified; processing an image to be classified by using an image classification model, and determining the type of the image to be classified; and updating the current membrane potential of the neuron of the image classification model by using the synapse weight membrane potential increment in the process of image processing of the image classification model, wherein the synapse weight membrane potential increment is stored in a memory. The synaptic weight membrane potential increment is directly stored in the memory, and is directly read from the memory without extra calculation when the current membrane potential of the neuron is updated, so that the calculation amount is reduced.
Description
Technical Field
The invention belongs to the field of image recognition based on a neural network, and relates to an image classification method and a related device.
Background
As a third generation impulse neural network, the information propagation structure of the axon-dendrite of the biological nervous system of the impulse neural network adopts an impulse neuron model and utilizes impulses to transmit information. Compared with the traditional neural network, the impulse neural network has better biological interpretability and computational efficiency, and is an important way for realizing neural computation and artificial intelligence in the future.
Although the prior art adopts a pulse propagation and sparse time coding mode, the prior art still needs convolution operation, and a large amount of operation is generated during the convolution operation.
Disclosure of Invention
The application provides an image classification method and a related device, and the method can reduce the calculation amount.
In a first aspect, the present application provides an image classification method, including: acquiring an image to be classified; processing an image to be classified by using an image classification model, and determining the type of the image to be classified; and updating the current membrane potential of the neuron of the image classification model by using the synapse weight membrane potential increment in the process of image processing of the image classification model, wherein the synapse weight membrane potential increment is stored in a memory.
The image classification method comprises the following steps of processing an image to be classified by using an image classification model, and determining the type of the image to be classified, wherein the step of determining the type of the image to be classified comprises the following steps: filtering the image to be classified by using a filtering layer, and coding to obtain a first output result; processing the first output result by utilizing the cascaded convolutional layer, pooling layer, convolutional layer and pooling layer to obtain a second output result; determining the type of the image to be classified based on the second output result.
Wherein the step of processing the first output result using the convolutional layer comprises: acquiring synapse weight membrane potential increment of a neuron to be updated from the memory; updating the current membrane potential of the neuron to be updated in the convolutional layer by using the synapse weight membrane potential increment; and processing the first output result by using the updated convolutional layer.
Wherein the step of obtaining the synaptic weight membrane potential increment from the memory comprises:
acquiring a pulse signal, wherein the pulse signal comprises position information of the pulse signal; calculating to obtain first address information and second address information of the neuron to be updated according to the size information of the convolutional layer based on the pulse signal; and acquiring the synaptic weight membrane potential increment from the memory based on the first address information and the second address information.
After the step of updating the current membrane potential of the neuron to be updated in the convolutional layer by using the synaptic weight membrane potential increment, the method further comprises the following steps: determining whether the updated membrane potential is greater than a threshold; in response to the updated membrane potential being greater than the threshold, the neuron to be updated pulses and modifies an inhibition flag of the neuron to be updated such that the inhibition flag characterizes the neuron to be updated as not being able to pulse; storing an updated film potential into the memory based on the first address information and the second address information in response to the updated film potential being less than or equal to the threshold.
Wherein the method further comprises: acquiring a training sample set, wherein each sample image in the training sample set is marked with the type of the image; and training an initial model by using the training sample set to obtain an image classification model.
Wherein, the step of training the initial model by using the training sample set to obtain the image classification model comprises the following steps: carrying out fixed point processing on the synapse weight of the image classification model to obtain a fixed point weight; storing a current membrane potential of each neuron and an inhibition flag of the neuron in a first memory; and storing the fixed point weights in a second memory; the step of obtaining the synaptic weight membrane potential increment from the memory based on the first address information and the second address information comprises: obtaining a current membrane potential of the neuron from the first memory based on the first address information; acquiring the fixed point weight value from the second address information based on the second address information; determining the synaptic weight membrane potential increment based on the current membrane potential of the neuron and the fixed point weight.
Wherein, a buffer layer is arranged among the filter layer, the convolution layer, the pooling layer, the convolution layer and the pooling layer; temporarily storing the received data by using the buffer layer, and judging whether the processing layer corresponding to the output direction is idle or not; and responding to the idle state, and transmitting the temporarily stored data to the processing layer, wherein the processing layer comprises at least one of a convolutional layer, a pooling layer, a convolutional layer and a pooling layer.
In a second aspect, the present application provides an electronic device, comprising a processor and a memory coupled to each other, wherein the memory is configured to store program instructions for implementing any one of the methods described above; the processor is configured to execute the program instructions stored by the memory.
In a third aspect, the present application provides a computer-readable storage medium, in which a program file is stored, the program file being executable to implement the method of any one of the above.
The image processing method of the present invention is different from the prior art, and includes: acquiring an image to be classified; processing an image to be classified by using an image classification model, and determining the type of the image to be classified; and updating the current membrane potential of the neuron of the image classification model by using the synapse weight membrane potential increment in the process of image processing of the image classification model, wherein the synapse weight membrane potential increment is stored in a memory. The synaptic weight membrane potential increment is directly stored in the memory, and is directly read from the memory without extra calculation when the current membrane potential of the neuron is updated, so that the calculation amount is reduced.
Drawings
FIG. 1 is a flowchart illustrating an image classification method according to a first embodiment of the present invention;
FIG. 2 is a flowchart illustrating an image classification method according to a second embodiment of the present invention;
FIG. 3 is a schematic diagram of an embodiment of a convolutional layer;
FIG. 4 is a schematic structural diagram of an embodiment of a pooling layer;
FIG. 5 is a schematic diagram of a pulse neural network image processing system;
FIG. 6 is a schematic diagram of a deployment structure of an image classification model corresponding to the image classification method of the present invention;
FIG. 7 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;
FIG. 8 is a structural diagram of an embodiment of a computer-readable storage medium according to the invention.
Detailed description of the invention
To further illustrate the technical means and effects of the present invention for achieving the predetermined objects, a detailed description is given below of an efficient image classification circuit based on an AER pure pulse depth convolutional neural network according to the present invention with reference to the accompanying drawings and the detailed description. The foregoing and other technical matters, features and effects of the present invention will be more clearly understood from the following detailed description of the embodiments taken in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The drawings are only for reference and illustration purposes and are not intended to limit the invention.
Referring to fig. 1, a schematic flow chart of a first embodiment of the image classification method of the present invention specifically includes:
step S11: and acquiring an image to be classified.
Step S12: processing an image to be classified by using an image classification model, and determining the type of the image to be classified; and updating the current membrane potential of the neuron of the image classification model by using the synapse weight membrane potential increment in the process of image processing of the image classification model, wherein the synapse weight membrane potential increment is stored in a memory.
Specifically, the image classification includes a gaussian filter layer, three convolution layers, and three pool layers. The process of processing the image to be classified by the image classification model comprises the steps of filtering the image to be classified by using a filtering layer, and coding to obtain a first output result; processing the first output result by utilizing the cascaded convolutional layer, pooling layer, convolutional layer and pooling layer to obtain a second output result; determining the type of the image to be classified based on the second output result. In this embodiment, the processing process of the image classification model on the image to be classified is as follows: the image to be classified → the gaussian filter layer → the first output result → the convolution layer (again, the first convolution layer) → the pooling layer (again, the first pooling layer) → the convolution layer (again, the second convolution layer) → the pooling layer (again, the second pooling layer) → the convolution layer (again, the third convolution layer) → the pooling layer (again, the third pooling layer) → the second output result.
It should be noted that the image classification model is obtained by training an initial model in advance, where the initial model may be a lif (leak integration and fire) neuron model without considering leakage, and the model can perform first pulse trigger time coding on an input image and convert the input image into a time pulse signal.
It should be noted that the input image size of the image to be classified is 28 × 28, and the image filtering is performed by 6 different gaussian difference filtering kernel functions in the gaussian filtering layer, so that the size of the generated first output result is 6 × 28. And the first pulse trigger time coding is carried out on the first pulse trigger time coding to further obtain a first output result. In the first convolution layer, the convolution kernel size was 6 × 5 × 30, the neuron threshold was 15, and the network size after the convolution operation was 30 × 28. The pooling layer performs maximal pooling operation, and the neurons detect input pulses of the front neurons in the pooling window region and immediately emit pulses. The pooling step size of the first pooling layer was 2, so the size of the pooled network was 30 x 14. The second convolution kernel size is 30 x 3 x 250, the neuron threshold is 10, and the network size after convolution is 250 x 14. The second pooling layer step size was 3 and the size of the pooled network was reduced to 250 x 5. The threshold of the third convolutional layer was set to infinity, the effects of the input pulses on the neurons in the layer were accumulated, the convolution kernel was 250 x 5 x 200, and the net size was 200 x 5. The third pooling layer is a global pooling layer, and the maximum of the membrane potential of each channel is extracted as a feature vector output after propagation is ended.
In one embodiment, the membrane potential needs to be updated because the input pulse causes the membrane potential of the synaptic neuron to increase according to a certain mapping relationship. Specifically, the step of processing the first output result by using convolution layers (a first convolution layer, a second convolution layer, and a third convolution layer) includes: acquiring synaptic weight membrane potential increment of a neuron to be updated from the memory; updating the current membrane potential of the neuron to be updated in the convolutional layer by using the synapse weight membrane potential increment; and processing the first output result by using the updated convolutional layer.
Specifically, a pulse signal is obtained, wherein the pulse signal comprises position information of the pulse signal, and based on the pulse signal, first address information and second address information of the neuron to be updated are obtained through calculation according to size information of the convolutional layer; and acquiring the synapse weight membrane potential increment from the memory based on the first address information and the second address information.
Specifically, after the current membrane potential of the neuron is updated by using the synaptic weight membrane potential increment, determining whether the updated membrane potential is greater than or equal to a threshold value; in response to the updated membrane potential being greater than the threshold, the neuron to be updated pulses and modifies an inhibit flag of the neuron to be updated such that the inhibit flag characterizes the neuron to be updated as not being able to pulse. It should be noted that the suppression flag is a mechanism of the spiking neural network, which is used to increase the specificity of feature detection and reduce the operations of spiking update. The pulsing neuron accumulates a membrane potential, which may be pulsed when it reaches a threshold. After a pulse is sent, the neuron is considered to enter a refractory period and cannot send the next pulse. Since the determination of whether the membrane potential reaches the threshold is after the membrane potential updating operation, if these neurons have failed to pulse, it is not necessary to update them with membrane potential, nor to detect whether their membrane potential value is greater than the threshold. The basis for this determination is their pulse flag. That is, when the updated membrane potential is greater than the threshold, the neuron to be updated sends out a pulse, enters a refractory period, and cannot send out the next pulse. In response to the updated film potential being less than or equal to the threshold, storing the updated film potential into the memory based on the first address information and the second address information.
And directly obtaining a classification result according to the second output result according to the division of the R-STDP algorithm. In the R-STDP training process by using the MINST data set, the output second output result is uniformly divided into ten non-overlapping intervals, the classification result is judged according to the interval where the maximum value of the multi-dimensional feature vector is located, and the reward and punishment signal is determined by comparing the label of the image to be classified, so that the network classification accuracy is improved by a learning-enhanced strategy. For the network after training, the classification result of the network for the image can be directly obtained according to the output feature vector.
In an embodiment of the present application, as shown in fig. 2, compared to fig. 1, fig. 2 further includes:
step S13: a training sample set is obtained, and each sample image in the training sample set is marked with the type of the image.
Step S14: and training an initial model by using the training sample set to obtain an image classification model.
Specifically, the weights of the various layers of the impulse convolutional neural network are iteratively trained using MNIST (Modified National Institute of Standard and Technology database, Modified National Institute of standards and Technology database) handwriting data sets. The network trains each layer respectively by adopting a mode of combining STDP and R-STDP algorithms, and verifies the classification accuracy of the input data set. Through experimental tests, after 680 times of iterative learning, the classification accuracy of the data set is 94.7%.
After training, carrying out fixed-point processing on the synapse weight of the image classification model to obtain a fixed-point weight; storing a current membrane potential of each neuron and an inhibition flag of the neuron in a first memory; and storing the fixed point weights in a second memory; the step of obtaining the synaptic weight membrane potential increment from the memory based on the first address information and the second address information comprises: obtaining a current membrane potential of the neuron from the first memory based on the first address information; acquiring the fixed point weight from the second address information based on the second address information; determining the synaptic weight membrane potential increment based on the current membrane potential of the neuron and the fixed point weight.
Specifically, first address information (membrane potential memory address) and second address information (weight memory address) of the neuron to be updated are obtained based on the obtained position information of the pulse signal and the size information of the convolution layer.
And obtaining the area positions of all convolutional layer pulse neurons needing to be updated according to the position of the input pulse signal and the size information of the convolutional cores in the convolutional layers. The region position is a rectangular region centered on the input pulse signal position and having a size equal to the size of the convolution kernel in the convolution layer. In particular, for a convolutional layer structure containing multiple convolutional channels, the range of the neuron to be updated is a three-dimensional region formed by overlapping multi-channel two-dimensional structure regions.
After the updated region is obtained, traversal updating is carried out on all neurons to be updated in the region according to a specified sequence. First memory read address information (read address of the neuron membrane potential memory) and second memory read address information (read address of the weight memory) are generated in order.
And obtaining the operation elements required by the current neuron membrane potential updating from the memories according to the address information, obtaining the original membrane potential and the inhibition flag bit from the first memory, and obtaining the weight from the second memory. First, checking the neuron inhibition flag bit, if the neuron is not inhibited, starting the membrane potential updating operation to obtain the updated membrane potential value.
And judging whether the updated membrane potential is larger than a threshold value. If the number of the neurons is larger than the threshold value, the currently updated neurons send pulses, the pulse information is the position information (coordinates in the convolutional layer) of the neurons, and the inhibition flag position is set to be zero, so that the neurons are prevented from sending pulses again. The updated membrane potential and the suppression flag are restored to the read position in the first memory, and the membrane potential update process for the single neuron ends.
Specifically, the trained network synapse weight is fixed-point processed, and the classification accuracy is verified, because the network synapse weight is limited to 0-1 during training, N-bit unsigned fixed-point numbers can be adopted to represent the weight, wherein 1 integer bit, and N-1 decimal bits. In determining the fixed point number of bits, a trade-off needs to be made between the fixed point number of bits and the network classification accuracy. Finally, the selected weight is represented by 2-bit fixed point number, and the corresponding network classification precision is reduced by 1.3% compared with the non-fixed point condition.
In one embodiment, the input pulse may be processed as time, and the neural membrane potential in the convolution layer may be updated using pulse convolution mapping. In one embodiment, the input pulse, i.e., the pulse signal, includes three parts, which are the channel to which the pulse belongs, the row position where the pulse is located, and the column position where the pulse is located. By expanding the bit width of the memory, the membrane potential at the same position of all channels and the suppression flag bit at the position are stored in the memory address of the same first memory. And storing the fixed point weight values at the same position in the same address which is the second memory.
The neuron membrane potential updating process is to update the original membrane potential of the neuron to be updated by using a prespecified synapse weight. The specific operation is as follows: the original membrane potential and the weight calculated according to the mapping algorithm are respectively taken out from the two memories, and the original membrane potential (which is 0 in the most initial case) plus the membrane potential increment (weight) is the new membrane potential. And judging whether the newly generated membrane potential reaches a threshold value or not, and then accessing the newly generated membrane potential to the memory position of the original membrane potential. This operation continues throughout the image classification process until the classification is complete.
Specifically, referring to fig. 3, the convolutional layer is composed of an address generation module, a first memory, a second memory, an addition calculation module, and a detection comparison module. The bit width of the second memory is determined by the number of convolution channels, and for K channel M × M convolution, under the condition that the fixed point bit number takes 2, the storage bit width of the second memory is 2K bits. The depth of the second memory is M, which is used for storing all the fixed point weight information. The bit width B of the first memory depends on the threshold of the film potential, and if the threshold of the convolution layer is Vth, 2B>Vth and 2B-1<Vth. And meanwhile, the inhibition is realized, the highest bit of the membrane potential needs to store an inhibition flag bit, and the bit width of the first memory is B x K + 1. In the software model, the boundary zero filling operation is needed to be carried out on the input image during convolution, namely, the neuron with the suppressed boundary is added in hardware, and the width of the boundary zero filling is wide
The first memory depth is (W +2 pad) (H +2 pad). The input pulse signal is processed by the address generation module, and first address information (the first address information is address information in the first memory) and corresponding second address information (the second address information is address information in the second memory) of the neuron to be updated are obtained through calculation according to the size information of the current convolutional layer. And extracting the membrane potential of the neuron from the first memory and the second memory according to the first address information and the second address information, and sending the membrane potential to an addition calculation module to update the membrane potentials of multiple channels in parallel. Specifically, the current membrane potential of the neuron is read from the first memory according to the first address information, the fixed point weight is read from the second memory according to the second address information, the current membrane potential and the fixed point weight of the neuron are sent to the addition calculation module to be added, so that the membrane potential increment of the synaptic weight is obtained, and then the current membrane potential of the neuron to be updated is updated by utilizing the membrane potential increment of the synaptic weight.
By arranging the storage space of the membrane potential and the weight, the multichannel parallel convolution operation can be realized when multichannel convolution is carried out. Specifically, because both the neuron membrane potentials and the weights are agreed to be fixed bit widths in the software model, when the convolutional layer structure contains multi-channel convolution, the neuron membrane potentials and the weights of a plurality of channels are combined and arranged in the memory units at corresponding positions. The combination process is to splice a plurality of low-order data to form high-order data, and the high-order data is arranged in a storage address of the memory. During the splicing operation, the bit width information stored in the first memory storage unit is the film potential at the same position of each convolution channel and the splicing data of the suppression flag bit. The bit width information stored in the second memory storage unit is splicing data of convolution kernel (weight) information of each channel of the multi-convolution layer. Therefore, when a multi-channel convolution operation is performed in the convolution layer, the original neuron membrane potentials and the synaptic weights to be updated of a plurality of channels can be taken out from the first memory and the second memory in the access period of one memory, the obtained spliced data is split, and then the membrane potential updating operation is performed on each channel neuron through the parallel adder arrays. The updated membrane potential and the inhibition flag are restored to the read position in the first memory, and the membrane potential updating process for the single neuron is ended.
Specifically, referring to fig. 6, in a convolutional layer having a plurality of convolutional channels, the convolutional operation occurs here by performing an update process of adding an original film potential and a weight to be updated in parallel to obtain a new film potential in each channel. The first row in fig. 6 is an address representation form of the input pulse, and the convolutional layer calculates the storage address of the neuron membrane potential to be updated in the first memory and the storage address of the weight to be updated in the second memory according to the address information of the pulse. The second row of fig. 6 is a storage format of the weights in the second memory, and the weight at each position of each convolution channel is represented by a binary number n that determines the bit width, and each binary number n is narrow-bit-width data (low-bit-width data). The first convolution channel is weighted chn1, and the second convolution channel is represented as weighted chn 2-a total of M channels, requiring the use of M n-bit binary representations. And then, the binary numbers are spliced to form binary wide-bit data with M x n bits. This number is then stored in a memory location in the second memory. Similarly, for the film potential information in the first memory, as shown in the third row of fig. 6, each channel is represented as MP channel1(MP chn1) to MP channel M (MP chnM), and the corresponding M × M bit binary numbers are also stored by using this concatenation method. In addition, a suppression flag bit is spliced in the highest bit of the first memory to implement a suppression mechanism of the impulse neural network, and the storage bit width of the first memory is M × M + 1.
When a multi-channel convolution operation occurs and the membrane potential needs to be updated, the weight values are read from the second memory first, and the membrane potential is read from the first memory. Splitting the obtained splicing data, and then performing membrane potential updating operation of each channel. And splicing the updated membrane potential again, and then storing the membrane potential back to the first memory.
According to the memory arrangement and the membrane potential updating strategy, the time delay overhead of reading and storing of the memory can be saved through reasonable memory arrangement under the condition that no extra memory overhead is generated, the multichannel convolution is parallel updating of the membrane potential of the pulse neuron, the multichannel convolution has the same processing time as the single channel convolution, and the processing speed of image classification is improved.
Further, the detection and comparison module further judges whether the updated membrane potential reaches a threshold value, and if the updated membrane potential reaches the threshold value, a pulse is sent out and the inhibition flag bit of the neuron is modified. The inhibit flag is stored in a first memory, where the inhibit flag is modified such that the inhibit flag indicates that the current neuron is unable to issue a pulse signal. And finally, storing the updated membrane potential into the first memory according to the read address, and finishing the updating process of the convolution operation on the single membrane potential. For the convolution operation of one input pulse, the membrane potential of a plurality of convolution layer neurons needs to be updated, and the pipeline operation can be realized in the process, so that the updating efficiency of the membrane potential is improved.
Referring to fig. 4, the pooling layer includes a pulse signal generating module, a third memory, a suppression determining module, and a suppression generating module. The data channel and the pulse signal input flag bit of the input pulse signal are connected to the pulse signal generation module, and the input data of the pulse signal generation module comes from the buffer layer behind the previous layer. The pulse signal generation module caches the generated pooling pulse information to the inhibition judgment module, simultaneously generates a reading address of an inhibition zone bit and stores the inhibition zone bit to a third memory, and the inhibition zone bit read out by the third memory is connected with the inhibition judgment module. And the inhibition judging module transmits the pulse signal to the inhibition generating module under the condition of no inhibition, changes the position inhibition flag bit and stores the position inhibition flag bit in a third memory.
Since the pooling layer neurons also need to follow the inhibition mechanism of pulse emission, it is necessary to store the inhibition flag bits of the pooling layer neurons that emit pulses using a third memory, and to implement the neuron inhibitory function using an inhibition flag bit determination circuit. The third memory bit in the pooling layer is only one bit wide and has a depth of H W, which is the same size as the pooling layer neurons. If no inhibition occurs, a pulse is sent to the next layer by a pulse transmitting circuit, and the inhibition zone bit is modified.
Because the processing period among all layers is determined by parameters, the generation frequency of the pulse is not uniform, and buffer layers are inserted among all layers of circuit modules for ensuring the stability of network operation. Specifically, buffer layers are arranged among the filter layer, the convolution layer, the pooling layer, the convolution layer and the pooling layer; temporarily storing the received data by using the buffer layer, and judging whether the processing layer corresponding to the output direction is idle or not; and responding to the idle state, and transmitting the temporarily stored data to the processing layer, wherein the processing layer comprises at least one of a convolutional layer, a pooling layer, a convolutional layer and a pooling layer.
Specifically, it is assumed that the image classification model of the present application includes a filter layer, a first convolution layer, a first pooling layer, a second convolution layer, a second pooling layer, a third convolution layer, and a third pooling layer. The image classification model after the buffer layer is arranged is as follows: the buffer layer comprises a filter layer, a first buffer layer, a first coiling layer, a second buffer layer, a first pooling layer, a third buffer layer, a second coiling layer, a fourth buffer layer, a second pooling layer, a fifth buffer layer, a third coiling layer, a sixth buffer layer and a third pooling layer.
Further, in a specific application, as shown in fig. 5, when the image classification model is deployed, the MCU, the BUS and BUS INTERFACE, the Input and output data signal channel AER Input Date and the control signal channel AER Input Flag, and the Global controller in the pulse convolution neural network module need to be set. One end of the BUS BUS is connected with the MCU, and the other end of the BUS BUS is connected with the BUS INTERFACE BUS INTERFACE of the pulse convolution neural network. The BUS INTERFACE BUS INTERFACE generates a control signal AER Input Flag after receiving Data content sent by the BUS BUS, the AER pulse signal is transmitted through a Data path AER Input Data, a classification Result and a Flag bit Feature Result Output Flag of the classification Result are received through Feature Result Output, and the channels are connected with a global controller in a pulse convolution neural network module. The Global controller is connected with the AER RAM, provides write data and addresses of the AER RAM, simultaneously generates a propagation enable signal PROP EN, and feeds back the Feature Vector obtained by propagation to the Global controller through the pulse convolution neural network module.
The buffering of data is achieved without affecting the pulse propagation order. And after the processing period of the current pulse is finished, reading out the pulse signal of the next neuron to be processed from the buffer layer. The depth of the buffer layer of each layer is determined by software verification, and the buffered pulse signals are guaranteed not to overflow and lose. The width of the buffer layer is determined by the width of the input pulse signal connected to the preceding stage of the buffer layer. Specifically, taking the example of inserting the buffer layer in the filter layer and the convolutional layer, the output of the filter layer is buffered in the buffer layer, whether the convolutional layer is in space or not is determined, and if the convolutional layer is idle, the output of the filter layer is transmitted to the convolutional layer. It can be understood that the buffer layer may adopt a first-in first-out queue, and the plurality of data are arranged in order, and when the convolutional layer is idle, the data with the earliest timestamp is transmitted to the convolutional layer for processing. The processing procedure of the buffer layer between other layers is the same, and is not described herein again.
According to the image classification method, the buffer layer is inserted between each layer of module and used for caching the pulse signals generated by the upper layer, so that each functional layer is not interfered with each other when processing the pulse information, a pipeline structure is formed, and the processing speed of the circuit on the pulse information is improved. The depth of the buffer layer is calculated in a software model, and allowance is reserved to ensure that all pulse information cannot be lost. And when all the functional layers enter an idle state and all the buffer layers are empty, considering that the pulse propagation of the current input image is finished, and obtaining the maximum value of the film potential of each channel pulse of the third convolutional layer in the global pooling layer as a 200-dimensional characteristic vector of the input image.
Specifically, on the basis of not affecting the image classification performance, the image classification method performs fixed-point processing on a weight parameter obtained by adopting pulse synapse plasticity training based on a pulse convolution neural network, designs a corresponding circuit structure and completes the pulse propagation process; based on an address-event representation method, network operation in the pulse transmission process is simplified, and the processing power consumption of a circuit is reduced; based on the multichannel parallel and pipeline design, the processing time of the circuit for the input image is shortened.
Aiming at the pulse convolution neural network, the method utilizes an Address-Event Representation (AER) method to realize the pulse propagation process in the network, addresses the connected post-synaptic neurons through a deconvolution mapping algorithm and updates the membrane potential of the post-synaptic neurons, thereby avoiding convolution operation in the full-image range and saving a large amount of computing resources. In addition, the circuit combines the parallel computing characteristic of each module of the digital circuit with the parallel characteristic of membrane potential updating in the convolution process, and simultaneously updates the membrane potentials of a plurality of convolution channels in one period, thereby greatly improving the computing speed of convolution operation. Meanwhile, FIFO is inserted into each layer of computing unit to perform data caching to form a pipeline structure, so that data exchange among modules in different processing periods is realized, the processing speed of a neural network on a single input image is improved, the power consumption is reduced, and the acceleration function of the pulse convolution neural network is realized.
Fig. 7 is a schematic structural diagram of an electronic device according to an embodiment of the invention. The electronic device comprises a memory 82 and a processor 81 connected to each other.
The memory 82 is used to store program instructions implementing the method of any one of the above.
The processor 81 may also be referred to as a CPU (Central Processing Unit). Processor 81 may be an integrated circuit chip having signal processing capabilities. Processor 81 may also be a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The storage 82 may be a memory bank, a TF card, etc., and may store all information in the electronic device, including the input raw data, the computer program, the intermediate operation results, and the final operation results. It stores and retrieves information based on the location specified by the controller. With the memory, the electronic device can only guarantee normal operation if it has a memory function. The storage of electronic devices can be classified into a main storage (internal storage) and an auxiliary storage (external storage) according to the use, and also into an external storage and an internal storage. The external memory is usually a magnetic medium, an optical disk, or the like, and can store information for a long period of time. The memory refers to a storage component on the main board, which is used for storing data and programs currently being executed, but is only used for temporarily storing the programs and the data, and the data is lost when the power is turned off or the power is cut off.
Please refer to fig. 8, which is a schematic structural diagram of a readable storage medium according to an embodiment of the present invention. A computer readable storage medium storing a program file executable to implement the method described above.
In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus implementation methods are merely illustrative, e.g., the division of modules or units into only one logical functional division, and other division methods may be implemented in practice, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, devices or units, and may be in an electrical, mechanical or other form.
The units described as separate parts may or may not be physically separate, and the parts displayed as units may or may not be physical units, i.e. may be located in one place, or may also be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the method.
In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a system server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the implementation method of the present application.
Fig. 7 is a schematic structural diagram of a computer-readable storage medium according to the present invention. The storage medium of the present application stores a program file 91 capable of implementing all the methods, where the program file 91 may be stored in the storage medium in the form of a software product, and includes several instructions to enable a computer device (which may be a personal computer, a server, or a network device) or a processor (processor) to execute all or part of the steps of each implementation method of the present application. The aforementioned storage device includes: various media capable of storing program codes, such as a usb disk, a mobile hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, or terminal devices, such as a computer, a server, a mobile phone, and a tablet.
The above description is only an implementation method of the present invention, and not intended to limit the scope of the present invention, and all equivalent structures or equivalent processes performed by the contents of the present specification and the accompanying drawings, or directly or indirectly applied to other related technical fields, are all included in the scope of the present invention.
Claims (10)
1. A method of image classification, the method comprising:
acquiring an image to be classified;
processing an image to be classified by using an image classification model, and determining the type of the image to be classified; and updating the current membrane potential of the neuron of the image classification model by using the synapse weight membrane potential increment in the process of image processing of the image classification model, wherein the synapse weight membrane potential increment is stored in a memory.
2. The image classification method according to claim 1, characterized in that the image to be classified is processed by an image classification model, and the step of determining the type of the image to be classified comprises:
filtering the image to be classified by using a filtering layer, and coding to obtain a first output result;
processing the first output result by utilizing the cascaded convolutional layer, pooling layer, convolutional layer and pooling layer to obtain a second output result;
determining the type of the image to be classified based on the second output result.
3. The image classification method according to claim 2, wherein the step of processing the first output result with a convolutional layer comprises:
acquiring synaptic weight membrane potential increment of a neuron to be updated from the memory;
updating the current membrane potential of the neuron to be updated in the convolutional layer by using the synapse weight membrane potential increment;
and processing the first output result by using the updated convolutional layer.
4. The image classification method according to claim 3, wherein the step of obtaining synaptic weight membrane potential increments from the memory comprises:
acquiring a pulse signal, wherein the pulse signal comprises position information of the pulse signal;
calculating to obtain first address information and second address information of the neuron to be updated according to the size information of the convolutional layer based on the pulse signal;
and acquiring the synaptic weight membrane potential increment from the memory based on the first address information and the second address information.
5. The image classification method according to claim 4, wherein the step of updating the current membrane potential of the neuron to be updated in the convolutional layer by using the synaptic weight membrane potential increment further comprises:
determining whether the updated membrane potential is greater than a threshold;
in response to the updated membrane potential being greater than the threshold, the neuron to be updated pulses and modifies an inhibit flag of the neuron to be updated such that the inhibit flag characterizes the neuron to be updated as not being able to pulse;
storing an updated film potential into the memory based on the first address information and the second address information in response to the updated film potential being less than or equal to the threshold.
6. The image classification method according to claim 1, characterized in that the method further comprises:
acquiring a training sample set, wherein each sample image in the training sample set is marked with the type of the image;
and training an initial model by using the training sample set to obtain an image classification model.
7. The image classification method according to claim 6, wherein the step of training an initial model by using the training sample set to obtain an image classification model, then comprises:
carrying out fixed point processing on the synapse weight of the image classification model to obtain a fixed point weight;
storing a current membrane potential of each neuron and an inhibition flag of the neuron in a first memory; and storing the fixed point weights in a second memory;
the step of obtaining the synaptic weight membrane potential increment from the memory based on the first address information and the second address information comprises:
obtaining a current membrane potential of the neuron from the first memory based on the first address information; acquiring the fixed point weight value from the second address information based on the second address information;
determining the synaptic weight membrane potential increment based on the current membrane potential of the neuron and the fixed point weight.
8. The image classification method according to claim 2, characterized in that buffer layers are provided between the filter layer, the convolutional layer, the pooling layer, the convolutional layer, and the pooling layer;
temporarily storing the received data by using the buffer layer, and judging whether the processing layer corresponding to the output direction is idle or not;
and responding to the idle state, and transmitting the temporarily stored data to the processing layer, wherein the processing layer comprises at least one of a convolutional layer, a pooling layer, a convolutional layer and a pooling layer.
9. An electronic device comprising a processor and a memory coupled to each other, wherein,
the memory for storing program instructions for implementing the method of any one of claims 1-8;
the processor is configured to execute the program instructions stored by the memory.
10. A computer-readable storage medium, characterized in that a program file is stored, which program file can be executed to implement the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210623187.5A CN115100458A (en) | 2022-06-02 | 2022-06-02 | Image classification method and related device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210623187.5A CN115100458A (en) | 2022-06-02 | 2022-06-02 | Image classification method and related device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115100458A true CN115100458A (en) | 2022-09-23 |
Family
ID=83289258
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210623187.5A Pending CN115100458A (en) | 2022-06-02 | 2022-06-02 | Image classification method and related device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115100458A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170213129A1 (en) * | 2016-01-26 | 2017-07-27 | Samsung Electronics Co., Ltd. | Recognition apparatus based on neural network and method of training neural network |
| CN107077636A (en) * | 2014-03-24 | 2017-08-18 | 高通股份有限公司 | COLD neuron spike timing backpropagations |
| CN110826602A (en) * | 2019-10-23 | 2020-02-21 | 中国科学院自动化研究所 | Image classification method and system based on membrane potential regulation and control pulse neural network |
| CN111723726A (en) * | 2020-06-17 | 2020-09-29 | 苏州大学 | Identification method of identification model of drying agent packaging bag of pulse neurolemma system |
| WO2021143066A1 (en) * | 2020-01-19 | 2021-07-22 | 北京大学 | Target recognition method, device, and system, and computer readable storage medium |
| CN113902092A (en) * | 2021-09-02 | 2022-01-07 | 四川晟锦汇科技有限公司 | Indirect supervised training method for impulse neural network |
-
2022
- 2022-06-02 CN CN202210623187.5A patent/CN115100458A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107077636A (en) * | 2014-03-24 | 2017-08-18 | 高通股份有限公司 | COLD neuron spike timing backpropagations |
| US20170213129A1 (en) * | 2016-01-26 | 2017-07-27 | Samsung Electronics Co., Ltd. | Recognition apparatus based on neural network and method of training neural network |
| CN110826602A (en) * | 2019-10-23 | 2020-02-21 | 中国科学院自动化研究所 | Image classification method and system based on membrane potential regulation and control pulse neural network |
| WO2021143066A1 (en) * | 2020-01-19 | 2021-07-22 | 北京大学 | Target recognition method, device, and system, and computer readable storage medium |
| CN111723726A (en) * | 2020-06-17 | 2020-09-29 | 苏州大学 | Identification method of identification model of drying agent packaging bag of pulse neurolemma system |
| CN113902092A (en) * | 2021-09-02 | 2022-01-07 | 四川晟锦汇科技有限公司 | Indirect supervised training method for impulse neural network |
Non-Patent Citations (5)
| Title |
|---|
| SRINIVASAN, G 等: "Spike Timing Dependent Plasticity Based Enhanced Self-Learning for Efficient Pattern Recognition in Spiking Neural Networks", 《2017 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN)》, 2 April 2018 (2018-04-02), pages 1847 - 1854 * |
| YOUSEFI, A 等: "Supervised Learning in a Single Layer Dynamic Synapses Neural Network", 《2011 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN)》, 1 January 2011 (2011-01-01), pages 2250 - 2257 * |
| YUEQI ZHANG等: "An AER-based spiking convolution neural network system for image classification with low latency and high energy efficiency", 《NEUROCOMPUTING》, vol. 564, 7 January 2024 (2024-01-07), pages 126984 - 126994 * |
| 付强: "脉冲神经网络模型优化及其应用", 《CNKI中国优秀硕士毕业论文全文库（信息科技辑）》, no. 1, 15 January 2019 (2019-01-15), pages 140 - 606 * |
| 张梦炜: "基于局部变量STDP的脉冲神经网络在线监督学习算法研究", 《CNKI中国优秀硕士毕业论文全文库（信息科技辑）》, no. 12, 15 December 2021 (2021-12-15), pages 140 - 111 * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10096134B2 (en) | Data compaction and memory bandwidth reduction for sparse neural networks | |
| CN108416327B (en) | Target detection method and device, computer equipment and readable storage medium | |
| US20210089871A1 (en) | Processing system and method for binary weight convolutional neural network | |
| EP4053718A1 (en) | Watermark information embedding method and apparatus | |
| CN111414987A (en) | Training method and training device for neural network and electronic equipment | |
| CN112163601A (en) | Image classification method, system, computer device and storage medium | |
| CN112395247A (en) | Data processing method and storage and calculation integrated chip | |
| JP6891626B2 (en) | Information processing equipment, information processing system, information processing program and information processing method | |
| US20190080226A1 (en) | Method of designing neural network system | |
| CN111768458A (en) | Sparse image processing method based on convolutional neural network | |
| CN109145107B (en) | Theme extraction method, device, medium and equipment based on convolutional neural network | |
| CN111210004B (en) | Convolution calculation method, convolution calculation device and terminal equipment | |
| CN112200310B (en) | Intelligent processor, data processing method and storage medium | |
| KR20220039313A (en) | Method and apparatus for processing neural network operation | |
| CN117574970A (en) | Inference acceleration method, system, terminal and medium for large-scale language model | |
| CN114138231B (en) | Method, circuit and SOC for executing matrix multiplication operation | |
| US10990525B2 (en) | Caching data in artificial neural network computations | |
| CN115100458A (en) | Image classification method and related device | |
| CN110009091B (en) | Optimization of learning network in equivalence class space | |
| CN113361621B (en) | Method and device for training model | |
| US20230072535A1 (en) | Error mitigation for sampling on quantum devices | |
| CN109799483A (en) | A kind of data processing method and device | |
| CN112784967B (en) | Information processing method and device and electronic equipment | |
| CN110378469B (en) | SCNN inference device based on asynchronous circuit, PE unit, processor and computer equipment thereof | |
| CN114090466A (en) | Instruction processing device and method, computer equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |