CN112712170A - Neural morphology vision target classification system based on input weighted impulse neural network - Google Patents
Neural morphology vision target classification system based on input weighted impulse neural network Download PDFInfo
- Publication number
- CN112712170A CN112712170A CN202110025992.3A CN202110025992A CN112712170A CN 112712170 A CN112712170 A CN 112712170A CN 202110025992 A CN202110025992 A CN 202110025992A CN 112712170 A CN112712170 A CN 112712170A
- Authority
- CN
- China
- Prior art keywords
- event
- pulse
- input
- module
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013528 artificial neural network Methods 0.000 title claims abstract description 55
- 230000001537 neural effect Effects 0.000 title claims abstract description 25
- 238000007781 pre-processing Methods 0.000 claims abstract description 30
- 230000000007 visual effect Effects 0.000 claims abstract description 23
- 238000010276 construction Methods 0.000 claims abstract description 22
- 238000000034 method Methods 0.000 claims abstract description 15
- 210000002569 neuron Anatomy 0.000 claims description 31
- 230000006870 function Effects 0.000 claims description 20
- 230000002123 temporal effect Effects 0.000 claims description 15
- 238000003062 neural network model Methods 0.000 claims description 12
- 230000002776 aggregation Effects 0.000 claims description 10
- 238000004220 aggregation Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 7
- 238000001208 nuclear magnetic resonance pulse sequence Methods 0.000 claims description 7
- 230000004931 aggregating effect Effects 0.000 claims description 6
- 230000008859 change Effects 0.000 claims description 6
- 230000006835 compression Effects 0.000 claims description 6
- 238000007906 compression Methods 0.000 claims description 6
- 238000006116 polymerization reaction Methods 0.000 claims description 5
- 238000011176 pooling Methods 0.000 claims description 5
- 230000004913 activation Effects 0.000 claims description 4
- 230000005540 biological transmission Effects 0.000 claims description 3
- 238000012421 spiking Methods 0.000 claims description 3
- 230000008569 process Effects 0.000 claims 1
- 238000004364 calculation method Methods 0.000 abstract 1
- 238000013459 approach Methods 0.000 description 4
- 238000009825 accumulation Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000006872 improvement Effects 0.000 description 3
- 210000005036 nerve Anatomy 0.000 description 3
- 238000012549 training Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000002474 experimental method Methods 0.000 description 2
- 239000012528 membrane Substances 0.000 description 2
- 230000000877 morphologic effect Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005265 energy consumption Methods 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 210000000225 synapse Anatomy 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Software Systems (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biophysics (AREA)
- Biomedical Technology (AREA)
- Mathematical Physics (AREA)
- Computational Linguistics (AREA)
- Health & Medical Sciences (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Image Analysis (AREA)
Abstract
The invention discloses a neural morphology visual target classification system based on an input weighted pulse neural network, which belongs to the technical field of artificial neural networks and comprises the following four method modules: the system comprises a data preprocessing module, a network construction module, a learning module and an inference module. The data preprocessing module is used for converting the acquired event camera asynchronous output space-time pulse event stream into an event frame sequence; the network construction module is used for constructing the input weighting unit and the impulse neural layer unit into an input weighting impulse neural network according to a certain network connection mode; the learning module is used for learning the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generating a model file; the reasoning module is used for loading the network model file output by the learning module to perform feedforward network calculation. The invention can lead the neural morphology vision classification pulse neural network to have low time delay and keep higher performance.
Description
Technical Field
The invention belongs to the field of deep learning in machine learning, and particularly relates to a neural morphology visual target classification system based on an input weighted impulse neural network.
Background
The event camera is an asynchronous neuromorphic visual sensor that produces a paradigm shift in the manner in which visual information is acquired. Different from the traditional vision sensor which samples light at a fixed moment, the event camera samples light according to scene dynamic, and generates a pulse event stream by asynchronously measuring the brightness change of each pixel point, wherein the pulse event stream encodes the time, the position and the brightness change polarity of the brightness change. The pulse Neural Networks (SNNs) are a new generation Artificial Neural network inspired by a brain operation mechanism and taking a pulse sequence as a data transmission form, and have the advantages of ultra-low time delay, low energy consumption and the like compared with the traditional Artificial Neural Networks (ANNs). Event camera high time resolution (microsecond level) event stream output, combined with an impulse neural network with ultra-low latency, has great potential in some computer vision application scenarios where efficiency and power consumption are required.
Theoretically, an Event-by-Event (Event-by-Event) processing mode is adopted for the pulse Event stream output by the neuromorphic visual sensor, and the internal state of the pulse neural network is changed due to the input of each Event, so that the output with the minimum time delay can be obtained. However, since the amount of information contained in a single event is extremely small, the output result at the lowest latency tends to be poor. Another approach is to somehow aggregate the pulse Event stream in a period of time into a new Event Frame (Event Frame), where the amount of information contained in the Event Frame is much larger than that of a single Event, which enables the network to obtain better performance, but introduces some delay. Therefore, new neuromorphic visual algorithms need to be researched and developed to achieve a balance between low latency and high performance.
Object recognition and classification is an important task of neuromorphic vision. The neural morphology visual target classification supervision learning method based on pulse frequency coding mainly comprises two methods:
(1) the conversion-based method: and converting the trained deep neural network (such as a convolutional neural network and a fully-connected neural network) into an impulse neural network by using a weight adjusting and normalizing method, and applying the impulse neural network to the neural morphology visual target classification. This approach can achieve comparable accuracy to deep neural networks, but has inherent limitations, such as limited use of activation functions and no utilization of temporal information in the event stream.
(2) Pulse-based methods: learning of the weights of synapses is performed by back-propagating the training errors in Time and space, and a representative algorithm is a Time-based back-propagation algorithm (BPTT). This approach has higher computational efficiency, but the accuracy in neuromorphic visual target classification cannot exceed the transform-based approach.
The main difficulty in solving the target classification and identification task of the neuromorphic vision lies in how to effectively extract the spatio-temporal characteristics from the pulse event stream spatio-temporal information by fully utilizing the pulse event stream spatio-temporal information so as to obtain higher classification precision. In the prior art, a pulse neural network based on a LIF neuron model updates weights by using a space-time error back propagation learning algorithm, so that a researcher can use a GPU (graphics processing unit) to perform accelerated training and can also use training tools such as a very mature Pythroch in deep learning. However, this method does not distinguish between the inputs to the network, which may affect the performance of the network to some extent. In fact, the inputs at different times contain different signal-to-noise ratios, giving the same input weight to all the input times weakens the network's ability to extract valid spatio-temporal features.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides a neural morphological vision target classification system based on an input weighted impulse neural network, impulse event stream data is firstly aggregated into event frame sequence data, then an input weight is provided for each event frame according to the difference of the information content contained in the event frame, and finally the weighted event frame is used as a new input of the impulse neural network, so that the impulse neural network can adaptively focus on the event frame which has a great influence on a classification result in the event frame sequence, and the accuracy of a target classification and identification task of the neural morphological vision can be improved while only a small amount of data is needed.
In order to achieve the purpose, the invention provides the following technical scheme:
the neural morphology visual target classification system based on the input weighted pulse neural network comprises a data preprocessing module, a network construction module, a learning module and an inference module which are connected in sequence;
the data preprocessing module is used for acquiring a space-time pulse event stream asynchronously output by an event camera, and space-time pulse events in the space-time pulse event stream are described by adopting an address event expression protocol; the system is used for aggregating the spatio-temporal pulse event stream into an event frame sequence according to the time resolution dt' of an event camera, and the event frame sequence is described by tensor; aggregating the event frame sequences with the time resolution dt' into new event frame sequences according to the set time resolution dt, wherein the event frame data are described by tensors; the event frame sequence tensor data is used as the output of the data preprocessing module;
the network construction module consists of an input weighting unit and a pulse neural layer unit and is used for constructing an input weighting pulse neural network;
the learning module is used for learning the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generating a model file;
and the reasoning module is used for reading the input weighted pulse neural network structure configured by the network construction module, loading the model file generated by the learning module to obtain the input weighted pulse neural network parameters to obtain a trained input weighted pulse neural network model, and taking a plurality of event frames output by the data preprocessing module as the input of the input weighted pulse neural network model to obtain a reasoning result.
The further improvement of the present invention lies in that the data preprocessing module aggregates the pulse event streams into an event frame sequence according to the set temporal resolution dt and according to the spatio-temporal pulse event streams output by the event camera, and specifically includes:
the spatiotemporal pulse event stream is composed of the set E ═ E { (E)i|ei＝[xi,yi,t′i,pi]Determining; wherein eiFor the ith pulse event in the stream of pulse events, (x)i,yi) Is the pixel coordinate of the ith pulse event, t'iTime stamp, p, in the entire time stream for the ith pulse eventiThe light intensity change polarity of the ith pulse event; the time resolution of the asynchronous pulse transmission event stream of the event camera is dt' and the spatial resolution is H multiplied by W; then, new spatio-temporal event frame data are aggregated according to the set time resolution dt.
The invention further improves that the data polymerization process is carried out in two steps, and specifically comprises the following steps:
first, several events E generated at time t 'are processed based on the temporal resolution dt' of the event camerat′Assembled to tensor Xt′； wherein ,Et′＝{ei|ei＝[xi,yi,t′,pi]}，Xt′∈RH×W×2；
Second, based on the set temporal resolution dt, using the formula Xt＝f(X′t) Generating an event frame tensor X at time tt∈ RH×W×2(ii) a Wherein dt ═ β × dt', β is a polymerization time factor; x't＝{Xt′|t′∈[β×t,β×(t+1)-1]}; f is an accumulate operation, a weighted accumulate operation, or an AND or NOR operation.
The invention further improves that the network construction module consists of one or more input weighting units, a pulse neural layer unit and one or more perceptron neuron output layers; the pulse neuron adopts an LIF neuron model; the input weighting unit is composed of three steps, and specifically comprises:
firstly, an event frame X (X ═ X) obtained by a data preprocessing module is processed1,X2,...,Xt,...,XT},X∈RH ×W×2×T) As an input, it is compressed into a vector z (z ═ z) using a compression function f1,z2,...,zt,...zT},z∈RT)；Wherein, when the compression function f is an average pooling function, the t-th tensor X in XtIs compressed into a value ztThe concrete formula is as follows:
secondly, inputting the vector z into a two-layer nonlinear full-connection network to obtain an output vector s (s belongs to R)T)：
s＝σ(W2δ(W1z))
Where, δ is the ReLU activation function,
weight matrix W1 and W2Is a trainable parameter, and r is an optional hyper-parameter;
thirdly, the value s in the vector s is measuredtAs event frame XtMultiplying each element in the event frame by the weight of (2) to obtain a new event frame
As output of the input weighting unit:
the further improvement of the invention is that the learning module comprises a feedforward network computing unit, an error back propagation unit and a weight updating unit, and specifically comprises:
the feedforward network computing unit selects T event frames from an event frame sequence converted from single event stream data by using a random time clipping method as input of the feedforward network computing unit, and the T event frames are described by adopting a tensor; in particular, T generated at a temporal resolution dt in a single spatiotemporal pulse event stream datatotalRandomly extracting T event frames from the event frames as input to a feedforward network computing unit, T<Ttotal；
Calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target by the output layer of the perceptron neuron according to the output pulse sequence;
the error back propagation unit calculates the error between the output target and the set target according to the set loss function and performs back propagation;
the weight updating unit updates the weight according to the set learning rate and the set error.
The invention has the further improvement that the reasoning module consists of a model loading unit and a feedforward network computing unit; the model loading unit reads an input weighted impulse neural network structure configured by the network construction module and loads a model file generated by the learning module to obtain a trained input weighted impulse neural network model; the feedforward network computing unit is used for computing input weighting and a pulse neuron layer in the impulse neural network model according to the event stream or the event frame provided by the data preprocessing module, and then the perceptron neuron output layer obtains an output target according to output pulses of the pulse neuron layer.
Compared with the prior art, the invention has at least the following beneficial technical effects:
aiming at the problems of high time delay and low accuracy rate of classification of a pulse neural network used in the existing neural form visual target classification method, the invention improves the existing pulse neural network structure, namely, an input weighting unit is introduced to adaptively give higher weight to an event frame which has great influence on a neural form visual classification result, thereby improving the space-time characteristic extraction capability of pulse event streams, and effectively improving the neural form visual classification accuracy rate of the pulse neural network while reducing the required data volume.
Drawings
FIG. 1 is a diagram of a neuromorphic visual target classification system based on an input weighted impulse neural network according to the present invention.
Fig. 2 is an output of event stream data (waving motion) in the neuromorphic-visual data set DVS128 Gesture after passing through the data preprocessing module. That is, 200,000 microsecond pulsed event streams are aggregated into 10 event frames with a resolution of 20 milliseconds.
FIG. 3 is a network connection of an input weighted impulse neural network.
FIG. 4 is a schematic diagram of the input weighting unit.
FIG. 5 is a schematic diagram of a nerve layer unit consisting of impulses.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
As shown in fig. 1, the neural morphology visual target classification system based on the weighted impulse neural network provided by the invention is composed of a data preprocessing module S101, a network construction module S102, a learning module S103 and an inference module S104 which are connected in sequence.
The data preprocessing module S101 is composed of an impulse event stream aggregation unit and an event frame aggregation unit, and is configured to aggregate spatiotemporal impulse event streams asynchronously output by the event camera into an event frame sequence.
The input of the data preprocessing module is a space-time pulse event stream output by an event camera, and the space-time pulse event is described by adopting an address event expression protocol; the pulse event stream aggregation unit is used for aggregating the pulse event streams into an event frame sequence according to the time resolution dt' of the event camera; the event frame aggregation unit is used for aggregating an event frame sequence with the resolution dt' into new event frame sequence data according to the set time resolution dt, wherein the event frame data are described by tensors; and the event frame sequence output by the event frame aggregation unit is used as the output of the data preprocessing module.
The spatio-temporal pulse event stream is composed of the set E ═ E { (E) }i|ei＝[xi,yi,t′i,pi]Determining; wherein eiFor the ith pulse event in the stream of pulse events, (x)i,yi) Is the pixel coordinate of the ith pulse event, t'iTime stamp, p, in the entire time stream for the ith pulse eventiThe intensity change polarity for the ith pulse event. The temporal resolution of the asynchronous pulse event stream of the event camera is dt' and the spatial resolution is H x W. Typically, the temporal resolution of an event camera is in the order of microseconds, i.e., dt' is 1 μ s.
The pulse event stream aggregation unit is configured to aggregate pulse event streams into an event frame sequence according to a time resolution dt' of an event camera, and specifically includes: several events E generated at time t 'based on the temporal resolution dt' of the event camerat′Assembled to tensor Xt′. wherein ,Et′＝{ei|ei＝[xi,yi,t′,pi]}，Xt′∈RH×W×2。
The event frame aggregation unit is configured to aggregate an event frame sequence with a resolution dt' ═ 1 μ s into new event frame sequence data according to a set time resolution dt, and specifically includes: based on the set temporal resolution dt, using formula Xt＝f(X′t) Generating an event frame tensor X at time tt∈RH×W×2. Wherein dt ═ β × dt', β is a polymerization time factor; x't＝{Xt′|t′∈[β×t,β× (t+1)-1]}; f may be an accumulation operation, a weighted accumulation operation, or an AND or NOR operation, etc.
Fig. 2 shows the output of the data preprocessing module when the input of a data (waving motion) in the data set DVS128 gettrue is a 200,000 microsecond stream of pulse events, the aggregation method is an or operation, and the aggregation time factor is 20,000. That is, fig. 2 shows 10 event frames with a resolution of 20 ms, which are aggregated by event streams.
When dt' ═ 1 μ s, β ═ 3, one example of an "or" polymerization method is as follows:
an example of the accumulation operation when dt ═ 1 μ s and β ═ 3 is as follows:
the network construction module S102 is composed of an input weighting unit and an impulse neural layer unit, and is configured to configure an input weighted impulse neural network. The network construction module consists of one or more input weighting units, a pulse nerve layer unit (a pulse nerve layer is formed by fully-connected or convoluted pulse neurons) and one or more sensor nerve cell output layers; the pulse neuron adopts a LIF neuron model. Fig. 3 shows a network connection formed by the input weighting unit and the pulse neuron layer unit.
The input weighting unit is shown in fig. 4. The unit is composed of three steps, and specifically comprises:
firstly, an event frame X (X ═ X) obtained by a data preprocessing module is processed1,X2,...,Xt,...,XT},X∈RH ×W×2×T) As an input, it is compressed into a vector z (z ═ z) using a compression function f1,z2,...,zt,...zT},z∈RT). Wherein the compression function f may be an average pooling function, a maximum pooling function, or the like. The t-th tensor X in X when f is the average pooling functiontIs compressed into a value z according to the following formulat：
Secondly, inputting the vector z into a two-layer nonlinear full-connection network to obtain an output vector s (s belongs to R)T)：
s＝σ(W2δ(W1z))
Where, δ is the ReLU activation function,sigma is a function of Sigmoid and is,
is a matrix of weights that can be trained,
is a trainable weight matrix and r is an optional parameter.
Thirdly, the value s in the vector s is measuredtAs event frame XtMultiplying each element in the event frame by the weight of (2) to obtain a new event frame
As output of the input weighting unit:
the spiking neural layer unit is composed of a LIF neuron model, as shown in fig. 5. Specifically, the LIF neuron layer expression is as follows:
wherein the function g is a step function,
denotes the Hadamard product, xt,n-1Event frame, h, input for layer n-1 at time tt-1,nIs the internal state quantity of the nth layer at the time t-1; u. oft,nIs the membrane potential; the membrane potential is related to the neuron threshold uthComparing and comparing xt,nIs passed as spatial output to the next layer, ht,nAs a temporal input of LIF neuronsThe output is passed to the next time. Current weight matrix
And spatial input xt,n-1When matrix multiplication is adopted, the LIF neuron model is a full-connection LIF neuron model; current weight matrix
And spatial input xt,n-1When convolution operation is adopted, the LIF neuron model is a convolution LIF neuron model.
The learning module S103 is composed of a feedforward network computing unit, an error back propagation unit and a weight updating unit, and is used for learning the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generating a model file.
The feedforward network computing unit selects T event frames from an event frame sequence converted from single event stream data by using a random time clipping method as input of the feedforward network computing unit, and the T event frames are described by adopting a tensor; in particular, T generated at a temporal resolution dt in a single spatiotemporal pulse event stream datatotalAn event frame from which T (T) is randomly extracted< Ttotal) Taking the event frame as the input of a feedforward network computing unit; calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target by the output layer of the perceptron neuron according to the output pulse sequence;
and the error back propagation unit calculates the error between the output target and the set target according to the set loss function and performs back propagation.
The weight updating unit updates the weight according to the set learning rate and error.
The inference module S104 is composed of a model loading unit and a feedforward network computing unit. The device is used for reading an input weighted impulse neural network structure configured by the network construction module, loading a model file generated by the learning module to obtain input weighted impulse neural network parameters, obtaining a trained input weighted impulse neural network model, and taking an event frame output by the data preprocessing module as the input of the input weighted impulse neural network model to obtain a reasoning result.
And the model loading unit reads the impulse neural network structure configured by the network construction module and loads the model file generated by the learning module to obtain the trained input weighted impulse neural network model.
The feedforward network computing unit takes the event stream provided by the data preprocessing module as input, computes input weighting and a pulse neuron layer in the pulse neural network model, and then obtains an output target by the perceptron neuron output layer according to output pulses of the pulse neuron layer.
The data preprocessing module, the network construction module, the learning module and the reasoning module are connected in sequence.
To better illustrate the beneficial effects of the present invention, the following is a description of the experiment of the method of the present invention on the neuromorphic visual target classification dataset DVS128 Gesture. In the experiment, we set 60 event frames as input and test the influence of the input weighting unit on the final result on the basis of different time resolutions dt:
experimental results of convolution-based input weighted pulse neural network on DVS128 Gesture
As can be seen from the above table, when performing neuromorphic visual target classification, compared to the conventional impulse neural network, the input weighted impulse neural network can achieve higher classification performance while requiring only a small amount of data. For example, when the time resolution is 1 millisecond, only 60 milliseconds of data are required to achieve 91.28% accuracy. The input weighted impulse neural network can improve the network performance at all time resolutions.
Claims (6)
1. The neural morphology visual target classification system based on the input weighted pulse neural network is characterized by comprising a data preprocessing module, a network construction module, a learning module and an inference module which are connected in sequence;
the data preprocessing module is used for acquiring a space-time pulse event stream asynchronously output by an event camera, and space-time pulse events in the space-time pulse event stream are described by adopting an address event expression protocol; the system is used for aggregating the spatio-temporal pulse event stream into an event frame sequence according to the time resolution dt' of an event camera, and the event frame sequence is described by tensor; aggregating the event frame sequences with the time resolution dt' into new event frame sequences according to the set time resolution dt, wherein the event frame data are described by tensors; the event frame sequence tensor data is used as the output of the data preprocessing module;
the network construction module consists of an input weighting unit and a pulse neural layer unit and is used for constructing an input weighting pulse neural network;
the learning module is used for learning the input weighted impulse neural network obtained by the network construction module according to the event frame sequence obtained by the preprocessing module and generating a model file;
and the reasoning module is used for reading the input weighted pulse neural network structure configured by the network construction module, loading the model file generated by the learning module to obtain the input weighted pulse neural network parameters to obtain a trained input weighted pulse neural network model, and taking a plurality of event frames output by the data preprocessing module as the input of the input weighted pulse neural network model to obtain a reasoning result.
2. The neuromorphic visual target classification system based on an input weighted impulse neural network as claimed in claim 1, wherein the data preprocessing module aggregates the pulse event streams into a sequence of event frames according to a set temporal resolution dt from the spatiotemporal pulse event streams output by the event cameras, specifically comprising:
the spatiotemporal pulse event stream is composed of the set E ═ E { (E)i|ei＝[xi，yi，t′i，pi]Determining; it is composed ofIn (e)iFor the ith pulse event in the stream of pulse events, (x)i，yi) Is the pixel coordinate of the ith pulse event, t'iTime stamp, p, in the entire time stream for the ith pulse eventiThe light intensity change polarity of the ith pulse event; the time resolution of the asynchronous pulse transmission event stream of the event camera is dt' and the spatial resolution is H multiplied by W; then, new spatio-temporal event frame data are aggregated according to the set time resolution dt.
3. The neuromorphic visual target classification system based on an input weighted impulse neural network as claimed in claim 2, wherein the data aggregation process is performed in two steps, specifically comprising:
first, several events E generated at time t 'are processed based on the temporal resolution dt' of the event camerat′Assembled to tensor Xt′； wherein ,Et′＝{ei|ei＝[xi，yi，t′，pi]}，Xt′∈RH×W×2；
Second, based on the set temporal resolution dt, using the formula Xt＝f(X′t) Generating an event frame tensor X at time tt∈RH ×W×2(ii) a Wherein dt ═ β × dt', β is a polymerization time factor; x't＝{Xt′|t′∈[β×t，β×(t+1)-1]}; f is an accumulate operation, a weighted accumulate operation, or an AND or NOR operation.
4. The input-weighted spiking neural network-based neuromorphic visual target classification system of claim 1, wherein the network construction module is composed of one or more input weighting units and spiking neural layer units, one or more perceptron neuron output layers; the pulse neuron adopts an LIF neuron model; the input weighting unit is composed of three steps, and specifically comprises:
firstly, an event frame X (X ═ X) obtained by a data preprocessing module is processed1，X2，...，Xt，...，XT}，X∈RH×W×2×T) As an input, it is compressed into a vector z (z ═ z) using a compression function f1，z2，...，zt，...zT}，z∈RT) (ii) a Wherein, when the compression function f is an average pooling function, the t-th tensor X in XtIs compressed into a value ztThe concrete formula is as follows:
secondly, inputting the vector z into a two-layer nonlinear full-connection network to obtain an output vector s (s belongs to R)T)：
s＝σ(W2δ(W1z))
Where, δ is the ReLU activation function,
weight matrix W1 and W2Is a trainable parameter, and r is an optional hyper-parameter;
thirdly, the value s in the vector s is measuredtAs event frame XtMultiplying each element in the event frame by the weight of (2) to obtain a new event frame
As output of the input weighting unit:
5. the neuromorphic visual target classification system based on an input weighted impulse neural network as claimed in claim 1, wherein the learning module comprises a feedforward network computing unit, an error back propagation unit and a weight updating unit, and specifically comprises:
the feedforward network computing unit uses random timeSelecting T event frames from an event frame sequence converted from single event stream data as the input of a feedforward network computing unit by an interval cutting method, wherein the T event frames are described by adopting tensors; in particular, T generated at a temporal resolution dt in a single spatiotemporal pulse event stream datatotalRandomly extracting T event frames from the event frames as input of a feedforward network computing unit, wherein T < Ttotal；
Calculating an output pulse sequence according to the input weighting unit, the pulse neural layer unit and the network connection mode in sequence, and then calculating an output target by the output layer of the perceptron neuron according to the output pulse sequence;
the error back propagation unit calculates the error between the output target and the set target according to the set loss function and performs back propagation;
the weight updating unit updates the weight according to the set learning rate and the set error.
6. The neuromorphic visual target classification system based on an input weighted impulse neural network as claimed in claim 1, wherein the inference module is composed of a model loading unit and a feedforward network computing unit; the model loading unit reads an input weighted impulse neural network structure configured by the network construction module and loads a model file generated by the learning module to obtain a trained input weighted impulse neural network model; the feedforward network computing unit is used for computing input weighting and a pulse neuron layer in the impulse neural network model according to the event stream or the event frame provided by the data preprocessing module, and then the perceptron neuron output layer obtains an output target according to output pulses of the pulse neuron layer.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110025992.3A CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110025992.3A CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112712170A true CN112712170A (en) | 2021-04-27 |
| CN112712170B CN112712170B (en) | 2023-06-20 |
Family
ID=75548618
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110025992.3A Active CN112712170B (en) | 2021-01-08 | 2021-01-08 | Neuromorphic visual target classification system based on input weighted impulse neural network |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112712170B (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113408671A (en) * | 2021-08-18 | 2021-09-17 | 成都时识科技有限公司 | Object identification method and device, chip and electronic equipment |
| CN114429491A (en) * | 2022-04-07 | 2022-05-03 | 之江实验室 | Pulse neural network target tracking method and system based on event camera |
| CN114723009B (en) * | 2022-04-12 | 2023-04-25 | 重庆大学 | Data representation method and system based on asynchronous event stream |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150278685A1 (en) * | 2014-03-31 | 2015-10-01 | Qualcomm Incorporated | Probabilistic representation of large sequences using spiking neural network |
| CN107403154A (en) * | 2017-07-20 | 2017-11-28 | 四川大学 | A kind of gait recognition method based on dynamic visual sensor |
| CN111247801A (en) * | 2017-09-28 | 2020-06-05 | 苹果公司 | System and method for event camera data processing |
| CN111462268A (en) * | 2020-03-31 | 2020-07-28 | 北京市商汤科技开发有限公司 | Image reconstruction method and device, electronic equipment and storage medium |
| CN111582445A (en) * | 2020-04-24 | 2020-08-25 | 浙江大学 | Efficient learning system and learning method based on impulse neural network |
| CN111612136A (en) * | 2020-05-25 | 2020-09-01 | 之江实验室 | Neural morphology visual target classification method and system |
| CN112001353A (en) * | 2020-09-03 | 2020-11-27 | 杭州云栖智慧视通科技有限公司 | Pedestrian re-identification method based on multi-task joint supervised learning |
-
2021
- 2021-01-08 CN CN202110025992.3A patent/CN112712170B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20150278685A1 (en) * | 2014-03-31 | 2015-10-01 | Qualcomm Incorporated | Probabilistic representation of large sequences using spiking neural network |
| CN107403154A (en) * | 2017-07-20 | 2017-11-28 | 四川大学 | A kind of gait recognition method based on dynamic visual sensor |
| CN111247801A (en) * | 2017-09-28 | 2020-06-05 | 苹果公司 | System and method for event camera data processing |
| CN111462268A (en) * | 2020-03-31 | 2020-07-28 | 北京市商汤科技开发有限公司 | Image reconstruction method and device, electronic equipment and storage medium |
| CN111582445A (en) * | 2020-04-24 | 2020-08-25 | 浙江大学 | Efficient learning system and learning method based on impulse neural network |
| CN111612136A (en) * | 2020-05-25 | 2020-09-01 | 之江实验室 | Neural morphology visual target classification method and system |
| CN112001353A (en) * | 2020-09-03 | 2020-11-27 | 杭州云栖智慧视通科技有限公司 | Pedestrian re-identification method based on multi-task joint supervised learning |
Non-Patent Citations (4)
| Title |
|---|
| MALU ZH.等: "Supervised learning in spiking neural networks with synaptic delay-weight plasticity", 《NEUROCOMPUTING》 * |
| MALU ZH.等: "Supervised learning in spiking neural networks with synaptic delay-weight plasticity", 《NEUROCOMPUTING》, 7 October 2020 (2020-10-07), pages 103 - 118 * |
| 张玉平 等: "基于卷积计算的多层脉冲神经网络的监督学习", 《计算机工程与科学》 * |
| 张玉平 等: "基于卷积计算的多层脉冲神经网络的监督学习", 《计算机工程与科学》, 28 February 2015 (2015-02-28), pages 348 - 353 * |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113408671A (en) * | 2021-08-18 | 2021-09-17 | 成都时识科技有限公司 | Object identification method and device, chip and electronic equipment |
| CN114429491A (en) * | 2022-04-07 | 2022-05-03 | 之江实验室 | Pulse neural network target tracking method and system based on event camera |
| CN114429491B (en) * | 2022-04-07 | 2022-07-08 | 之江实验室 | Pulse neural network target tracking method and system based on event camera |
| CN114723009B (en) * | 2022-04-12 | 2023-04-25 | 重庆大学 | Data representation method and system based on asynchronous event stream |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112712170B (en) | 2023-06-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Wu et al. | Deep spiking neural network with spike count based learning rule | |
| CN112712170A (en) | Neural morphology vision target classification system based on input weighted impulse neural network | |
| Zhao et al. | Feedforward categorization on AER motion events using cortex-like features in a spiking neural network | |
| Wu et al. | Training spiking neural networks with accumulated spiking flow | |
| CN112699956A (en) | Neural morphology visual target classification method based on improved impulse neural network | |
| Cordone et al. | Learning from event cameras with sparse spiking convolutional neural networks | |
| Zambrano et al. | Efficient computation in adaptive artificial spiking neural networks | |
| CN114998659B (en) | Image data classification method for training impulse neural network model on line along with time | |
| Srinivasan et al. | Training deep spiking neural networks for energy-efficient neuromorphic computing | |
| CN111612136A (en) | Neural morphology visual target classification method and system | |
| CN114186672A (en) | Efficient high-precision training algorithm for impulse neural network | |
| Zhang et al. | Efficient spiking neural networks with logarithmic temporal coding | |
| CN111062450A (en) | Image classification device and method based on FPGA and SCNN architecture | |
| CN114926737A (en) | Low-power-consumption target detection method based on convolutional pulse neural network | |
| Wei et al. | Cloudprednet: an ultra-short-term movement prediction model for ground-based cloud image | |
| US20220358346A1 (en) | Systems, methods, and media for generating and using spiking neural networks with improved efficiency | |
| Lu et al. | An event-based categorization model using spatio-temporal features in a spiking neural network | |
| Kalbande et al. | Performance comparison of deep spiking CNN with artificial deep CNN for image classification tasks | |
| CN116758331A (en) | Object detection method, device and storage medium | |
| Cai et al. | Spike Timing Dependent Gradient for Direct Training of Fast and Efficient Binarized Spiking Neural Networks | |
| CN116109489A (en) | Denoising method and related equipment | |
| CN114004353A (en) | Optical neural network chip construction method and system for reducing number of optical devices | |
| Sun et al. | Deep spiking neural network with ternary spikes | |
| Gerlinghoff et al. | Desire backpropagation: A lightweight training algorithm for multi-layer spiking neural networks based on spike-timing-dependent plasticity | |
| Vicol et al. | Real-time classification of LIDAR data using discrete-time Recurrent Spiking Neural Networks |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |