WO2020155741A1 - 卷积神经网络和脉冲神经网络的融合结构及方法 - Google Patents

卷积神经网络和脉冲神经网络的融合结构及方法 Download PDF

Info

Publication number
WO2020155741A1
WO2020155741A1 PCT/CN2019/117039 CN2019117039W WO2020155741A1 WO 2020155741 A1 WO2020155741 A1 WO 2020155741A1 CN 2019117039 W CN2019117039 W CN 2019117039W WO 2020155741 A1 WO2020155741 A1 WO 2020155741A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural network
pulse
convolutional neural
fusion
layer
Prior art date
Application number
PCT/CN2019/117039
Other languages
English (en)
French (fr)
Inventor
李兆麟
王明羽
周武
Original Assignee
清华大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 清华大学 filed Critical 清华大学
Publication of WO2020155741A1 publication Critical patent/WO2020155741A1/zh
Priority to US17/386,570 priority Critical patent/US20210357726A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent

Definitions

  • the invention relates to the technical field of high-speed image recognition, in particular to a fusion structure and method of a convolutional neural network and a pulse neural network.
  • convolutional neural networks are currently widely used for image classification and recognition, and they already have relatively mature network structures and training algorithms. Existing research results show that if training samples are of sufficient quality and sufficient, convolutional neural networks are Traditional image recognition has a high recognition rate. However, convolutional neural networks also have certain shortcomings. With the complexity of sample features, the structure of convolutional neural networks is becoming more and more complex, and the network hierarchy continues to increase, resulting in a sharp increase in the amount of calculation to complete network training and derivation. Network calculation delays are also increasing.
  • spiking neural network is a new type of neural network that uses discrete neural pulses for information processing. Compared with traditional artificial neural networks, it has better biological simulation performance and is one of the research hotspots in recent years.
  • the discrete pulses of the pulse neural network have the characteristics of sparseness, which can greatly reduce the amount of network operations, and have advantages in achieving high performance, low power consumption, and alleviating overfitting. Therefore, it is necessary to realize a fusion network of convolutional neural network and pulse neural network.
  • This fusion network can not only take advantage of the advantages of convolutional neural network in ensuring image recognition rate, but also play the role of pulse neural network in low power consumption. And the advantages of low latency, so as to achieve high-speed time-varying information feature extraction and accurate classification.
  • the present invention aims to solve one of the technical problems in the related art at least to a certain extent.
  • one purpose of the present invention is to propose a fusion structure of convolutional neural network and spiking neural network, which can take into account the advantages of convolutional neural network and spiking neural network at the same time, and uses convolutional neural network in the field of image recognition. It has the advantage of higher recognition rate, and can play the advantages of spiking neural network in terms of sparsity, low power consumption, and alleviating overfitting. It can be applied to the fields of feature extraction and accurate classification of high-speed time-varying information.
  • Another object of the present invention is to provide a fusion method of convolutional neural network and spiking neural network.
  • an embodiment of the present invention proposes a convolutional neural network and a spiking neural network fusion structure, including: a convolutional neural network structure, the convolutional neural network structure includes an input layer, a convolutional layer and Pooling layer, wherein the input layer is used to receive pixel-level image data, the convolution layer is used for convolution operation, and the pooling layer is used for pooling operation; pulse conversion and coding structure, the pulse conversion The coding structure includes a pulse conversion neuron and a configurable pulse encoder, wherein the pulse conversion neuron is used to convert the pixel-level image data into pulse information based on a preset encoding form, and the configurable pulse encoder uses In the configuration of the pulse conversion and encoding structure as time encoding or frequency encoding; a pulse neural network structure, the pulse neural network structure includes a pulse convolution layer, a pulse pooling layer, and a pulse output layer, wherein the pulse convolution layer and The pulse pooling layer is used to perform pulse con
  • the fusion structure of the convolutional neural network and the spiking neural network in the embodiment of the present invention has a clear fusion network structure and simple training algorithm. It can not only give play to the advantages of the convolutional neural network in ensuring the image recognition rate, but also play the role of the spiking neural network. It has the advantages of low power consumption and low latency, and has the ability to be tailored and universal, the implementation method is simple, the cost is moderate, and it can be quickly deployed to different practical engineering applications, and can be used in any need to achieve high-speed image recognition related In engineering projects, high-speed time-varying information feature extraction and accurate classification are completed through fusion network design.
  • the fusion structure of the convolutional neural network and the spiking neural network according to the above embodiment of the present invention may also have the following additional technical features:
  • the pulse conversion neuron is further used for mapping the pixel-level image data into an analog current according to pulse frequency conversion, and obtaining the pulse information according to the analog current.
  • the corresponding relationship between the pulse frequency and the analog current is:
  • Rate represents the pulse frequency
  • t ref represents the length of the neural refractory period
  • ⁇ RC represents the time constant determined according to the membrane resistance and the membrane capacitance
  • V(t 0 ) and V(t 1 ) represent t 0 and t respectively
  • I represents the analog current
  • the impulse convolution operation further includes: the pixel-level convolution kernel is based on the synaptic strength and synaptic extension of neurons based on the LIF (Leaky-Integrate-and-Fire Model) model.
  • the time mapping relationship generates a pulse convolution kernel, and generates a pulse convolution feature map according to the pulse convolution kernel and the pulse information through a pulse multiplication and addition operation.
  • the pulse pooling operation further includes: a pixel-level pooling window generates a pulse pooling window according to the synapse intensity and the synaptic delay mapping relationship, and according to the The pulse pooling window and the pulse information undergo the pulse accumulation operation to generate a pulse pooling characteristic map.
  • the synaptic strength and the synaptic delay mapping relationship further include: the pixel-level convolution kernel and the pixel-level pooling window are based on MP (McCulloch- The weight and bias of the artificial neuron of the Pitts Model model are respectively mapped to the synaptic strength and synaptic delay of the neuron based on the LIF model.
  • the mapping relationship between the synapse strength and the synapse delay further includes: mapping the weights and biases of the artificial neuron based on the MP model to the neuron based on the LIF model.
  • mapping the weights and biases of the artificial neuron based on the MP model to the neuron based on the LIF model.
  • the principle of analog current superposition is used to realize the superposition of pulse information.
  • the pulse accumulation operation further includes: the pixel-level convolution kernel based on the weight and bias of the artificial neuron of the MP model are respectively mapped to the neuron based on the LIF model Meta’s synaptic strength and synaptic delay.
  • another embodiment of the present invention proposes a fusion method of a convolutional neural network and a spiking neural network, which includes the following steps: establishing a corresponding relationship between an equivalent convolutional neural network and a fusion neural network; Correspondence transforms the learning training result of the equivalent convolutional neural network and the fusion network learning training result of the convolutional neural network and the impulse neural network to obtain the fusion result of the convolutional neural network and the impulse neural network.
  • the fusion method of the convolutional neural network and the spiking neural network in the embodiment of the present invention has a clear fusion network structure and simple training algorithm. It can not only take advantage of the advantages of the convolutional neural network in ensuring the image recognition rate, but also play the role of the pulsed neural network. It has the advantages of low power consumption and low latency, and has the ability to be tailored and universal, the implementation method is simple, the cost is moderate, and it can be quickly deployed to different practical engineering applications, and can be used in any need to achieve high-speed image recognition related In engineering projects, high-speed time-varying information feature extraction and accurate classification are completed through fusion network design.
  • the fusion method of convolutional neural network and spiking neural network may also have the following additional technical features:
  • the corresponding relationship between the equivalent convolutional neural network and the fused neural network includes a network layer structure, weights and biases, and a mapping relationship between activation functions.
  • FIG. 1 is a schematic structural diagram of a fusion structure of a convolutional neural network and a spiking neural network according to an embodiment of the present invention
  • FIG. 2 is a block diagram of a convolutional neural network and a spiking neural network fusion network structure according to an embodiment of the present invention
  • Figure 3 is a hierarchical structure diagram of a convolutional neural network and a spiking neural network fusion network according to an embodiment of the present invention
  • Fig. 4 is a flowchart of a pulse convolution operation according to an embodiment of the present invention.
  • FIG. 5 is a flowchart of a pulse pooling operation according to an embodiment of the present invention.
  • Fig. 6 is a flowchart of pulse multiplication and pulse accumulation operations according to an embodiment of the present invention.
  • Fig. 7 is a flowchart of a method for learning and training a fusion network according to an embodiment of the present invention.
  • Fig. 8 is a flowchart of a fusion method of a convolutional neural network and a spiking neural network according to an embodiment of the present invention.
  • Fig. 1 is a schematic structural diagram of a fusion structure of a convolutional neural network and a spiking neural network according to an embodiment of the present invention.
  • the fusion structure 10 of the convolutional neural network and the pulsed neural network includes: a convolutional neural network structure 100, a pulse conversion and coding structure 200, and a pulsed neural network structure 300.
  • the convolutional neural network structure 100 includes an input layer, a convolution layer, and a pooling layer.
  • the input layer is used for receiving pixel-level image data
  • the convolution layer is used for convolution operation
  • the pooling layer is used for pooling operation.
  • the pulse conversion and encoding structure 200 includes a pulse conversion neuron and a configurable pulse encoder.
  • the pulse conversion neuron is used to convert pixel-level image data into pulse information based on a preset encoding form, and the configurable pulse encoder is used to convert the pulse
  • the conversion and coding structure is configured as time coding or frequency coding.
  • the pulse neural network structure 300 includes a pulse convolution layer, a pulse pooling layer, and a pulse output layer.
  • the pulse convolution layer and the pulse pooling layer are used to perform pulse convolution operations and pulse pooling operations on pulse information to obtain calculation results.
  • the pulse output layer is used to output the calculation results.
  • the structure 10 of the embodiment of the present invention can take into account the advantages of convolutional neural networks and spiking neural networks at the same time, taking advantage of the high recognition rate of convolutional neural networks in the field of image recognition, and at the same time can give play to the sparseness and low
  • the advantages of power consumption and over-fitting can be applied to fields such as feature extraction and accurate classification of high-speed time-varying information.
  • the convolutional neural network and spiking neural network fusion network structure 10 includes three parts, namely: a convolutional neural network structure part, a spiking neural network structure part, and a pulse conversion and coding part.
  • the convolutional neural network structure part further includes: input layer, convolutional layer and output layer;
  • the impulse neural network structure part further includes: impulse convolution layer, impulse layer and impulse output layer.
  • the convolutional neural network structure part further includes: MP model-based artificial neuron implementation (MPN) input layer, convolutional layer and pooling layer, respectively used to receive external pixel-level image data Input, convolution and pooling operations.
  • MPN MP model-based artificial neuron implementation
  • the number of network layers involved in the convolutional operation or pooling operation involved in the convolutional neural network structure part can be appropriately increased or deleted according to actual application tasks.
  • MP model or McCulloch-Pitts Model is a binary switch model that can be combined in different ways to complete various logic operations.
  • the pulse conversion and coding part further includes: pulse conversion neuron (SEN) and a configurable pulse encoder to convert pixel-level data into pulse information based on a specific encoding form. That is to say, the pulse conversion and coding part involves the conversion and coding process of converting pixel-level data into pulse information.
  • the hierarchical structure of this part is configurable and can be configured as time coding, frequency coding or other new coding methods as needed.
  • the impulse neural network structure part further includes: impulse convolution layer, impulse pooling layer and impulse output layer realized by impulse neuron (LIFN) based on LIF model.
  • the number of network layers involved in the convolution operation or pooling operation involved in the structure of the spiking neural network can be appropriately increased or deleted according to actual application tasks.
  • the pulse convolution layer and the pulse pooling layer further include: pulse convolution operation and pulse pooling operation, respectively used to process the convolution and pooling operation based on pulse information after the previous network level conversion and the final result output.
  • the "LIF model” or Leaky-Integrate-and-Fire Model is a neuron dynamic differential equation that describes the transfer relationship of action potentials in neurons.
  • the pulse conversion neuron is further used for mapping the pixel-level image data into an analog current according to the pulse frequency conversion, and obtaining pulse information according to the analog current.
  • the pulse conversion neuron (SEN) and the configurable pulse encoder further include: according to the pulse frequency conversion formula, the pixel-level output data of the convolutional neural network is mapped into analog current, and the pixel-level data is converted into frequency-based Encoded pulse information.
  • the corresponding relationship between the pulse frequency and the analog current is:
  • Rate represents the pulse frequency
  • t ref represents the length of the neural refractory period
  • ⁇ RC represents the time constant determined according to the membrane resistance and membrane capacitance
  • V(t 0 ) and V(t 1 ) represent the time t 0 and t 1 respectively
  • I represents the analog current.
  • membrane resistance means the physical quantities used to represent the biophysical characteristics of the cell membrane in the LIF model, and are used to describe the ion current of neurons in synapses. The conduction relationship.
  • the pulse conversion and encoding part further includes: the conversion and encoding between pixel-level data and pulse information, for example, the correspondence between the pulse firing frequency of the pulse neuron based on the LIF model and the analog current can be expressed by the formula 1Description:
  • Rate represents the pulse frequency
  • t ref represents the length of the neural refractory period
  • ⁇ RC represents the time constant determined according to the membrane resistance and membrane capacitance
  • V(t 0 ) and V(t 1 ) represent the time t 0 and t 1 respectively
  • the membrane voltage, I represents the analog current.
  • Formula 1 can be simplified to the description of Formula 2:
  • the pixel-level output data of the convolutional neural network can be mapped into analog current, and then t ref and ⁇ RC constants can be adjusted appropriately according to actual needs, and the pixel-level data can be converted into pulse information based on frequency encoding.
  • Formula 1 and Formula 2 can also adopt other deformations or higher-order correction forms according to actual needs.
  • the impulse convolution operation further includes: the pixel-level convolution kernel generates the impulse convolution kernel according to the synaptic strength and the synaptic delay mapping relationship of the neuron based on the LIF model, and according to the pulse
  • the convolution kernel and pulse information are subjected to pulse multiplication and addition operations to generate pulse convolution feature maps.
  • the pulse convolution operation further includes: the pixel-level convolution kernel generates a pulse convolution kernel according to the synaptic strength and the synaptic delay mapping relationship, and the input pulse information and the completed pulse convolution kernel undergo a pulse multiplication and addition operation Generate pulse convolution feature map.
  • the mapping relationship between synaptic strength and synaptic delay further includes: pixel-level convolution kernel and pixel-level pooling window roots are mapped to the weights and biases of artificial neurons based on the MP model. Synaptic strength and synaptic delay based on LIF model neurons.
  • mapping relationship between synaptic strength and synaptic delay further includes: pixel-level convolution kernel and pooling window.
  • the weights and biases of artificial neurons based on the MP model are mapped to synapses based on the LIF model neurons.
  • Strength and synaptic delay method is also included.
  • the impulse convolution operation in the structure of the spiking neural network further includes: mapping and replacing methods based on the mapping and replacement of the artificial neuron based on the MP model and the impulse neuron based on the LIF model in the process of completing the convolution operation.
  • the weight and bias of the artificial neuron based on the MP model are respectively mapped to the synaptic strength and synaptic delay of the neuron based on the LIF model.
  • the pulse pooling operation further includes: the pixel-level pooling window generates a pulse pooling window according to the synaptic strength and synaptic delay mapping relationship, and according to the pulse pooling window and pulse information
  • the pulse pooling feature map is generated after pulse accumulation operation.
  • the pulse pooling operation further includes: the pixel-level pooling window generates a pulse pooling window according to the synaptic strength and synaptic delay mapping relationship, and the input pulse information and the completed mapping pulse pooling window are generated by the pulse accumulation operation Pulse pooling feature map.
  • the pulse pooling operation in the structure of the pulse neural network further includes: mapping and mapping based on the MP model artificial neuron and the LIF model pulse neuron in the process of completing the convolution operation. Replacement method.
  • the weight and bias of the artificial neuron based on the MP model are respectively mapped to the synaptic strength and synaptic delay of the neuron based on the LIF model.
  • the pulse convolution feature map is controlled by the pooling function (mean pooling or maximum pooling, etc.), and the pooling window is adjusted to traverse the pulse convolution feature map, and finally the pulse pooling feature map is output.
  • the pulse accumulation operation further includes: the pixel-level convolution kernel based on the MP model of the artificial neuron weight and bias are respectively mapped to the synaptic strength and synapse strength and synapse based on the LIF model neuron Delay.
  • the pulse multiplication and addition operation further includes: the pixel-level convolution kernel based on the MP model of the artificial neuron weight and bias are respectively mapped to the synaptic strength and synaptic delay method based on the LIF model neuron.
  • the mapping relationship between synaptic strength and synaptic delay further includes: mapping the weights and biases of the artificial neurons based on the MP model to the synapses based on the LIF model neurons.
  • mapping the weights and biases of the artificial neurons based on the MP model to the synapses based on the LIF model neurons.
  • the principle of analog current superposition is used to realize the superposition of pulse information.
  • mapping relationship between synaptic strength and synaptic delay further includes: mapping the weights and biases of artificial neurons based on the MP model to the synaptic strength and synaptic delay of neurons based on the LIF model.
  • the pulse information superposition method is realized by adopting the principle of analog current superposition.
  • the pulse multiplication and pulse accumulation operations involved in the pulse convolution operation and the pulse pooling operation in the pulse neural network structure part further include a method of realizing pulse information superposition based on analog current superposition.
  • the analog current superposition is described by Equation 3:
  • Equation 3 I (t) represents the analog current, S i, and d i respectively represent synapses and synaptic strength delay, ⁇ (t) represents a correction function, can be adjusted according to the project.
  • pulse pooling operations involve pulse multiplication and addition, pulse accumulation, or pulse comparison operations.
  • Pulse accumulation is a special form of pulse multiplication and addition (weighting coefficient is 1).
  • Figure 6 shows more details of the pulse multiplication and addition operation.
  • the pulse comparison operation can be compared with the pulse frequency by a simple pulse counter.
  • the pulse multiplication and pulse accumulation operations are based on mapping the weights and biases of the artificial neurons based on the MP model to the synaptic strength and synaptic delay of the neurons based on the LIF model, and are implemented by using the principle of analog current superposition Pulse information superimposition, Figure 6 shows more details of the implementation process of pulse multiplication and pulse accumulation operations, specifically:
  • pulse neuron when the pulse neuron receives the output signal of the upper network, it first determines whether the signal is pulse information or pixel-level data. If it is pixel-level data, it needs to complete a pulse conversion and encoding (pulse information Conversion and coding 1); otherwise, directly follow the formula (3) to complete the superposition of the analog current, the superposition of the analog current follows the synaptic strength and synaptic delay mapping relationship, and the superimposed analog current will charge and discharge the membrane capacitor again.
  • Pulse conversion and coding (pulse information conversion and coding 2) can characterize the multiplication or accumulation of pulse information, and the accumulation operation can be understood as a special case of the multiplication and addition operation (weighting coefficient is 1).
  • a method for implementing fusion network training based on equivalent convolutional neural networks further includes: establishing the corresponding relationship between equivalent convolutional neural networks and fusion neural networks to realize equivalent convolutional neural network learning and training results and convolution Convergence of neural network and spiking neural network network learning training result conversion.
  • the corresponding relationship between the equivalent convolutional neural network and the fusion neural network further includes: the mapping relationship between the equivalent convolutional neural network and the fusion network in terms of network layer structure, weights and biases, and activation functions.
  • the fusion network learning training of convolutional neural network and spiking neural network uses a method based on equivalent convolutional neural network to achieve fusion network training.
  • the equivalent convolutional neural network and the fusion network respectively establish a one-to-one correspondence in the network layer structure, weights and biases, and activation functions.
  • Figure 6 shows more details of learning and training of the fusion network of convolutional neural network and spiking neural network, specifically:
  • the equivalent convolutional neural network is first generated according to the fusion network structure parameters of the convolutional neural network and the spiking neural network, and then the equivalent convolutional neural network is replaced or adjusted according to formula (1) or formula (2)
  • the activation function monitors the convergence of the training algorithm during the back propagation calculation process until the appropriate equivalent activation function is selected.
  • map the corresponding network parameters (such as weights, biases, etc.) according to synaptic strength and synaptic delay to obtain a convolutional neural network and a spiking neural network fusion network Training results.
  • the fusion network of the convolutional neural network and the spiking neural network of the present invention has the following advantages and beneficial effects compared with the prior art:
  • the fusion network proposed by the present invention can not only exert the advantages of the convolutional neural network in ensuring the image recognition rate, but also take advantage of the low power consumption and low latency of the impulse neural network.
  • the sparseness of pulse information is fully utilized in the structure of the pulse neural network, which greatly reduces the amount of network calculation and calculation delay, and is more in line with the real-time requirements of actual high-speed target recognition engineering applications.
  • the fusion network proposed by the present invention provides a method to realize image recognition on the basis of the impulse neural network, the impulse conversion and coding method, impulse convolution operation method,
  • the impulse pooling operation method has strong versatility and can be applied to any problem that may need to use the spiking neural network structure for feature extraction and classification, and solve the problem of how to use the spiking neural network to achieve feature extraction and accurate classification.
  • the convolutional neural network part, the impulse conversion and coding part, the impulse neural network part, and the number of network layers that complete convolution operation or pooling operation involved in the fusion network structure proposed by the present invention can be based on actual application tasks It needs to be added or deleted appropriately, can adapt to any scale of neural network structure, and has strong flexibility and scalability.
  • the mapping and replacement methods between MP model artificial neurons and LIF model pulse neurons involved in the fusion network proposed by the present invention are simple and clear, and the training method of the fusion network is borrowed from traditional convolutional neural networks
  • the training method, the synaptic strength and synaptic delay mapping method are simple and feasible.
  • the fusion network proposed by the present invention can be quickly deployed in actual engineering applications and has high practicability.
  • the fusion network structure is clear and the training algorithm is simple. It can not only take advantage of the advantages of the convolutional neural network in ensuring the image recognition rate, but also exert the impulse neural network.
  • the network has the advantages of low power consumption and low latency, and is tailorable and universal.
  • the implementation method is simple, the cost is moderate, and it can be quickly deployed to different practical engineering applications, and can achieve high-speed images in any need In the identification of related engineering projects, high-speed time-varying information feature extraction and accurate classification are completed through fusion network design.
  • Fig. 8 is a flowchart of a method for fusion of a convolutional neural network and a spiking neural network according to an embodiment of the present invention.
  • the fusion method of the convolutional neural network and the spiking neural network includes the following steps:
  • Step S801 establishing the corresponding relationship between the equivalent convolutional neural network and the fused neural network
  • Step S802 Convert the learning and training result of the equivalent convolutional neural network and the fusion network learning and training result of the convolutional neural network and the impulse neural network according to the corresponding relationship to obtain the fusion result of the convolutional neural network and the impulse neural network.
  • the corresponding relationship between the equivalent convolutional neural network and the fused neural network includes the network layer structure, weights and biases, and the mapping relationship between activation functions.
  • the structure of the fusion network is clear and the training algorithm is simple. It can not only give play to the advantages of convolutional neural network in ensuring the image recognition rate, but also give play to the impulse neural network.
  • the network has the advantages of low power consumption and low latency, and is tailorable and universal.
  • the implementation method is simple, the cost is moderate, and it can be quickly deployed to different practical engineering applications, and can achieve high-speed images in any need In the identification of related engineering projects, high-speed time-varying information feature extraction and accurate classification are completed through fusion network design.
  • first and second are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first” and “second” may explicitly or implicitly include at least one of the features. In the description of the present invention, "a plurality of” means at least two, such as two, three, etc., unless otherwise specifically defined.
  • the "on” or “under” of the first feature on the second feature may be in direct contact with the first and second features, or the first and second features may be indirectly through an intermediary. contact.
  • the "above”, “above” and “above” of the first feature on the second feature may mean that the first feature is directly above or obliquely above the second feature, or it simply means that the first feature is higher in level than the second feature.
  • the “below”, “below” and “below” of the second feature of the first feature may mean that the first feature is directly below or obliquely below the second feature, or it simply means that the level of the first feature is smaller than the second feature.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Complex Calculations (AREA)

Abstract

一种卷积神经网络和脉冲神经网络的融合结构(10)及方法，其中，结构包括：卷积神经网络结构(100)、脉冲转换与编码结构(200)和脉冲神经网络结构(300)，其中，卷积神经网络结构(100)包括输入层、卷积层和池化层；脉冲转换与编码结构(200)包括脉冲转换神经元和可配置脉冲编码器；脉冲神经网络结构(300)包括脉冲卷积层、脉冲池化层和脉冲输出层。该结构(10)能够同时兼顾卷积神经网络和脉冲神经网络的优点，利用了卷积神经网络在图像识别领域具有较高识别率的优势，同时能发挥脉冲神经网络在稀疏性、低功耗、缓解过拟合等方面的优势，可以应用于对高速时变信息进行特征提取和准确分类等领域。

Description

卷积神经网络和脉冲神经网络的融合结构及方法
相关申请的交叉引用
本申请要求清华大学于2019年01月29日提交的、发明名称为“卷积神经网络和脉冲神经网络的融合结构及方法”的、中国专利申请号“201910087183.8”的优先权。
技术领域
本发明涉及高速图像识别技术领域，特别涉及一种卷积神经网络和脉冲神经网络的融合结构及方法。
背景技术
在图像识别领域中，目前广泛采用卷积神经网络进行图像分类与识别，并且已经具有相对成熟的网络结构和训练算法，现有研究成果显示，如果训练样本保证质量且充分，卷积神经网络在传统图像识别中具有较高的识别率。然而，卷积神经网络也具有一定的缺陷，随着样本特征的复杂化，卷积神经网络的结构也越来越复杂，网络层级结构不断增加，导致完成网络训练和推导的计算量急剧增加，网络计算延时也越来越大。
因此，在高速图像识别领域，尤其针对一些实时性嵌入式系统领域，卷积神经网络难以满足这些系统的计算延时需求。另一方面，脉冲神经网络是一种利用离散神经脉冲进行信息处理的新型神经网络，与传统人工神经网络相比，具有更好的生物仿真性能，是近年来的研究热点之一。脉冲神经网络的离散脉冲具有稀疏性特征，可以大量减少网络运算量，在实现高性能、低功耗以及缓解过拟合等方面具有优势。因此，有必要实现一种卷积神经网络和脉冲神经网络的融合网络，这种融合网络既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，从而实现高速时变信息特征提取和准确分类。
发明内容
本发明旨在至少在一定程度上解决相关技术中的技术问题之一。
为此，本发明的一个目的在于提出一种卷积神经网络和脉冲神经网络的融合结构，该结构能够同时兼顾卷积神经网络和脉冲神经网络的优点，利用了卷积神经网络在图像识别领域具有较高识别率的优势，同时能发挥脉冲神经网络在稀疏性、低功耗、缓解过拟合等方面的优势，可以应用于对高速时变信息进行特征提取和准确分类等领域。
本发明的另一个目的在于提出一种卷积神经网络和脉冲神经网络的融合方法。
为达到上述目的，本发明一方面实施例提出了一种卷积神经网络和脉冲神经网络的融 合结构，包括：卷积神经网络结构，所述卷积神经网络结构包括输入层、卷积层和池化层，其中，所述输入层用于接收像素级图像数据，所述卷积层用于卷积运算，所述池化层用于池化运算；脉冲转换与编码结构，所述脉冲转换与编码结构包括脉冲转换神经元和可配置脉冲编码器，其中，所述脉冲转换神经元用于将所述像素级图像数据换成基于预设编码形式的脉冲信息，所述可配置脉冲编码器用于将所述脉冲转换与编码结构配置成时间编码或频率编码；脉冲神经网络结构，所述脉冲神经网络结构包括脉冲卷积层、脉冲池化层和脉冲输出层，其中，脉冲卷积层和脉冲池化层分别用于对所述脉冲信息进行脉冲卷积运算和脉冲池化运算得到运算结果，所述脉冲输出层用于输出所述运算结果。
本发明实施例的卷积神经网络和脉冲神经网络的融合结构，融合网络结构明确，训练算法简单，既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，并具有可裁剪性和普适性，实现方法简单，代价适中，且可以快速部署到不同的实际工程应用中，可以在任何需要实现高速图像识别相关工程项目中，通过融合网络设计完成高速时变信息特征提取和准确分类。
另外，根据本发明上述实施例的卷积神经网络和脉冲神经网络的融合结构还可以具有以下附加的技术特征：
进一步地，在本发明的一个实施例中，所述脉冲转换神经元进一步用于根据脉冲频率转换将所述像素级图像数据映射成模拟电流，并根据所述模拟电流得到所述脉冲信息。
进一步地，在本发明的一个实施例中，所述脉冲频率与所述模拟电流之间的对应关系为：
Figure PCTCN2019117039-appb-000001
其中，Rate表示所述脉冲频率，t ref表示神经不应期时间长度，τ RC表示根据膜电阻和膜电容确定的时间常数，V(t 0)和V(t 1)分别表示t 0和t 1时刻的膜电压，I表示模拟电流。
进一步地，在本发明的一个实施例中，所述脉冲卷积运算进一步包括：像素级卷积核根据基于LIF(Leaky-Integrate-and-Fire Model)模型神经元的突触强度和突触延时映射关系生成脉冲卷积核，并根据所述脉冲卷积核和所述脉冲信息且经过脉冲乘加运算生成脉冲卷积特征图。
进一步地，在本发明的一个实施例中，所述脉冲池化运算进一步包括：像素级池化窗口根据所述突触强度和所述突触延时映射关系生成脉冲池化窗口，并根据所述脉冲池化窗口和所述脉冲信息经过所述脉冲累加运算生成脉冲池化特征图。
进一步地，在本发明的一个实施例中，所述突触强度和所述突触延时映射关系进一步包括：所述像素级卷积核和所述像素级池化窗口根基于MP(McCulloch-Pitts Model)模型 的人工神经元的权重和偏置分别映射成所述基于LIF模型神经元的突触强度和突触延时。
进一步地，在本发明的一个实施例中，所述突触强度和所述突触延时映射关系进一步包括：在把基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时的基础上，采用模拟电流叠加原理实现脉冲信息叠加。
进一步地，在本发明的一个实施例中，所述脉冲累加运算进一步包括：所述像素级卷积核基于所述MP模型的人工神经元的权重和偏置分别映射成基于所述LIF模型神经元的突触强度和突触延时。
为达到上述目的，本发明另一方面实施例提出了一种卷积神经网络和脉冲神经网络的融合方法，包括以下步骤：建立等效卷积神经网络和融合神经网络的对应关系；根据所述对应关系转换等效卷积神经网络学习训练结果与卷积神经网络和脉冲神经网络的融合网络学习训练结果，以得到卷积神经网络和脉冲神经网络的融合结果。
本发明实施例的卷积神经网络和脉冲神经网络的融合方法，融合网络结构明确，训练算法简单，既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，并具有可裁剪性和普适性，实现方法简单，代价适中，且可以快速部署到不同的实际工程应用中，可以在任何需要实现高速图像识别相关工程项目中，通过融合网络设计完成高速时变信息特征提取和准确分类。
另外，根据本发明上述实施例的卷积神经网络和脉冲神经网络的融合方法还可以具有以下附加的技术特征：
进一步地，在本发明的一个实施例中，所述等效卷积神经网络和融合神经网络的对应关系包括网络层结构、权重和偏置，以及激活函数之间的映射关系。
本发明附加的方面和优点将在下面的描述中部分给出，部分将从下面的描述中变得明显，或通过本发明的实践了解到。
附图说明
本发明上述的和/或附加的方面和优点从下面结合附图对实施例的描述中将变得明显和容易理解，其中：
图1为根据本发明一个实施例的卷积神经网络和脉冲神经网络的融合结构的结构示意图；
图2为根据本发明一个实施例的卷积神经网络和脉冲神经网络融合网络结构框图；
图3为根据本发明一个实施例的卷积神经网络和脉冲神经网络融合网络层级结构图；
图4为根据本发明一个实施例的脉冲卷积运算流程图；
图5为根据本发明一个实施例的脉冲池化运算流程图；
图6为根据本发明一个实施例的脉冲乘加运算和脉冲累加运算流程图；
图7为根据本发明一个实施例的融合网络学习训练方法流程图；
图8为根据本发明一个实施例的卷积神经网络和脉冲神经网络的融合方法的流程图。
具体实施方式
下面详细描述本发明的实施例，所述实施例的示例在附图中示出，其中自始至终相同或类似的标号表示相同或类似的元件或具有相同或类似功能的元件。下面通过参考附图描述的实施例是示例性的，旨在用于解释本发明，而不能理解为对本发明的限制。
下面参照附图描述根据本发明实施例提出的卷积神经网络和脉冲神经网络的融合结构及方法，首先将参照附图描述根据本发明实施例提出的卷积神经网络和脉冲神经网络的融合结构。
图1是本发明一个实施例的卷积神经网络和脉冲神经网络的融合结构的结构示意图。
如图1所示，该卷积神经网络和脉冲神经网络的融合结构10包括：卷积神经网络结构100、脉冲转换与编码结构200和脉冲神经网络结构300。
其中，卷积神经网络结构100包括输入层、卷积层和池化层，其中，输入层用于接收像素级图像数据，卷积层用于卷积运算，池化层用于池化运算。脉冲转换与编码结构200包括脉冲转换神经元和可配置脉冲编码器，其中，脉冲转换神经元用于将像素级图像数据换成基于预设编码形式的脉冲信息，可配置脉冲编码器用于将脉冲转换与编码结构配置成时间编码或频率编码。脉冲神经网络结构300包括脉冲卷积层、脉冲池化层和脉冲输出层，其中，脉冲卷积层和脉冲池化层分别用于对脉冲信息进行脉冲卷积运算和脉冲池化运算得到运算结果，脉冲输出层用于输出运算结果。本发明实施例的结构10能够同时兼顾卷积神经网络和脉冲神经网络的优点，利用了卷积神经网络在图像识别领域具有较高识别率的优势，同时能发挥脉冲神经网络在稀疏性、低功耗、缓解过拟合等方面的优势，可以应用于对高速时变信息进行特征提取和准确分类等领域。
具体而言，如图2所示，卷积神经网络和脉冲神经网络融合网络结构10包括三个部分，分别是：卷积神经网络结构部分、脉冲神经网络结构部分和脉冲转换与编码部分。其中，卷积神经网络结构部分进一步包括：输入层、卷积层和输出层；脉冲神经网络结构部分进一步包括：脉冲卷积层、脉冲层和脉冲输出层。
其中，如图3所示，卷积神经网络结构部分进一步包括：基于MP模型的人工神经元实现(MPN)的输入层、卷积层和池化层，分别用于接收外部的像素级图像数据输入、卷积和池化运算。卷积神经网络结构部分中所涉及的完成卷积运算或池化运算的网络层数量可根据实际应用任务需要适当增加或者删减。需要说明的是，“MP模型”即McCulloch-Pitts Model，是一种按不同方式组合可完成各种逻辑运算的二值开关模型。
脉冲转换与编码部分进一步包括：脉冲转换神经元(SEN)和可配置脉冲编码器，实现把像素级数据转换成基于特定编码形式的脉冲信息。也就是说，脉冲转换与编码部分涉及把像素级数据转换成脉冲信息的转换和编码过程，该部分层级结构可配置，可根据需要 配置成时间编码、频率编码或者其他新型编码方式。
脉冲神经网络结构部分进一步包括：基于LIF模型的脉冲神经元(LIFN)实现的脉冲卷积层、脉冲池化层和脉冲输出层。脉冲神经网络结构部分中所涉及的完成卷积运算或池化运算的网络层数量可根据实际应用任务需要适当增加或者删减。脉冲卷积层和脉冲池化层进一步分别包括：脉冲卷积运算和脉冲池化运算，分别用于处理经过上一网络层级转换后的基于脉冲信息的卷积和池化运算以及最终结果输出。需要说明的是，“LIF模型”即Leaky-Integrate-and-Fire Model，是描述动作电位在神经元中传递关系的神经元动力学微分方程。
进一步地，在本发明的一个实施例中，脉冲转换神经元进一步用于根据脉冲频率转换将像素级图像数据映射成模拟电流，并根据模拟电流得到脉冲信息。
可以理解的是，脉冲转换神经元(SEN)和可配置脉冲编码器进一步包括：根据脉冲频率转换公式把卷积神经网络的像素级输出数据映射成模拟电流，实现把像素级数据转换成基于频率编码的脉冲信息。
其中，在本发明的一个实施例中，脉冲频率与模拟电流之间的对应关系为：
Figure PCTCN2019117039-appb-000002
其中，Rate表示脉冲频率，t ref表示神经不应期时间长度，τ RC表示根据膜电阻和膜电容确定的时间常数，V(t 0)和V(t 1)分别表示t 0和t 1时刻的膜电压，I表示模拟电流。需要说明的是，“膜电阻”、“膜电容”和“膜电压”，均是指在LIF模型中用来表示细胞膜生物物理学特征的物理量，并且用于描述神经元离子电流在突触中的传导关系。
具体而言，脉冲转换与编码部分进一步包括：像素级数据与脉冲信息之间的转换与编码实现方法，例如，基于LIF模型的脉冲神经元的脉冲发放频率和模拟电流之间的对应关系可用公式1描述：
Figure PCTCN2019117039-appb-000003
其中，Rate表示脉冲频率，t ref表示神经不应期时间长度，τ RC表示根据膜电阻和膜电容确定的时间常数，V(t 0)和V(t 1)分别表示t 0和t 1时刻的膜电压，I表示模拟电流。特别地，在t 0到t 1时间区间，膜电压从0上升到1时，公式1可以简化为公式2描述：
Figure PCTCN2019117039-appb-000004
根据公式1或公式2，卷积神经网络的像素级输出数据可以映射成模拟电流，再根据实际需要适当调整t ref和τ RC常数，可把像素级数据转换成基于频率编码的脉冲信息。公式1和公式2还可以根据实际需要采用其他变形或者更高阶的修正形式。
进一步地，在本发明的一个实施例中，脉冲卷积运算进一步包括：像素级卷积核根据基于LIF模型神经元的突触强度和突触延时映射关系生成脉冲卷积核，并根据脉冲卷积核和脉冲信息且经过脉冲乘加运算生成脉冲卷积特征图。
可以理解的是，脉冲卷积运算进一步包括：像素级卷积核根据突触强度和突触延时映射关系生成脉冲卷积核，输入脉冲信息和已完成映脉冲卷积核经过脉冲乘加运算生成脉冲卷积特征图。
其中，在本发明的一个实施例中，突触强度和突触延时映射关系进一步包括：像素级卷积核和像素级池化窗口根基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时。
可以理解的是，突触强度和突触延时映射关系进一步包括：像素级卷积核和池化窗口基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时方法。
具体而言，如图4所示，首先像素级卷积核根据一一对应关系分别映射为突触强度和突触延时，然后输入脉冲信息和已完成映射的脉冲卷积核经过脉冲乘加运算生成脉冲卷积特征图。具体地，脉冲神经网络结构部分中脉冲卷积运算进一步包括：基于MP模型人工神经元和基于LIF模型脉冲神经元在完成卷积运算过程中所建立对应关系实现映射和替换的方法。其中，基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时。
进一步地，在本发明的一个实施例中，脉冲池化运算进一步包括：像素级池化窗口根据突触强度和突触延时映射关系生成脉冲池化窗口，并根据脉冲池化窗口和脉冲信息经过脉冲累加运算生成脉冲池化特征图。
可以理解的是，脉冲池化运算进一步包括：像素级池化窗口根据突触强度和突触延时映射关系生成脉冲池化窗口，输入脉冲信息和已完成映脉冲池化窗口经过脉冲累加运算生成脉冲池化特征图。
具体而言，如图5所示，脉冲神经网络结构部分中脉冲池化运算进一步包括：基于MP模型人工神经元和基于LIF模型脉冲神经元在完成卷积运算过程中所建立对应关系实现映射和替换的方法。其中，基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时。脉冲卷积特征图在池化函数(均值池化或最大池化等)控制下，调整池化窗口遍历脉冲卷积特征图，最终输出脉冲池化特征图。
进一步地，在本发明的一个实施例中，脉冲累加运算进一步包括：像素级卷积核基于 MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时。
可以理解的是，脉冲乘加运算进一步包括：像素级卷积核基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时方法。
进一步地，在本发明的一个实施例中，突触强度和突触延时映射关系进一步包括：在把基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时的基础上，采用模拟电流叠加原理实现脉冲信息叠加。
可以理解的是，突触强度和突触延时映射关系进一步包括：在把基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时的基础上，通过采用模拟电流叠加原理实现脉冲信息叠加方法。
具体而言，如图6所示，脉冲神经网络结构部分中脉冲卷积运算和脉冲池化运算所涉及的脉冲乘加运算和脉冲累加运算进一步包括：基于模拟电流叠加实现脉冲信息叠加的方法。其中，模拟电流叠加由公式3描述：
Figure PCTCN2019117039-appb-000005
公式3中的I(t)表示模拟电流，S i和d i分别表示突触强度和突触延时，Ψ(t)表示一个修正函数，可根据实际工程需要调整。
更进一步地，脉冲池化运算涉及脉冲乘加、脉冲累加或脉冲比较运算。脉冲累加是脉冲乘加的特殊形式(加权系数为1)，图6显示脉冲乘加运算的更多细节，脉冲比较运算可由简单脉冲计数器对脉冲频率实现比较。
脉冲乘加运算和脉冲累加运算在把基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时的基础上，通过采用模拟电流叠加原理实现脉冲信息叠加，图6显示了脉冲乘加运算或脉冲累加运算实施流程的更多细节，具体地：
如图6所示，当脉冲神经元接收到上一层网络的输出信号时，首先判断该信号是否为脉冲信息或者像素级数据，如果是像素级数据则需要完成一次脉冲转换与编码(脉冲信息转换与编码①)；否则，直接按照公式(3)完成模拟电流的叠加，模拟电流的叠加遵循突触强度和突触延时映射关系，完成叠加的模拟电流对膜电容充放电过程再经过一次脉冲转换与编码(脉冲信息转换与编码②)可以表征脉冲信息的乘加或者累加，其中累加运算可以理解为乘加运算的特殊情况(加权系数为1)。
进一步地，一种基于等效卷积神经网络实现融合网络训练的方法进一步包括：通过建立等效卷积神经网络和融合神经网络的对应关系，实现等效卷积神经网络学习训练结果与卷积神经网络和脉冲神经网络的融合网络学习训练结果转换。等效卷积神经网络和融合神经网络的对应关系进一步包括：等效卷积神经网络和融合网络分别在网络层结构、权重和偏置，以及激活函数等方面的映射关系。
具体而言，卷积神经网络和脉冲神经网络的融合网络学习训练利用了一种基于等效卷积神经网络实现融合网络训练的方法。其中，等效卷积神经网络和融合网络分别在网络层结构、权重和偏置、以及激活函数等方面建立一一对应关系。图6显示了卷积神经网络和脉冲神经网络的融合网络学习训练的更多细节，具体地：
如图6所示，首先根据卷积神经网络和脉冲神经网络的融合网络结构参数生成等效卷积神经网络，然后根据公式(1)或公式(2)替换或调整等效卷积神经网络的激活函数，在反向传播计算过程中监控训练算法的收敛性，直到选择合适的等效激活函数。在等效卷积神经网络的训练结果达到要求后，将相应的网络参数(如权重、偏置等)按照突触强度和突触延时映射，从而获得卷积神经网络和脉冲神经网络融合网络的训练结果。
综上，本发明的卷积神经网络和脉冲神经网络的融合网络与现有技术相比较具有以下优点和有益效果：
(1)相比于传统卷积神经网络，本发明提出的融合网络既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，在脉冲神经网络结构部分充分利用脉冲信息稀疏性，极大降低网络运算量和计算延时，更加符合实际高速目标识别工程应用的实时性需求。
(2)相比于传统脉冲神经网络，本发明提出的融合网络提供了一种在脉冲神经网络基础上实现图像识别的方法，融合网络所涉及的脉冲转换与编码方法、脉冲卷积运算方法、脉冲池化运算方法等均具有较强的通用性，可以应用在任何可能需要利用脉冲神经网络结构进行特征提取和分类的问题中，解决了如何利用脉冲神经网络实现特征提取和准确分类的问题。
(3)本发明提出的融合网络结构所涉及的卷积神经网络部分、脉冲转换与编码部分、脉冲神经网络部分，以及其中完成卷积运算或池化运算的网络层数量均可根据实际应用任务需要适当增加或者删减，能够适配任意规模的神经网络结构，具有较强的灵活性和可扩展性。
(4)本发明提出的融合网络的中所涉及的基于MP模型人工神经元和基于LIF模型脉冲神经元之间的映射和替换方法简单明确，并且融合网络的训练方法借鉴于传统卷积神经网络的训练方法，突触强度和突触延时映射方法简单可行，本发明提出的融合网络可以迅速部署到实际工程应用中，具有较高的实用性。
根据本发明实施例提出的卷积神经网络和脉冲神经网络的融合结构，融合网络结构明确，训练算法简单，既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，并具有可裁剪性和普适性，实现方法简单，代价适中，且可以快速部署到不同的实际工程应用中，可以在任何需要实现高速图像识别相关工程项目中，通过融合网络设计完成高速时变信息特征提取和准确分类。
其次参照附图描述根据本发明实施例提出的卷积神经网络和脉冲神经网络的融合方法。
图8是本发明一个实施例的卷积神经网络和脉冲神经网络的融合方法的流程图。
如图8所示，该卷积神经网络和脉冲神经网络的融合方法包括以下步骤：
步骤S801，建立等效卷积神经网络和融合神经网络的对应关系；
步骤S802，根据对应关系转换等效卷积神经网络学习训练结果与卷积神经网络和脉冲神经网络的融合网络学习训练结果，以得到卷积神经网络和脉冲神经网络的融合结果。
进一步地，在本发明的一个实施例中，等效卷积神经网络和融合神经网络的对应关系包括网络层结构、权重和偏置，以及激活函数之间的映射关系。
需要说明的是，前述对卷积神经网络和脉冲神经网络的融合结构实施例的解释说明也适用于该实施例的卷积神经网络和脉冲神经网络的融合方法，此处不再赘述。
根据本发明实施例提出的卷积神经网络和脉冲神经网络的融合方法，融合网络结构明确，训练算法简单，既能够发挥卷积神经网络在保证图像识别率方面的优势，同时也能发挥脉冲神经网络在低功耗和低延时等方面的优势，并具有可裁剪性和普适性，实现方法简单，代价适中，且可以快速部署到不同的实际工程应用中，可以在任何需要实现高速图像识别相关工程项目中，通过融合网络设计完成高速时变信息特征提取和准确分类。
此外，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。在本发明的描述中，“多个”的含义是至少两个，例如两个，三个等，除非另有明确具体的限定。
在本发明中，除非另有明确的规定和限定，第一特征在第二特征“上”或“下”可以是第一和第二特征直接接触，或第一和第二特征通过中间媒介间接接触。而且，第一特征在第二特征“之上”、“上方”和“上面”可是第一特征在第二特征正上方或斜上方，或仅仅表示第一特征水平高度高于第二特征。第一特征在第二特征“之下”、“下方”和“下面”可以是第一特征在第二特征正下方或斜下方，或仅仅表示第一特征水平高度小于第二特征。
在本说明书的描述中，参考术语“一个实施例”、“一些实施例”、“示例”、“具体示例”、或“一些示例”等的描述意指结合该实施例或示例描述的具体特征、结构、材料或者特点包含于本发明的至少一个实施例或示例中。在本说明书中，对上述术语的示意性表述不必须针对的是相同的实施例或示例。而且，描述的具体特征、结构、材料或者特点可以在任一个或多个实施例或示例中以合适的方式结合。此外，在不相互矛盾的情况下，本领域的技术人员可以将本说明书中描述的不同实施例或示例以及不同实施例或示例的特征进行结合和组合。
尽管上面已经示出和描述了本发明的实施例，可以理解的是，上述实施例是示例性的，不能理解为对本发明的限制，本领域的普通技术人员在本发明的范围内可以对上述实施例进行变化、修改、替换和变型。

Claims (10)

  1. 一种卷积神经网络和脉冲神经网络的融合结构，其特征在于，包括：
    卷积神经网络结构，所述卷积神经网络结构包括输入层、卷积层和池化层，其中，所述输入层用于接收像素级图像数据，所述卷积层用于卷积运算，所述池化层用于池化运算；
    脉冲转换与编码结构，所述脉冲转换与编码结构包括脉冲转换神经元和可配置脉冲编码器，其中，所述脉冲转换神经元用于将所述像素级图像数据换成基于预设编码形式的脉冲信息，所述可配置脉冲编码器用于将所述脉冲转换与编码结构配置成时间编码或频率编码；以及
    脉冲神经网络结构，所述脉冲神经网络结构包括脉冲卷积层、脉冲池化层和脉冲输出层，其中，脉冲卷积层和脉冲池化层分别用于对所述脉冲信息进行脉冲卷积运算和脉冲池化运算得到运算结果，所述脉冲输出层用于输出所述运算结果。
  2. 根据权利要求1所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述脉冲转换神经元进一步用于根据脉冲频率转换将所述像素级图像数据映射成模拟电流，并根据所述模拟电流得到所述脉冲信息。
  3. 根据权利要求2所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述脉冲频率与所述模拟电流之间的对应关系为：
    Figure PCTCN2019117039-appb-100001
    其中，Rate表示所述脉冲频率，t ref表示神经不应期时间长度，τ RC表示根据膜电阻和膜电容确定的时间常数，V(t 0)和V(t 1)分别表示t 0和t 1时刻的膜电压，I表示模拟电流。
  4. 根据权利要求1所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述脉冲卷积运算进一步包括：
    像素级卷积核根据基于LIF模型神经元的突触强度和突触延时映射关系生成脉冲卷积核，并根据所述脉冲卷积核和所述脉冲信息且经过脉冲乘加运算生成脉冲卷积特征图。
  5. 根据权利要求4所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述脉冲池化运算进一步包括：
    像素级池化窗口根据所述突触强度和所述突触延时映射关系生成脉冲池化窗口，并根据所述脉冲池化窗口和所述脉冲信息经过所述脉冲累加运算生成脉冲池化特征图。
  6. 根据权利要求5任一项所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述突触强度和所述突触延时映射关系进一步包括：
    所述像素级卷积核和所述像素级池化窗口根基于MP模型的人工神经元的权重和偏置 分别映射成所述基于LIF模型神经元的突触强度和突触延时。
  7. 根据权利要求6任一项所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述突触强度和所述突触延时映射关系进一步包括：
    在把基于MP模型的人工神经元的权重和偏置分别映射成基于LIF模型神经元的突触强度和突触延时的基础上，采用模拟电流叠加原理实现脉冲信息叠加。
  8. 根据权利要求7所述的卷积神经网络和脉冲神经网络的融合结构，其特征在于，所述脉冲累加运算进一步包括：
    所述像素级卷积核基于所述MP模型的人工神经元的权重和偏置分别映射成基于所述LIF模型神经元的突触强度和突触延时。
  9. 一种如权利要求1所述的卷积神经网络和脉冲神经网络的融合方法，其特征在于，包括以下步骤：
    建立等效卷积神经网络和融合神经网络的对应关系；
    根据所述对应关系转换等效卷积神经网络学习训练结果与卷积神经网络和脉冲神经网络的融合网络学习训练结果，以得到卷积神经网络和脉冲神经网络的融合结果。
  10. 根据权利要求9所述的卷积神经网络和脉冲神经网络的融合方法，其特征在于，所述等效卷积神经网络和融合神经网络的对应关系包括网络层结构、权重和偏置，以及激活函数之间的映射关系。
PCT/CN2019/117039 2019-01-29 2019-11-11 卷积神经网络和脉冲神经网络的融合结构及方法 WO2020155741A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/386,570 US20210357726A1 (en) 2019-01-29 2021-07-28 Fusion structure and method of convolutional neural network and spiking neural network

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910087183.8A CN109816026B (zh) 2019-01-29 2019-01-29 卷积神经网络和脉冲神经网络的融合装置及方法
CN201910087183.8 2019-01-29

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US17/386,570 Continuation US20210357726A1 (en) 2019-01-29 2021-07-28 Fusion structure and method of convolutional neural network and spiking neural network

Publications (1)

Publication Number Publication Date
WO2020155741A1 true WO2020155741A1 (zh) 2020-08-06

Family

ID=66605701

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/117039 WO2020155741A1 (zh) 2019-01-29 2019-11-11 卷积神经网络和脉冲神经网络的融合结构及方法

Country Status (3)

Country Link
US (1) US20210357726A1 (zh)
CN (1) CN109816026B (zh)
WO (1) WO2020155741A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112085768A (zh) * 2020-09-02 2020-12-15 北京灵汐科技有限公司 光流信息预测方法、装置、电子设备和存储介质
CN113628615A (zh) * 2021-10-12 2021-11-09 中国科学院自动化研究所 语音识别方法、装置、电子设备及存储介质

Families Citing this family (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109816026B (zh) * 2019-01-29 2021-09-10 清华大学 卷积神经网络和脉冲神经网络的融合装置及方法
CN110322010B (zh) * 2019-07-02 2021-06-25 深圳忆海原识科技有限公司 用于类脑智能与认知计算的脉冲神经网络运算系统及方法
CN110555523B (zh) * 2019-07-23 2022-03-29 中建三局智能技术有限公司 一种基于脉冲神经网络的短程跟踪方法及系统
CN110543933B (zh) * 2019-08-12 2022-10-21 北京大学 基于flash存算阵列的脉冲型卷积神经网络
CN110458136B (zh) * 2019-08-19 2022-07-12 广东工业大学 一种交通标志识别方法、装置和设备
CN113033792A (zh) * 2019-12-24 2021-06-25 财团法人工业技术研究院 神经网络运算装置及方法
CN112188093B (zh) * 2020-09-24 2022-09-02 北京灵汐科技有限公司 双模态信号融合系统和方法
CN112257846A (zh) * 2020-10-13 2021-01-22 北京灵汐科技有限公司 神经元模型、拓扑结构、信息处理方法和视网膜神经元
CN112381857A (zh) * 2020-11-12 2021-02-19 天津大学 一种基于脉冲神经网络的类脑目标跟踪方法
CN112633497B (zh) * 2020-12-21 2023-08-18 中山大学 一种基于重加权膜电压的卷积脉冲神经网络的训练方法
CN113159276B (zh) * 2021-03-09 2024-04-16 北京大学 模型优化部署方法、系统、设备及存储介质
KR20230096657A (ko) * 2021-12-23 2023-06-30 한국전자기술연구원 Snn/cnn 동시 구동이 가능한 데이터 처리 시스템 및 방법
CN115238857B (zh) * 2022-06-15 2023-05-05 北京融合未来技术有限公司 基于脉冲信号的神经网络及脉冲信号处理方法
CN116205274B (zh) * 2023-04-27 2023-07-21 苏州浪潮智能科技有限公司 一种脉冲神经网络的控制方法、装置、设备及存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040193559A1 (en) * 2003-03-24 2004-09-30 Tetsuya Hoya Interconnecting neural network system, interconnecting neural network structure construction method, self-organizing neural network structure construction method, and construction programs therefor
CN105095961A (zh) * 2015-07-16 2015-11-25 清华大学 一种人工神经网络和脉冲神经网络的混合系统
CN105095966A (zh) * 2015-07-16 2015-11-25 清华大学 人工神经网络和脉冲神经网络的混合计算系统
CN105095965A (zh) * 2015-07-16 2015-11-25 清华大学 人工神经网络和脉冲神经网络神经的混合通信方法
CN108717570A (zh) * 2018-05-23 2018-10-30 电子科技大学 一种脉冲神经网络参数量化方法
CN109816026A (zh) * 2019-01-29 2019-05-28 清华大学 卷积神经网络和脉冲神经网络的融合结构及方法

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9195934B1 (en) * 2013-01-31 2015-11-24 Brain Corporation Spiking neuron classifier apparatus and methods using conditionally independent subsets
US20160358069A1 (en) * 2015-06-03 2016-12-08 Samsung Electronics Co., Ltd. Neural network suppression
CN105760930B (zh) * 2016-02-18 2018-06-05 天津大学 用于aer的多层脉冲神经网络识别系统
CN109214250A (zh) * 2017-07-05 2019-01-15 中南大学 一种基于多尺度卷积神经网络的静态手势识别方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040193559A1 (en) * 2003-03-24 2004-09-30 Tetsuya Hoya Interconnecting neural network system, interconnecting neural network structure construction method, self-organizing neural network structure construction method, and construction programs therefor
CN105095961A (zh) * 2015-07-16 2015-11-25 清华大学 一种人工神经网络和脉冲神经网络的混合系统
CN105095966A (zh) * 2015-07-16 2015-11-25 清华大学 人工神经网络和脉冲神经网络的混合计算系统
CN105095965A (zh) * 2015-07-16 2015-11-25 清华大学 人工神经网络和脉冲神经网络神经的混合通信方法
CN108717570A (zh) * 2018-05-23 2018-10-30 电子科技大学 一种脉冲神经网络参数量化方法
CN109816026A (zh) * 2019-01-29 2019-05-28 清华大学 卷积神经网络和脉冲神经网络的融合结构及方法

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112085768A (zh) * 2020-09-02 2020-12-15 北京灵汐科技有限公司 光流信息预测方法、装置、电子设备和存储介质
WO2022048582A1 (zh) * 2020-09-02 2022-03-10 北京灵汐科技有限公司 光流信息预测方法、装置、电子设备和存储介质
CN112085768B (zh) * 2020-09-02 2023-12-26 北京灵汐科技有限公司 光流信息预测方法、装置、电子设备和存储介质
CN113628615A (zh) * 2021-10-12 2021-11-09 中国科学院自动化研究所 语音识别方法、装置、电子设备及存储介质
CN113628615B (zh) * 2021-10-12 2022-01-04 中国科学院自动化研究所 语音识别方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN109816026A (zh) 2019-05-28
US20210357726A1 (en) 2021-11-18
CN109816026B (zh) 2021-09-10

Similar Documents

Publication Publication Date Title
WO2020155741A1 (zh) 卷积神经网络和脉冲神经网络的融合结构及方法
WO2021012752A1 (zh) 一种基于脉冲神经网络的短程跟踪方法及系统
Kim et al. Deep neural networks with weighted spikes
Jabbari et al. EMG-based hand gesture classification with long short-term memory deep recurrent neural networks
Wu et al. Training spiking neural networks with accumulated spiking flow
CN108717570A (zh) 一种脉冲神经网络参数量化方法
CN111612136B (zh) 一种神经形态视觉目标分类方法及系统
CN113190654B (zh) 一种基于实体联合嵌入和概率模型的知识图谱补全方法
CN113205048B (zh) 一种手势识别方法及识别系统
CN112712170A (zh) 基于输入加权脉冲神经网络的神经形态视觉目标分类系统
Xu et al. Biologically inspired structure learning with reverse knowledge distillation for spiking neural networks
Li et al. Reduction 93.7% time and power consumption using a memristor-based imprecise gradient update algorithm
Xiaoyuan et al. A new improved BP neural network algorithm
Lou et al. Towards digital retina in smart cities: A model generation, utilization and communication paradigm
CN114399041A (zh) 一种脉冲神经网络训练方法、装置及芯片
CN109886419A (zh) 一种基于对称矩阵空间子空间学习的格拉斯曼流形域自适应方法
Tang et al. Energy efficient spiking neural network design with RRAM devices
Zhang et al. [Retracted] Research on Pattern Recognition of Lower Limb Motion Based on Convolutional Neural Network
Guo et al. Multi-attention feature fusion network for accurate estimation of finger kinematics from surface electromyographic signals
Xiong et al. Hand gesture recognition using instant high-density EMG graph via deep learning method
Zou et al. A novel conversion method for spiking neural network using median quantization
CN113469357A (zh) 一种人工神经网络到脉冲神经网络的映射方法
Guo et al. A multi-conductance states memristor-based cnn circuit using quantization method for digital recognition
CN111860460A (zh) 一种基于改进的lstm模型在人体行为识别上的应用方法
Chen et al. Conversion of artificial neural network to spiking neural network for hardware implementation

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19912664

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19912664

Country of ref document: EP

Kind code of ref document: A1