WO2023103149A1 - 脉冲事件决策装置、方法、芯片及电子设备 - Google Patents

脉冲事件决策装置、方法、芯片及电子设备 Download PDF

Info

Publication number
WO2023103149A1
WO2023103149A1 PCT/CN2022/072038 CN2022072038W WO2023103149A1 WO 2023103149 A1 WO2023103149 A1 WO 2023103149A1 CN 2022072038 W CN2022072038 W CN 2022072038W WO 2023103149 A1 WO2023103149 A1 WO 2023103149A1
Authority
WO
WIPO (PCT)
Prior art keywords
decision
pulse
count
making
neural network
Prior art date
Application number
PCT/CN2022/072038
Other languages
English (en)
French (fr)
Inventor
周芃
邢雁南
乔宁
任宇迪
柯政
胡雅伦
李波
刘雨杭
龚熙文
尤艾尔阿明 西克·萨迪克
理查德•缪尔迪兰
代米尔吉图芭
Original Assignee
成都时识科技有限公司
深圳时识科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 成都时识科技有限公司, 深圳时识科技有限公司 filed Critical 成都时识科技有限公司
Priority to US18/270,888 priority Critical patent/US20240086690A1/en
Publication of WO2023103149A1 publication Critical patent/WO2023103149A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present invention relates to a pulse event decision-making device, method, chip and electronic equipment, in particular to a pulse event decision-making device, method, chip and electronic equipment for adaptive decision-making at the edge side of the Internet of Things (IoT) according to the speed of event occurrence.
  • IoT Internet of Things
  • Neuromorphic Computing is a new type of computing architecture developed recently. This computing method uses bionic (eye, brain, etc.) methods to complete reasoning on input signals, breaking through the von Neumann architecture for the first time.
  • Event imaging devices including but not limited to dynamic vision sensors, generate pulse events according to the light and dark changes at the corresponding positions of the pixels, and are in a silent state when there is no change in light and dark, and do not generate redundant data.
  • Frame-based image processing logic is completely different.
  • Neuromorphic chips commonly known as brain-like chips, are deployed with a novel neural network, which is the latest generation of neural networks - Spiking Neural Networks (SNN).
  • SNN neural networks - Spiking Neural Networks
  • Neurons in traditional Artificial Neural Networks (ANN) are computing units based on mathematical functions, while neurons in spiking neural networks are more bionic: if the accumulated membrane voltage (memory) of the input exceeds the threshold (sparseness) ), it will emit pulses (bionicity), which is highly similar to the dynamic characteristics of biological neurons.
  • neuromorphic computing is highly bionic, it also has the characteristics of ultra-low power consumption of biological brains, which coincides with the demand for edge computing in the Internet of Things (IoT). Since it is deployed at the edge, it naturally has low latency advantages compared to cloud solutions, but extremely ultra-low latency information processing is also the goal pursued by neuromorphic computing. In addition, the accuracy of inference results is also a basic and core performance indicator.
  • IoT Internet of Things
  • a pulse event decision-making device the pulse event decision-making device makes a decision according to the output pulse event of the pulse neural network in the neuromorphic chip
  • the pulse event decision-making device includes: a first counting module, which is configured to count the pulse of the pulse neural network The number of input pulse events is counted; the decision-making module is configured to, when the count of the first counting module reaches the first preset value, according to the counting of several neurons in the output layer of the pulse neural network respectively issuing pulse events Making a decision result; wherein, the several neurons are part or all of the neurons in the output layer.
  • the pulse event decision device further includes: a second counting module, which is configured to perform one of the following operations to calculate the number of pulse events issued by several neurons in the output layer of the spiking neural network Counting: (i) count the pulses emitted by each neuron in the output layer, and then add up to obtain the total count of pulses emitted by some or all neurons in the output layer; (ii) uniformly and directly count the pulses emitted by several neurons in the output layer Count to get the total count; (iii) count the spikes fired by each neuron in the output layer, but do not add them.
  • a second counting module which is configured to perform one of the following operations to calculate the number of pulse events issued by several neurons in the output layer of the spiking neural network Counting: (i) count the pulses emitted by each neuron in the output layer, and then add up to obtain the total count of pulses emitted by some or all neurons in the output layer; (ii) uniformly and directly count the pulses emitted by several neurons
  • the decision-making module is configured to count when the count of the first counting module reaches the first preset value, or the count of the first counting module reaches the first preset value and the count of the second counting module reaches the second When the preset value is reached, the decision-making module is triggered to make a decision.
  • the second counting module uses a plurality of sub-counters to count the pulse events emitted by each of the several neurons in the output layer; removes the earliest count in the sub-counters, and uses the sub-counters that are reset to zero Count the newly issued pulse events; the sum of the counts in all sub-counters corresponding to each neuron in the output layer is used as the count based on the decision result.
  • one or more of the first counting module, the second counting module, and the decision-making module in the pulse event decision device are implemented inside or outside the neuromorphic chip.
  • a pulse event decision-making device the pulse event decision-making device makes a decision according to the output pulse event of the pulse neural network in the neuromorphic chip
  • the pulse event decision-making device includes: a second counting module, which is configured to count the pulse of the pulse neural network Some or all of the output pulse events are counted; the decision-making module is configured to, when the total count of the second counting module reaches the second preset value, according to the counts of the respective pulse events issued by the neurons in the output layer of the pulse neural network make decisions.
  • the second counting module uses a plurality of sub-counters to count the pulse events emitted by each of the several neurons in the output layer; removes the earliest count in the sub-counters, and uses the sub-counters that are reset to zero Count the newly issued pulse events; the sum of the counts in all sub-counters corresponding to each neuron in the output layer is used as the count based on the decision result.
  • the second counting module or/and the decision module in the pulse event decision device are implemented inside or outside the neuromorphic chip.
  • a pulse event decision-making method makes decisions according to the output pulse events of the pulse neural network in the neuromorphic chip, counts the number of input pulse events of the pulse neural network, and obtains the first count value, When the first count value reaches the first preset value, a decision result is made according to the counts of the number of neurons in the output layer of the spiking neural network that emit spiking events; or, the output layer part or the output layer of the spiking neural network The output pulse events of all neurons are counted, and a second count value is obtained. When the second count value reaches a second preset value, the output pulse events of the neurons in the output layer of the pulse neural network are counted according to the output pulse events. Decision results.
  • each spiking event of some or all neurons in the layer is counted by multiple sub-counters; the earliest count in the sub-counter is removed, and the new spiking event is counted by the sub-counter reset to zero; each spiking event in the output layer
  • the sum of counts in all sub-counters corresponding to neurons is used as the count on which the decision result is based.
  • a chip on which a pulse neural network is deployed, and also includes any one of the aforementioned pulse event decision-making devices, or applies the above-mentioned pulse event decision-making method, and the pulse event decision-making device or pulse event decision-making method is used for pulse nerve
  • the output pulse events of the network make decisions.
  • the pulse event decision-making device here may be distributed in multiple different physical devices (such as chips) to jointly complete the decision-making function.
  • the chip includes any one of the aforementioned pulse event decision-making devices and a time-window-based decision-making device, and the chip is configured to allow switching between the pulse event decision-making device and the time-window-based decision-making device make a choice.
  • An electronic device which includes an event imaging device, and includes the pulse event decision-making device and chip as described in any one of the preceding items, or applies any of the above-mentioned pulse event decision-making methods; the pulse event decision-making device or chip Or the pulse event decision method is used to make a decision on the output pulse events obtained by the event imaging device after processing.
  • a pulse neural network training method comprising the following steps: for the data set collected by an event imaging device, a plurality of images are respectively constructed according to the number of fixed pulse events; the artificial neural network is trained according to the plurality of images; the artificial neural network is The network is converted into a spiking neural network; and the converted spiking neural network is deployed to a neuromorphic chip.
  • the final decision-making result can be determined according to the speed of the event
  • the sliding counting window can effectively improve the overall reliability of decision-making results.
  • Fig. 1 is a schematic diagram of a scheme based on a time window in the prior art
  • Fig. 2 is a schematic diagram of the low-latency decision-making device proposed by the present invention.
  • Fig. 3 is a schematic diagram of a sliding counting window scheme in a certain type of embodiment
  • Fig. 4 is a multi-window counting scheme of the second counting module in a certain embodiment.
  • Fig. 5 is a schematic diagram of a decision-making scheme in a certain type of embodiment
  • Figure 6 is a schematic diagram of high-frequency decision-making results that cannot pass low-pass filtering
  • Fig. 7 is a schematic diagram of low-frequency decision-making results passing through low-pass filtering
  • Fig. 8 is a schematic diagram of a training scheme for matching count-based decisions.
  • the present invention will describe various key points for being combined into various specific embodiments, and these key points will be combined into various methods and products. In the present invention, even if only the key points described when introducing the method/product solution mean that the corresponding product/method solution also explicitly includes the technical feature.
  • modules described in the present invention can be implemented by hardware, such as FPGA, ASIC or other IC design methods, such as realizing by designing the interface circuit between the sensor and the processor, and can be designed in In the same bare crystal, it can also be designed in different bare crystals, or it can be realized by combining software and hardware.
  • the technical solution described in the present invention can adopt different physical implementation forms. The method is not limited.
  • FIG. 1 it is a schematic diagram of a solution based on a time window in the prior art.
  • Event imaging devices such as dynamic vision sensors, sense changes in environmental signals and generate sparse sequences of input spike events (or spikes), which are fed into neuromorphic chips.
  • On the neuromorphic chip a large number of simulated synapses, neuron circuits and other auxiliary circuits are deployed. These pulse events are sent to the spiking neural network, and the pulse events are issued and transmitted in sequence according to the configured spiking neural network. operate.
  • At the front end of the network there is an input layer.
  • the leftmost side of the spike neural network in Figure 1 shows an example of neurons represented by three black circles to form the input layer.
  • the neuron At the end of the network, there is an output layer, and several neurons included in it will also emit pulses or pulse trains.
  • the neurons represented by 2 black circles are shown as an example to form the output layer.
  • several hidden layers may be included.
  • the spiking neural network exemplarily draws neurons represented by gray circles to form three hidden layers.
  • the technical solution based on the time window usually sets a time window (such as 100ms, 10ms, 2ms, etc.), reads the count of pulses or pulse events output by each neuron in the output layer within the time window, and based on these counts Or the output of the neuron is sent to the decision-making module to make a decision.
  • Decision-making strategies may be different, and the present invention is not limited to a specific decision-making strategy.
  • the time window means inherent delay (or delay), especially when the time window is large, this delay may cause the solution to not match the scene requirements. For example, in autonomous driving scenarios, extremely short response times are required for the detection of foreign objects such as pedestrian intrusion and rolling stones. That is, for events that occur faster, if the time window is longer at this time, the inference result will be slower to output the decision result due to the inherent time window, and it is difficult to meet the demand for lower latency.
  • This scheme also has another shortcoming: the speed of the event can be fast or slow, such as the waving action can be slow or fast, and it varies from time to time and from person to person. For example, for actions/events that occur slowly, the time window is relatively short, and the data volume of the output pulse event during this period may be insufficient. Since a decision must be made within this short period of time, the reasoning accuracy is not high, and the result is not good. reliable.
  • the present invention proposes a counting-based decision-making device, method, chip and electronic equipment.
  • an event imaging device such as a DVS (taken as an example in the following), receives the light and shade changes brought by the target object in the environment, and converts this change into a pulse event (or pixel event, pulse), And pass the event to the neuromorphic chip through a transmission scheme such as USB.
  • DVS and neuromorphic chip are integrated together to form an overall solution, which can reduce signal jitter caused by USB transmission.
  • CN2021110755934 the disclosure is hereby incorporated by reference in its entirety into this application.
  • no matter whether the event imaging device is integrated on the chip or not, as long as the chip deployed with the spiking neural network can be regarded as a neuromorphic chip.
  • the number of input pulse events of the spiking neural network is counted by the first counting module.
  • the counting technology is a well-known technology in the art, and it can be realized by software, hardware or a combination of software and hardware. It can be implemented in the chip or outside the chip. The present invention does not limit the technical implementation means.
  • the neuromorphic chip receives the input pulse event and sends it to the spiking neural network.
  • the spiking neural network performs a reasoning process on the input pulse event and transmits the information to the output layer (a collection of several neurons) of the spiking neural network.
  • the spiking neural network here is built based on various hardware circuits (such as synaptic circuits and neuron circuits) and specific neural network models to organize and manage these neuron circuits and synaptic circuits.
  • a spiking neural network includes an input layer, a hidden layer, and an output layer.
  • the leftmost side of the spiking neural network in Figure 2 exemplarily draws neurons represented by 3 black circles to form the input layer, and the rightmost example of the spiking neural network is Draw the neurons represented by 2 black circles to form the output layer, and draw the neurons represented by the gray circles as an example in the middle of the spiking neural network to form 3 hidden layers.
  • the spiking sequence is issued through several neurons in the output layer.
  • Each spike in a spike train typically carries the address of the neuron that fired the spike.
  • the counts of all pulse addresses (different addresses correspond to different neurons in the output layer) that emit pulses in the time window are counted, that is, the number of pulse events that are respectively emitted by several neurons in the output layer of the pulse neural network.
  • Count and decide the result by the decision-making module. For example, within the time window, the result corresponding to the neuron with the largest number of firing pulses is used as the decision result.
  • the first counting module counts the number of input pulse events of the spiking neural network (including the pulse events of the chip entering the neuromorphism and not all of which are input into the spiking neural network), and the input
  • the pulse event can be an input pulse event generated by the event imaging device but preprocessed by filtering, image segmentation, etc., or an input pulse event generated by the event imaging device but not preprocessed by noise filtering, image segmentation, etc.
  • the input pulse event of the neural network is substantially the same as the input pulse event of the neuromorphic chip), which is not limited in the present invention.
  • the second counting module counts the pulses emitted by some or all neurons of the output layer.
  • the counting mode of the second counting module can be at least: (1), respectively counting the pulses issued by each neuron in the output layer, and then adding up to obtain the total count of the pulses issued by some or all neurons in the output layer; (2) ), also can be the unified direct counting of the pulse that some or all neurons of the output layer emit, and obtain the total count; (3), respectively count the pulse of each neuron in the output layer, but do not need to add.
  • Different counting methods are applied to different types of embodiment solutions to match different needs.
  • the total count here can be the count of the pulses emitted by part of the output layer (such as for some special needs) or all the pulse neurons, but it is preferably the count of the pulses emitted by all the pulse neurons in the output layer.
  • the total count of the second counting module is the sum of the counts of firing events of some or all neurons in the output layer.
  • the first counting module or/and the second counting module are implemented in the neuromorphic chip (on-chip counting); and in another type of embodiment, the first counting module or/and the second counting module Modules are implemented outside of the neuromorphic chip (off-chip count). In the present invention, preferably, both the first counting module and the second counting module are on-chip counting schemes.
  • the pulse event decision-making device may only have the above-mentioned second counting module (this solution is not shown in the figure), and may also preferably have both the first counting module and the second counting module (shown in Figure 2 ).
  • the decision-making module is triggered to make a decision. For example, when the count of the first counting module reaches a first preset value, or in another embodiment, when the total count of the second counting module reaches a second preset value, the decision-making module is triggered to make a decision.
  • the first counting module counts the input pulse events of the spiking neural network, and when the count value reaches the first preset value ⁇ 1 , the count values ( It can be the third counting method of the second counting module). Based on the obtained count values, a decision result is made in the decision module. For example, the result corresponding to the neuron with the largest number of pulses in the output layer may be used as the decision result, for example, the first neuron corresponds to the first category, and the second neuron corresponds to the second category.
  • the second counting module performs a total count of the pulse events emitted by some or all neurons in the output layer (for example, the first and second counting modes of the second counting module), if the total count reaches the first
  • the preset value ⁇ is 2
  • the count values of pulse events emitted by several neurons in the output layer are respectively obtained, and a decision is made in the decision-making module based on the obtained count values.
  • the counting-based decision-making mechanism can be realized: when the target object moves faster at the same position in the DVS receptive field, more pulse events are generated per unit time, so the counting of the first counting module or/and the second counting module The count reaches the corresponding first preset value or second preset value sooner, then the decision-making module can obtain the output reasoning result in time, which can break the inherent delay of the time window; when the target object is in the same position of the DVS receptive field When the movement speed is slow, there are fewer pulse events per unit time, so the counting of the first counting module or/and the counting of the second counting module reach the corresponding first preset value or second preset value slowly, and this It will overcome the defect that the number of pulse events counted/accumulated by the decision-making module in a fixed time window is insufficient and the inference results are not accurate.
  • the decision-making module can be implemented as on-chip decision-making and off-chip decision-making.
  • the so-called on-chip decision-making means that the reasoning is completed inside the neuromorphic chip, and the decision-making or reasoning results are given; while the off-chip decision-making means that the chip only completes the reasoning process of the neural network, and transmits the pulse events issued by the output layer pulse neurons to the outside of the chip.
  • the decision-making is completed by the latter-level system (such as MCU, FPGA, CPU, etc.).
  • the decision-making modules (some embodiments also include low-pass filtering devices), the first counting module, the second counting module, etc. belonging to the pulse event decision-making device can be partially or completely implemented on-chip or in accordance with actual needs.
  • the pulse event decision-making device may be implemented in the neuromorphic chip alone, or dispersed in multiple different physical components, which is not limited in the present invention.
  • the off-chip or on-chip solutions can be selected according to the characteristics of the chip and the needs of actual application scenarios. For example, a neuromorphic chip does not support the counting-based decision-making solution of the present invention, but the present invention can be implemented in an off-chip manner Adaptive decision-making schemes, all of which are within the protection scope of the present invention.
  • a sliding counting window is introduced to consider temporal correlation.
  • the first counting window W1 completes one cumulative count, and at time t1', a decision is made.
  • the count value (the count value of the first counting module or the total count value of the second counting module) is not cleared or reset to zero, but a certain value (fixed or floating, such as ⁇ 1 /4, ⁇ 2 /4), continue counting.
  • the necessary adjustments are also made to adapt to the window sliding.
  • FIG. 4 it discloses a counting scheme for adapting to a sliding counting window in the second counting module. Because the aforementioned counting value is not cleared, but a certain value is subtracted, after sliding to the next window, the counting starting point of each neuron in the output layer should also be adjusted, starting from the new counting starting point ( instead of being forced to zero) to start counting.
  • each neuron in the output layer corresponds to setting 4 sub- Counters (recorded as sub-counts 1 to 4), each sub-counter corresponds to 1/4 counting window.
  • Figure 4 exemplarily shows that the output layer of the spiking neural network has 3 neurons, where the black circle, white circle and gray circle represent the first neuron, the second neuron and the third neuron in the output layer respectively , and the first neuron corresponds to category 1, the second neuron corresponds to category 2, and the third neuron corresponds to category 3.
  • the earliest counted sub-count 1 is removed, other sub-counts are sequentially shifted (moving to the left in the figure), and the new sub-count is moved into sub-count 4.4
  • the count of the neuron corresponding to the moved window is subcount 2+subcount 3+subcount 4+new subcount.
  • the earliest count in the sub-counters is removed first, and the newly issued pulse events are counted by using the zeroed sub-counter (which may be the aforementioned sub-counter 1, or the sub-counter 4 after sequentially shifted and stored).
  • the counting window moves, and at t2', the subtracted certain value is supplemented by the new count, so A decision is made at time t2', for example, the counting window is moved between t2-t2', such as the second counting window W2.
  • a decision is also made at t3' and t4', which represent the counting windows of W3 and W4 respectively.
  • the aforementioned time intervals between t1-t4' are not necessarily equal in length.
  • Fig. 5 and Fig. 7 take the output layer including 3 neurons as an example, wherein the black circle, white circle and gray circle respectively represent the first neuron and the second neuron in the output layer unit and the third neuron.
  • the decision timing the counting results are shown respectively.
  • a decision result is made according to the decision logic.
  • some preferred schemes also include a low-pass filtering device/step (software, hardware, a combination of software and hardware, which can be part of the decision-making module or independent of the decision-making module, can be located inside the neuromorphic chip, or can be located outside the neuromorphic chip), and is used to filter unstable decision outcomes.
  • the decision result repeatedly jumps between different results in a continuous short period of time (that is, the decision result has changed), then such a result has high frequency characteristics, such as the decision result repeatedly jumps between classification 1 and classification 2 : 1-2-1-2-1-2-1-2-1-1-2-1-3 (refer to FIG. 6 ). Then such a decision result, that is, multiple consecutive decision results whose jump rate is not lower than the first threshold, cannot pass the low-pass filter, so the unreliable result will not be output, or a decision result indicating that the result is uncertain will not be output. On the contrary, the decision result with the largest number of occurrences among the multiple consecutive decision results should be output.
  • the decision-making module that makes multiple decision-making results continuously outputs a certain decision-making result, such as 2-2-2-2-1-2-2-2-2-2-2 -2-2-2 (refer to Figure 7), this result has low-frequency characteristics, and can be filtered by low-pass to output the most reliable result, such as the classification 2 represented by the corresponding output layer neuron.
  • the decision result jumps can be counted, and both the number of jumps and the jump rate can be used as the above indicators.
  • the aforementioned decision result sequence that cannot pass the low-pass filter has 11 jumps in 13 decision results; while the aforementioned decision result sequence that can be low-pass filtered has 2 jumps in the 14 decision results.
  • the ratio of the decision result can be used as the aforementioned indicator.
  • the ratio of the decision result to category 1 is 54%, and the ratio of category 2 to 38%.
  • the ratio of the decision result to category 2 is 93%. If the set ratio is higher than the second threshold 70%, the decision result with the largest number of consecutive decision results will be output, and if it is lower than 70%, the unreliable result will not be output, or the output indicating that the result is uncertain Decision results.
  • the above-mentioned first threshold and the second threshold may be used in combination, that is, the final decision result can only be output when they are satisfied at the same time.
  • FIG. 8 it discloses a count-based spiking neural network training method and device.
  • the events collected by the event imaging device such as DVS are firstly generated into a data set, and then frames are compressed according to a fixed number of events to form an image, and the event imaging device of this kind is used to generate a data set.
  • the resulting dataset of images is fed into an artificial neural network (ANN) model for training.
  • ANN artificial neural network
  • the present invention is not limited to a specific artificial neural network, so any suitable ANN model and its training method are applicable in the present invention, such as convolutional neural network and error backpropagation training.
  • the ANN model is converted into an SNN model, which may involve quantization of weights, conversion of neuron models (such as activation functions) and other processes in this step, and the conversion of ANN to SNN is a technical means well known in the art.
  • the means of implementation are not limited.
  • the trained neural network configuration parameters are finally deployed to the neuromorphic chip, which acquires reasoning capabilities and can be put into practical application scenarios, such as gesture detection and fall detection.
  • a method for training a spiking neural network comprising the steps of: constructing a plurality of images according to a fixed number of pulse events for the data set collected by an event imaging device; training artificial intelligence according to the plurality of images A neural network; converting the artificial neural network into a spiking neural network; deploying the converted spiking neural network into a neuromorphic chip.
  • the present invention also discloses: a pulse event decision-making method, the pulse event decision-making method makes decisions based on the output pulse events of the pulse neural network in the neuromorphic chip, counts the number of input pulse events of the pulse neural network, and Obtaining the first count value, when the first count value reaches the first preset value, making a decision result according to the counts of the respective pulse events of several neurons in the output layer of the spiking neural network; or, making a decision on the spiking neural network
  • the output pulse events of some or all neurons in the output layer of the network are counted, and a second count value is obtained.
  • the second count value reaches a second preset value, according to the neurons in the output layer of the pulse neural network, each neuron fires The count of pulse events makes the decision result.
  • each spiking event of some or all of the neurons is counted by multiple sub-counters; the earliest count in the sub-counter is removed, and the new spiking event is counted by the zeroed sub-counter; each of the output layer The sum of the counts in all sub-counters corresponding to the neuron is used as the count on which the decision result is made.
  • a chip on which a pulse neural network is deployed, and also includes any one of the aforementioned pulse event decision-making devices, or applies the above-mentioned pulse event decision-making method, and the pulse event decision-making device or pulse event decision-making method is used for pulse nerve
  • the output pulse events of the network make decisions.
  • the chip is a neuromorphic chip.
  • the above-mentioned traditional technical solution based on time window and the technical solution based on pulse counting in the present invention can also be configured as optional decision-making solutions, and the choice is determined by the user or other factors (such as the needs of actual application scenarios) Specifically which decision-making scheme to use, which increases the flexibility of the chip in use.
  • An electronic device which includes an event imaging device, and includes the pulse event decision-making device and chip as described in any one of the preceding items, or applies any of the above-mentioned pulse event decision-making methods; the pulse event decision-making device or chip Or the pulse event decision method is used to make a decision on the output pulse events obtained by the event imaging device after processing.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Artificial Intelligence (AREA)
  • Neurology (AREA)
  • Image Analysis (AREA)

Abstract

本发明涉及一种脉冲事件决策装置、方法、芯片及电子设备。为了解决神经拟态芯片输出脉冲事件在读出决策时的固有延时问题，本发明采用的第一计数模块，被配置为对脉冲神经网路的输入脉冲事件的个数进行计数；第二计数模块，被配置为对脉冲神经网络输出层中若干神经元各自发放脉冲事件的个数进行计数；决策模块，其被配置为在第一计数模块的计数达到第一预设值时或在第二计数模块的总计数达到第二预设值时，依据脉冲神经网络输出层中若干神经元各自发放脉冲事件的计数做出决策结果。本发明基于计数、滑动计数窗等技术手段，解决了决策时的固有延时或决策数据不足问题，实现了根据事件发生快慢而自适应地做出最终决策的技术效果。

Description

脉冲事件决策装置、方法、芯片及电子设备
本发明要求以2021年12月06日向中国提出的标题为“脉冲事件决策装置、方法、芯片及电子设备”、申请号为202111476156.3的发明专利申请为优先权，并将申请文件全部内容引入本发明中。
技术领域
本发明涉及一种脉冲事件决策装置、方法、芯片及电子设备，具体涉及一种在物联网(IoT)边缘侧根据事件发生快慢而自适应决策的脉冲事件决策装置、方法、芯片及电子设备。
背景技术
神经拟态计算(Neuromorphic Computing)是近期发展起来的一种新型计算架构，这种计算方式采用仿生(眼、脑等)的方式来完成针对输入信号的推理，首次突破冯诺伊曼架构。事件成像装置，包括但不限于动态视觉传感器，根据像素对应位置处光线的明暗变化而生成脉冲事件，在明暗没有变化时则处于静默状态，不产生冗余数据，这种事件驱动特点与传统的基于帧的图像处理逻辑完全不同。
神经拟态芯片，俗称类脑芯片，其上部署有一种新颖的神经网络，也即最新一代的神经网络——脉冲神经网络(Spiking Neural Networks，SNN)。传统的人工神经网络(Artificial Neural Networks，ANN)中神经元是基于数学函数的计算单元，而脉冲神经网络中的神经元则更加仿生：如果输入累积的膜电压(记忆性)超过阈值(稀疏性)，则会发放脉冲(仿生性)，这与生物神经元的动力学特性高度相似。
因为神经拟态计算高度仿生，因此也具有生物脑的超低功耗特点，这与物联网(IoT)中对边缘计算的需求不谋而合。由于部署在边缘端，相比于云端解决方案天然更具有低延迟优势，但极致超低延迟的信息处理，同样是神经拟态计算所追求的目标。此外，推理结果的准确性，同样是一个基础且核心的性能指标。
然而，现有技术中依然存在某些不合理的固有延时(图1)，消除该些延时，将有助于芯片/系统更快地获得决策结果，为后一级系统做出更及时的响应赢取更为宝贵的时间。此外，可能因为这种固有延时，在某些情况下会导致芯片的推理结果不可靠。因此，降低芯片处理延迟实现超低延迟决策或/和提升芯片推理结果可靠性，均是本领域所期待实现的技术目标。
发明内容
为了解决或缓解上述部分或全部技术问题，本发明是通过如下技术方案实现的：
一种脉冲事件决策装置，该脉冲事件决策装置依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，该脉冲事件决策装置包括：第一计数模块，其被配置为对所述脉冲神经网络的输入脉冲事件的个数进行计数；决策模块，其被配置为在第一计数模块的计数达到第一预设值时，依据所述脉冲神经网络的输出层中若干神经元各自发放脉冲事件的计数做出决策结果；其中，所述若干神经元为输出层中部分或全部神经元。
在某类实施例中，所述脉冲事件决策装置还包括：第二计数模块，其被配置为执行如下操作之一对所述脉冲神经网络的输出层中若干神经元发放脉冲事件的个数进行计数：(i)对输出层中每个神经元发放的脉冲进行计数，然后相加获得输出层部分或全部神经元发放脉冲的总计数；(ii)对输出层若干神经元发放的脉冲统一直接计数，得到总计数；(iii)对输出层中每个神经元发放的脉冲进行计数，但不相加。
在某类实施例中，决策模块，被配置为在第一计数模块的计数达到第一预设值，或第一计数模块的计数达到第一预设值且第二计数模块的计数达到第二预设值时，触发决策模块做出决策。
在某类实施例中，至少存在某相邻两次决策时，第一计数模块中对应的两次计数所对应的输入脉冲事件之间有部分重叠；或/和，第二计数模块中对应的两次计数所对应的脉冲神经网络的输出层若干神经元发放的脉冲事件之间有部分重叠。
在某类实施例中，第二计数模块对所述输出层中若干神经元的每一个发放的脉冲事件使用多个子计数器进行计数；移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
在某类实施例中，若连续多个决策结果满足如下条件中的一个或多个，则输出所述连续多个决策结果中出现次数最多的决策结果：(i)若所述连续多个决策结果的跳变率或跳变数低于第一阈值；(ii)若所述连续多个决策结果中出现次数最多的决策结果所占比率高于第二阈值；否则，不输出决策结果或输出指示结果不确定的决策结果。
在某类实施例中，所述脉冲事件决策装置中的第一计数模块、第二计数模块、决策模块中的一个或多个被实施在神经拟态芯片之内或之外。
一种脉冲事件决策装置，该脉冲事件决策装置依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，该脉冲事件决策装置包括：第二计数模块，其被配置为对所述脉冲神经网络的部分或全部输出脉冲事件进行计数；决策模块，其被配置为在第二计数模块的总计数达到第二预设值时，依据所述脉冲神经网络的输出层中神经元各自发放脉冲事件的计数做出决策结果。
在某类实施例中，第二计数模块对所述输出层中若干神经元的每一个发放的脉冲事件使用多个子计数器进行计数；移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
在某类实施例中，若连续多个决策结果满足如下条件中的一个或多个，则输出所述连续多个决策结果中出现次数最多的决策结果：(i)若所述连续多个决策结果的跳变率或跳变数低于第一阈值；(ii)若所述连续多个决策结果中出现次数最多的决策结果所占比率高于第二阈值；否则，不输出决策结果或输出指示结果不确定的决策结果。
在某类实施例中，所述脉冲事件决策装置中的第二计数模块或/和决策模块被实施在神经拟态芯片之内或之外。
一种脉冲事件决策方法，该脉冲事件决策方法依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，对所述脉冲神经网络的输入脉冲事件的个数进行计数，并获得第一计数值，在第一计数值达到第一预设值时，依据所述脉冲神经网络的输出层中若干神经元各自发放脉冲事件的计数做出决策结果；或者，对所述脉冲神经网络的输出层部分或全部神经元的输出脉冲事件进行计数，并获得第二计数值，在第二计数值达到第二预设值时，依据所述脉冲神经网络的输出层中神经元各自发放脉冲事件的计数做出决策结果。
在某类实施例中，至少存在某相邻两次做出决策结果时，相邻两次获得的第一计数值所对应的输入脉冲事件之间有部分重叠；或/和，对所述输出层的部分或全部神经元的每一个发放的脉冲事件使用多个子计数器进行计数；移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
一种芯片，该芯片上部署有脉冲神经网络，且还包括前述任意一项脉冲事件决策装置，或应用上述脉冲事件决策方法，所述的脉冲事件决策装置或脉冲事件决策方法用于对脉冲神经网络的输出脉冲事件进行决策。
此处的脉冲事件决策装置，可能分布在多个不同的物理装置(如芯片)中，共同完成决策功能。
在某类实施例中，所述芯片包括前述任意一项脉冲事件决策装置，以及基于时间窗的决策装置，并且芯片被配置为允许在所述脉冲事件决策装置和基于时间窗的决策装置之间做出选择。
一种电子设备，该电子设备包括事件成像装置，以及包括如前任意一项所述的脉冲事件决策装置、芯片，或应用上述任一项脉冲事件决策方法；所述的脉冲事件决策装置或芯片或脉冲事件决策方法用于对事件成像装置经过处理后获得的输出脉冲事件进行决策。
一种脉冲神经网络训练方法，该方法包括如下步骤：对事件成像装置所采集的数据集，分别按照固定脉冲事件数量构建成若干幅图像；依据所述若干幅图像训练人工神经网络；将人工神经网络转换成脉冲神经网络；将转换成后的脉冲神经网络部署至神经拟态芯片中。
本发明的部分或全部实施例，具有如下有益技术效果：
1、可以根据事件发生的快慢，来决定最终的决策结果；
2、消除固有延迟，提升响应速度；
3、避免因脉冲事件数量不足影响决策精度；
4、滑动计数窗可以有效提升决策结果整体可靠性。
更多的有益效果将在优选实施例中作进一步的介绍。
以上披露的技术方案/特征，旨在对具体实施方式部分中所描述的技术方案、技术特征进行概括，因而记载的范围可能不完全相同。但是该部分披露的这些新的技术方案同样属于本发明文件所公开的众多技术方案的一部分，该部分披露的技术特征与后续具体实施方式部分公开的技术特征、未在说明书中明确描述的附图中的部分内容，以相互合理组合的方式披露更多的技术方案。
本发明任意位置所披露的所有技术特征所组合出的技术方案，用于支撑对技术方案的概括、专利文件的修改、技术方案的披露。
附图说明
图1是现有技术中基于时间窗口方案的示意图；
图2是本发明所提出的低延迟决策装置的示意图；
图3是某类实施例中滑动计数窗口方案示意图；
图4是某类实施例中第二计数模块多窗口计数方案。
图5是某类实施例中决策方案示意图；
图6是高频决策结果无法通过低通过滤的示意图；
图7是低频决策结果通过低通过滤的示意图；
图8是匹配基于计数决策的训练方案示意图。
具体实施方式
由于不能穷尽描述各种替代方案，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案中的要点内容进行清楚、完整地描述。对于下文未详细披露的其它的技术方案和细节，一般均属于本领域通过常规手段即可实现的技术目标或技术特征，限于篇幅，本发明不对其详细介绍。
除非是除法的含义，本发明中任意位置的“/”均表示逻辑“或”。本发明任意位置中的“第一”、“第二”等序号仅仅用于描述上的区分标记，并不暗示时间或空间上的绝对顺序，也不暗示冠以这种序号的术语与冠以其它定语的相同术语必然是不同的指代。
本发明会对各种用于组合成各种不同具体实施例的要点进行描述，这些要点将被组合至各种方法、产品中。在本发明中，即便仅在介绍方法/产品方案时所描述的要点，意味着对应的产品/方法方案也明确地包括该技术特征。
本发明中任意位置处描述存在或包括某步骤、模块、特征时，并不暗示这种存在是排它性地唯一存在，本领域技术人员完全可以根据本发明所披露的技术方案而辅以其它技术手段而获得其它实施例；基于本发明中具体实施例描述的要点，本领域技术人员完全可以对某些技术特征施加替换、删减、增加、组合、调换顺序等手段，获得一个仍遵循本发明构思的技术方案。这些未脱离本发明技术构思的方案也在本发明保护范围之内。
本发明所述的各种模块(以及其构成的装置)可以通过硬件，如FPGA、ASIC或其它IC设计方式来实施，比如通过设计传感器与处理器之间的接口电路来实现，可以被设计在同一个裸晶中，也可以被设计在不同裸晶中，也可以是通过软硬件相结合的方式实现，实现本发明所描述的技术方案可以采取不同的物理实施形态，本发明对具体的实施方式不做限定。
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
目前现有的技术中，系统做出决策是基于时间窗口的，而时间窗口意味着整个方案会存在固有延迟。
参考图1，其是现有技术中基于时间窗口的方案示意图。事件成像装置，比如动态视觉传感器，其感受环境信号的变化，并产生稀疏的输入脉冲事件(或脉冲)序列，这些脉冲事件被送入神经拟态芯片。而神经拟态芯片上部署有大量的模拟突触、神经元的电路以及其它辅助电路，这些脉冲事件被送入脉冲神经网络，根据被配置好的脉冲神经网络依序完成脉冲事件的发放、传递等操作。在网络的前端，有一个输入层，图1中脉冲神经网络最左侧示例地画出由3个黑色圆圈代表的神经元构成输入层。在网络的末端，有一个输出层，，其包括的若干神经元也会发放脉冲或脉冲序列，图1中脉冲神经网络最右侧示例地画出由2个黑色圆圈代表的神经元构成输出层。在输入层和输出层之间，可包含若干隐藏层，图1中脉冲神经网络示例地画出由灰色圆圈代表的神经元构成3层隐藏层。基于时间窗口的技术方案，通常是设定一个时间窗口(比如100ms、10ms、2ms等)，在该时间窗口内读取输出层每个神经元输出的脉冲或脉冲事件的计数，并基于这些计数或神经元的输出，送入决策模块做决策。决策策略可能各不相同，在本发明中不以某种特定决策策略为限制。
时间窗口意味着固有延迟(或延时)，尤其是当时间窗口较大时，这种延迟可能导致方案与场景需求不匹配。比如自动驾驶场景中，对于行人闯入、滚石等异物检测，需要极短的响应时间。即针对发生较快的事件，如果此时时间窗口较长，那么获得推理结果将会因固有时间窗口而较慢输出决策结果，难以满足更低延时的需求。
这种方案(基于时间窗的决策装置)还存在另外一个缺点：事件发生的速度有快有慢，比如挥手动作可慢可快，因时、因人而异。比如针对较慢发生的动作/事件，时间窗口相对较短，该期间的输出脉冲事件的数据量可能不足，由于在该较短的时间内就得做出决策，导致推理精度不高，结果不可靠。
为此，本发明提出一种基于计数的决策装置、方法、芯片及电子设备。
参考图2，事件成像装置，比如DVS(后文以此为例)，接收环境中目标对象带来的光线的明暗变化，并将这种变化转换成脉冲事件(或称像素事件、脉冲)，并通过USB等传输方案将事件传递给神经拟态芯片。在某类优选实施例中，DVS与神经拟态芯片集成在一起，形成整体解决方案，可以降低USB传输带来的信号抖动等问题，具体至少可以参见申请人其它专利文件(如CN2021110755934)。本发明以全文引入的方式，将其并入本申请。本发明中，不论是否片上集成事件成像装置，只要部署有脉冲神经网络的芯片都可以视为神经拟态芯片。
对于神经拟态芯片，在接收事件成像装置输出的脉冲事件后，在某类实施例中，由第一计数模块对脉冲神经网络的输入脉冲事件的个数进行计数。计数技术是本领域公知的技术，这里可以通过软件、硬件或软硬件相结合的方式实现，其可以被实施于芯片中，也可以实施在芯片之外，本发明不限定该技术实现手段。神经拟态芯片接收输入脉冲事件，并将其送入脉冲神经网络，脉冲神经网络对输入脉冲事件执行推理过程，并传递信息至脉冲神经网络的输出层(若干神经元的集合)中。此处的脉冲神经网络基于各种硬件电路(比如突触电路、神经元电路)和特定的神经网络模型，组织、管理这些神经元电路、突触电路，搭建而成的。举例地，脉冲神经网络包括输入层、隐藏层和输出层，图2中脉冲神经网络最左侧示例地画出由3个黑色圆圈代表的神经元构成输入层，脉冲神经网络最右侧示例地画出由2个黑色圆圈代表的神经元构成输出层，脉冲神经网络中间示例地画出由灰色圆圈代表的神经元构成3层隐藏层。
经过脉冲神经网络的推理运算，在输出层经由若干神经元发放脉冲序列。脉冲序列中的每个脉冲通常会携带发放该脉冲的神经元地址。在前述基于时间窗口的方案中，统计该时间窗口内所有发放脉冲的脉冲地址(不同地址对应输出层中不同的神经元)的计数，即脉冲神经网络输出层中若干神经元各自发放脉冲事件的计数，并由决策模块判决结果。比如，将该时间窗口内，发放脉冲计数最多的神经元对应的结果，作为决策结果。
在本发明某类实施例中，第一计数模块对脉冲神经网络的输入脉冲事件(包括进入神经拟态的芯片的脉冲事件并未全部输入脉冲神经网络情形)的个数进行计数，所述的输入脉冲事件可以是事件成像装置生成但经过滤、图像分割等预处理后的输入脉冲事件，也可以是事件成像装置生成但未经噪声过滤、图像分割等预处理后的输入脉冲事件(此时脉冲神经网络的输入脉冲事件与神经拟态芯片的输入脉冲事件实质等同)，本发明对此不做限定。
本发明中，在某类实施例中，由第二计数模块对输出层部分或全部神经元发放的脉冲进行计数。第二计数模块的计数方式，至少可以是：(1)、分别针对输出层中每个神经元发放的脉冲进行计数，然后相加获得输出层部分或全部神经元发放脉冲的总计数；(2)、也可以是对输出层部分或全部神经元发放的脉冲统一直接计数，得到总计数；(3)、分别针对输出层中每个神经元发放的脉冲进行计数，但不必相加。不同的计数方式应用于不同类型的实施例方案，以匹配不同的需要。这里的总计数，可以是输出层部分(比如出于某种特 殊需要)或全部脉冲神经元所发放脉冲的计数，但优选为输出层全部脉冲神经元所发放脉冲的计数。换言之，第二计数模块的总计数是输出层中部分或全部神经元各自发放脉冲事件的计数之和。
在某类实施例中，第一计数模块或/和第二计数模块被实施于神经拟态芯片之中(片内计数)；而另一类实施例中，第一计数模块或/和第二计数模块被实施在神经拟态芯片之外(片外计数)。本发明优选第一计数模块和第二计数模块均为片内计数方案。
在某类实施例中，脉冲事件决策装置可以仅存在上述第二计数模块(该方案未在图中示出)，也可以优选地同时存在第一计数模块和第二计数模块(图2所示)。
基于第一计数模块或/和第二计数模块的计数结果，触发决策模块做出决策。比如在第一计数模块的计数达到第一预设值时，或者另一类实施例中在第二计数模块的总计数达到第二预设值时，触发决策模块做出决策。
在某类实施例中，第一计数模块对脉冲神经网络的输入脉冲事件进行计数，当计数值到达第一预设值θ 1时，分别获取输出层中不同神经元发放脉冲事件的计数值(可以是第二计数模块的第3种计数方式)。基于所获取的若干计数值，在决策模块中做出决策结果。比如，可以将输出层发放脉冲数量最多的神经元所对应的结果作为决策结果，示例地第1个神经元对应的第1分类、第2个神经元对应的第2分类。
在某类实施例中，第二计数模块对输出层部分或全部神经元所发放的脉冲事件进行总计数(比如，第二计数模块的第1、2种计数方式)，若该总计数达到第二预设值θ 2时，分别获取输出层中若干神经元各自发放脉冲事件的计数值，基于所获取的若干计数值，在决策模块中做出决策。
如此以来，基于计数的决策机制可以实现：当目标对象在DVS感受野的同一位置运动速度较快时，单位时间产生的脉冲事件较多，因此第一计数模块的计数或/和第二计数模块的计数较快就达到对应的第一预设值或第二预设值，那么决策模块就可以及时地获得输出推理结果，可打破时间窗的固有延迟；当目标对象在DVS感受野的同一位置运动速度较慢时，单位时间产生的脉冲事件较少，因此第一计数模块的计数或/和第二计数模块的计数较慢达到对应的第一预设值或第二预设值，而这将克服固定时间窗口内决策模块所统计/积攒的脉冲事件数量不足进而推理结果不精确的缺陷。
对于决策模块的具体实施方式，其可以是任意一种合理的决策逻辑，本申请不做限定。决策模块可以被实施为片内决策和片外决策。所谓片内决策即在神经拟态芯片内部完成推理，给出决策或推理结果；而片外决策则意味着芯片只完成神经网络的推理过程，并向芯片外传递输出层脉冲神经元发放的脉冲事件，在神经拟态芯片之外由后一级系统(如MCU、FPGA、CPU等)完成决策。
事实上，属于脉冲事件决策装置的决策模块(某些实施例还包括低通过滤装置)、第一计数模块、第二计数模块等，均可以根据实际需要而选择部分或全部实施于片内或片外，由此脉冲事件决策装置可能单独实施在神经拟态芯片中，或者分散在多个不同的物理部件中，本发明对此不做限定。片外或片内方案可以根据芯片的特点、实际应用场景的需要而做出不同的选择，比如某神经拟态芯片内部不支持本发明基于计数的决策方案，但通过片外的方式可以实施本发明的自适应决策方案，这些均在本发明保护范围之内。
参考图3，其展示的是某类改进的更优实施例。该类实施例中，引入滑动计数窗而可以考虑时间前后关联性。在t1-t1’时间段内，第一计数窗W1完成一次累积计数，在t1’时刻，做出一次决策。而此后并不将计数值(第一计数模块的计数值或第二计数模块的总计数值)清零或归零，而是减去某个数值(固定的或浮动的，比如θ 1/4、θ 2/4)后，继续计数。而对于输出层中不同神经元发放脉冲事件的计数值，则也同样做出必要的调整，以适应窗口 滑动。
参考图4，其公开的第二计数模块中为适应滑动计数窗而设置的计数方案。因为前述计数值并未被清零，而是减去某个数值，因此在滑到下一个窗口后，对于输出层中每个神经元的计数起点，也应该做出调整，从新的计数起点(而非强制归零)开始计数。若一个脉冲事件被4个连续滑动计数窗口有效统计过(比如每次减去第一计数值/4或第二计数值/4，那么4次后将恰好彻底滑过某一个窗口；在子计数器中表现为某次计数，经历4次求和统计才被移除；t4-t1’中某次计数，经历W1～W4一共4次滑窗)，那么输出层的每个神经元对应设置4个子计数器(被记为子计数1～4)，每个子计数器对应1/4计数窗口。图4示例地画出脉冲神经网络的输出层具有3个神经元，其中，黑色圆圈、白色圆圈和灰色圆圈分别代表输出层中的第1个神经元、第2个神经元和第3个神经元，且第1个神经元对应分类1、第2个神经元对应分类2及第3个神经元对应分类3。
在某类实施例中，当滑动一次窗口时，移除最早被计数的子计数1，其它子计数顺序移位(图中为向左移动)，并将新的子计数移入子计数4。4个子计数全部相加得到的计数，是当前窗口下输出层该神经元对应的脉冲事件计数。比如，对于输出层某个神经元，其子计数1+子计数2+子计数3+子计数4＝该神经元的计数，示例地计数1、计数2、计数3分别对应输出层中三个不同的神经元。而滑动一次窗口后，假如移动了1/4窗口，那么移动后窗口对应的该神经元的计数＝子计数2+子计数3+子计数4+新子计数。换言之，先移除子计数器中最早的计数，并利用归零的子计数器(可以是前述的子计数器1，或者顺序移位存储后的子计数器4)对新发放的脉冲事件计数。
继续参考图3，第一计数模块的计数值或第二计数模块的总计数值减去某个数值后，计数窗移动，在t2’时刻，被减去的某个数值被新的计数补足，因此t2’时刻做出一次决策，例如，计数窗移动到t2-t2’之间，如第二计数窗W2。依次类推，在t3’、t4’时刻同样各做出一次决策，并分别代表W3、W4计数窗。然而值得一提的是，脉冲事件因发生速率的不同，前述的t1～t4’之间各个时间间隔，如图所示的那样，并不一定等长。
参考图5、图6和图7，举例地，以输出层包括3个神经元为例，其中，黑色圆圈、白色圆圈和灰色圆圈分别代表输出层中的第1个神经元、第2个神经元和第3个神经元。参考图5，按照决策时序，分别对其进行计数结果做了展示。在每个决策时刻，根据决策逻辑做出决策结果。为了输出更可靠的结果，某类优选方案还包括一个低通过滤装置/步骤(软件、硬件、软硬件相结合，可以是决策模块的一部分或独立于决策模块，可以位于神经拟态芯片内部，也可以位于神经拟态芯片外部)，被用于过滤不稳定的决策结果。如果决策结果在某连续短时间内，反复在不同结果之间跳变(即决策结果发生了改变)，那么这样的结果具有高频特点，比如决策结果反复在分类1和分类2之间跳变：1-2-1-2-1-2-1-2-1-1-2-1-3(参考图6)。那么这样的决策结果，即连续多个决策结果跳变率不低于第一阈值，无法通过低通过滤，因而不会将该不可靠结果输出，或者输出指示结果不确定的决策结果。反之，则应输出所述连续多个决策结果中出现次数最多的决策结果。
如果在某连续短时间内，连续做出的多个决策结果的决策模块比较稳定地输出某一决策结果，如2-2-2-2-1-2-2-2-2-2-2-2-2-2(参考图7)，这种结果具有低频特性，可以通过低通过滤，输出最可靠结果，比如对应输出层神经元所代表的分类2。
衡量输出结果是否稳定的指标，可以采取多种手段，本发明不做具体限定。举例地，可以对决策结果跳变进行计数，不论是跳变数还是跳变率，都可以作为上述指标。比如，前述无法通过低通过滤的决策结果序列，在13次决策结果中出现了11次跳变；而前述可以低通过滤的决策结果序列，在14次决策结果中出现了2次跳变。
在另一个示例中，可以通过决策结果比率作为前述指标，前述第一例子中决策结果为 分类1的比率为54％、分类2的比率38％；第二例子中决策结果为分类2的比率为93％。若设定比率高于第二阈值70％则输出所述连续多个决策结果中出现次数最多的决策结果，而低于70％则不会将该不可靠结果输出，或者输出指示结果不确定的决策结果。而在另一实施例汇总，上述第一阈值和第二阈值可以被结合使用，即同时满足方可输出最终决策结果。
参考图8，其公开的是一种基于计数的脉冲神经网络的训练方法及装置。在该方案中，为了匹配前述的基于计数的决策方案，首先将经DVS这种事件成像装置所采集的事件生成数据集，然后按照固定事件数量进行压帧，形成一幅图像，并将该种图像所形成的数据集送入人工神经网络(ANN)模型进行训练。本发明不以某种特定的人工神经网络为限制，因此任意合适的ANN模型及其训练方法，在本发明中均可适用，比如卷积神经网络及误差反向传播训练。然后将ANN模型转化成SNN模型，这一步骤中可能涉及权重的量化、神经元模型(比如激活函数)的转化等过程，ANN到SNN的转化为本领域所熟知的技术手段，本发明对具体的实施手段不做限制。训练所得的神经网络配置参数最终被部署至神经拟态芯片中，该芯片即获得推理能力而可以投入到实际应用场景之中，比如手势检测、跌倒检测等。换言之，此处公开：一种脉冲神经网络训练方法，该方法包括如下步骤:对事件成像装置所采集的数据集，分别按照固定脉冲事件数量构建成若干幅图像；依据所述若干幅图像训练人工神经网络；将人工神经网络转换成脉冲神经网络；将转换成后的脉冲神经网络部署至神经拟态芯片中。
此外本发明还公开：一种脉冲事件决策方法，该脉冲事件决策方法依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，对所述脉冲神经网络的输入脉冲事件的个数进行计数，并获得第一计数值，在第一计数值达到第一预设值时，依据所述脉冲神经网络的输出层中若干神经元各自发放脉冲事件的计数做出决策结果；或者，对所述脉冲神经网络的输出层部分或全部神经元的输出脉冲事件进行计数，并获得第二计数值，在第二计数值达到第二预设值时，依据所述脉冲神经网络的输出层中神经元各自发放脉冲事件的计数做出决策结果。
在某实施例中，至少存在某相邻两次做出决策结果时，相邻两次获得的第一计数值所对应的输入脉冲事件之间有部分重叠；或/和，对所述输出层的部分或全部神经元的每一个发放的脉冲事件使用多个子计数器进行计数；移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
一种芯片，该芯片上部署有脉冲神经网络，且还包括前述任意一项脉冲事件决策装置，或应用上述脉冲事件决策方法，所述的脉冲事件决策装置或脉冲事件决策方法用于对脉冲神经网络的输出脉冲事件进行决策。在某类实施例中，该芯片为神经拟态芯片。
在某类实施例中，还可以将前述基于时间窗的传统技术方案与本发明基于脉冲计数的技术方案配置成可选决策方案，由用户或其它因素(比如实际应用场景的需要)去决定选择具体使用哪一种决策方案，这增加了芯片在使用时的灵活性。
一种电子设备，该电子设备包括事件成像装置，以及包括如前任意一项所述的脉冲事件决策装置、芯片，或应用上述任一项脉冲事件决策方法；所述的脉冲事件决策装置或芯片或脉冲事件决策方法用于对事件成像装置经过处理后获得的输出脉冲事件进行决策。
尽管已经参考本发明的具体特征和实施例描述了本发明，但是在不脱离本发明的情况下仍可以对其进行各种修改、组合、替换。本发明的保护范围旨在不限于说明书中描述的过程、机器、制造、物质组成、装置、方法和步骤的特定实施例，并且这些方法、模块可能还被实施在相关联、相互依赖、相互配合、前/后级的一个或多个产品、方法当中。
因此，说明书和附图应简单地视为由所附权利要求限定的技术方案的部分实施例的介绍，因而应根据最大合理解释原则对所附权利要求解读，并旨在尽可能涵盖本发明公开范围 内的所有修改、变化、组合或等同物，同时还应避免不合常理的解读方式。
为了实现更好的技术效果或出于某些应用的需求，本领域技术人员可能在本发明的基础之上，对技术方案做出进一步的改进。然而，即便该部分改进/设计具有创造性或/和进步性，只要依赖本发明的技术构思，覆盖了权利要求所限定的技术特征，该技术方案同样应落入本发明的保护范围之内。
所附的权利要求中所提及的若干技术特征可能存在替代的技术特征，或者对某些技术流程的顺序、物质组织顺序可以重组。本领域普通技术人员知晓本发明后，容易想到该些替换手段，或者改变技术流程的顺序、物质组织顺序，然后采用了基本相同的手段，解决基本相同的技术问题，达到基本相同的技术效果，因此即便权利要求中明确限定了上述手段或/和顺序，然而该些修饰、改变、替换，均应依据等同原则而落入权利要求的保护范围。
结合本文中所公开的实施例中描述的各方法步骤或模块，能够以硬件、软件或者二者的结合来实现，为了清楚地说明硬件和软件的可互换性，在上述说明中已经按照功能一般性地描述了各实施例的步骤及组成。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用或设计约束条件。本领域普通技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为在本发明所要求保护的范围之外。

Claims (16)

  1. 一种脉冲事件决策装置，该脉冲事件决策装置依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，其特征在于，该脉冲事件决策装置包括：
    第一计数模块，其被配置为对所述脉冲神经网络的输入脉冲事件的个数进行计数；
    决策模块，其被配置为在第一计数模块的计数达到第一预设值时，依据所述脉冲神经网络的输出层中若干神经元各自发放脉冲事件的计数做出决策结果。
  2. 根据权利要求1所述的脉冲事件决策装置，其特征在于：所述脉冲事件决策装置还包括：
    第二计数模块，其被配置为执行如下操作之一对所述脉冲神经网络的输出层中若干神经元发放脉冲事件的个数进行计数：
    (i)对输出层中每个神经元发放的脉冲进行计数，然后相加获得输出层部分或全部神经元发放脉冲的总计数；
    (ii)对输出层若干神经元发放的脉冲统一直接计数，得到总计数；
    (iii)对输出层中每个神经元发放的脉冲进行计数，但不相加。
  3. 根据权利要求1所述的脉冲事件决策装置，其特征在于：
    决策模块，被配置为在第一计数模块的计数达到第一预设值，或第一计数模块的计数达到第一预设值且第二计数模块的计数达到第二预设值时，触发决策模块做出决策。
  4. 根据权利要求1所述的脉冲事件决策装置，其特征在于：
    至少存在某相邻两次决策时，第一计数模块中对应的两次计数所对应的输入脉冲事件之间有部分重叠；或/和，第二计数模块中对应的两次计数所对应的脉冲神经网络的输出层若干神经元发放的脉冲事件之间有部分重叠。
  5. 根据权利要求2所述的脉冲事件决策装置，其特征在于：
    第二计数模块对所述输出层中若干神经元的每一个发放的脉冲事件使用多个子计数器进行计数；
    移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；
    将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
  6. 根据权利要求1所述的脉冲事件决策装置，其特征在于：
    若连续多个决策结果满足如下条件中的一个或多个，则输出所述连续多个决策结果中出现次数最多的决策结果：
    (i)若所述连续多个决策结果的跳变率或跳变数低于第一阈值；
    (ii)若所述连续多个决策结果中出现次数最多的决策结果所占比率高于第二阈值；
    否则，不输出决策结果或输出指示结果不确定的决策结果。
  7. 根据权利要求1-6任意一项所述的脉冲事件决策装置，其特征在于：
    所述脉冲事件决策装置中的第一计数模块、第二计数模块、决策模块中的一个或多个被实施在神经拟态芯片之内或之外。
  8. 一种脉冲事件决策装置，该脉冲事件决策装置依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，其特征在于，该脉冲事件决策装置包括：
    第二计数模块，其被配置为对所述脉冲神经网络的部分或全部输出脉冲事件进行计数；
    决策模块，其被配置为在第二计数模块的总计数达到第二预设值时，依据所述脉冲神经网络的输出层中神经元各自发放脉冲事件的计数做出决策结果。
  9. 根据权利要求8所述的脉冲事件决策装置，其特征在于：
    第二计数模块对所述输出层中若干神经元的每一个发放的脉冲事件使用多个子计数器进行计数；
    移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；
    将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
  10. 根据权利要求8所述的脉冲事件决策装置，其特征在于：
    若连续多个决策结果满足如下条件中的一个或多个，则输出所述连续多个决策结果中出现次数最多的决策结果：
    (i)若所述连续多个决策结果的跳变率或跳变数低于第一阈值；
    (ii)若所述连续多个决策结果中出现次数最多的决策结果所占比率高于第二阈值；
    否则，不输出决策结果或输出指示结果不确定的决策结果。
  11. 根据权利要求8-10任意一项所述的脉冲事件决策装置，其特征在于：
    所述脉冲事件决策装置中的第二计数模块或/和决策模块被实施在神经拟态芯片之内或之外。
  12. 一种脉冲事件决策方法，该脉冲事件决策方法依据神经拟态芯片中脉冲神经网络的输出脉冲事件进行决策，其特征在于：
    对所述脉冲神经网络的输入脉冲事件的个数进行计数，并获得第一计数值，在第一计数值达到第一预设值时，依据所述脉冲神经网络的输出层中若干神经元各自发放脉冲事件的计数做出决策结果；或者，
    对所述脉冲神经网络的输出层部分或全部神经元的输出脉冲事件进行计数，并获得第二计数值，在第二计数值达到第二预设值时，依据所述脉冲神经网络的输出层中神经元各自发放脉冲事件的计数做出决策结果。
  13. 根据权利要求12所述的脉冲事件决策方法，其特征在于：
    至少存在某相邻两次做出决策结果时，相邻两次获得的第一计数值所对应的输入脉冲事件之间有部分重叠；或/和，
    对所述输出层的部分或全部神经元的每一个发放的脉冲事件使用多个子计数器进行计 数；移除子计数器中最早的计数，并利用归零的子计数器对新发放的脉冲事件计数；将输出层中每个神经元对应的所有子计数器中计数的和，作为做出决策结果所依据的计数。
  14. 一种芯片，其特征在于：该芯片上部署有脉冲神经网络，且还包括权利要求1-11任意一项所述的脉冲事件决策装置，或在芯片中应用权利要求12-13任意一项所述的脉冲事件决策方法；所述的脉冲事件决策装置或脉冲事件决策方法用于对脉冲神经网络的输出脉冲事件进行决策。
  15. 根据权利要求14所述的芯片，其特征在于：所述芯片包括前述权利要求1-11任意一项脉冲事件决策装置，以及基于时间窗的决策装置，并且芯片被配置为允许在所述脉冲事件决策装置和基于时间窗的决策装置之间做出选择。
  16. 一种电子设备，其特征在于：该电子设备包括事件成像装置，以及包括权利要求1-11任意一项所述的脉冲事件决策装置，或权利要求14-15任意一项所述的芯片，或应用权利要求12-13任意一项所述的脉冲事件决策方法；所述的脉冲事件决策装置或芯片或脉冲事件决策方法用于对事件成像装置经过处理后获得的输出脉冲事件进行决策。
PCT/CN2022/072038 2021-12-06 2022-01-14 脉冲事件决策装置、方法、芯片及电子设备 WO2023103149A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US18/270,888 US20240086690A1 (en) 2021-12-06 2022-01-14 Spike event decision-making device, method, chip and electronic device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202111476156.3 2021-12-06
CN202111476156.3A CN113902106B (zh) 2021-12-06 2021-12-06 脉冲事件决策装置、方法、芯片及电子设备

Publications (1)

Publication Number Publication Date
WO2023103149A1 true WO2023103149A1 (zh) 2023-06-15

Family

ID=79195361

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2022/072038 WO2023103149A1 (zh) 2021-12-06 2022-01-14 脉冲事件决策装置、方法、芯片及电子设备

Country Status (3)

Country Link
US (1) US20240086690A1 (zh)
CN (1) CN113902106B (zh)
WO (1) WO2023103149A1 (zh)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113902106B (zh) * 2021-12-06 2022-02-22 成都时识科技有限公司 脉冲事件决策装置、方法、芯片及电子设备
CN114659553A (zh) * 2022-02-28 2022-06-24 联想(北京)有限公司 一种检测方法、装置、设备及存储介质
CN114372019B (zh) * 2022-03-21 2022-07-15 深圳时识科技有限公司 对脉冲事件进行传输的方法、装置及芯片
CN114466153B (zh) * 2022-04-13 2022-09-09 深圳时识科技有限公司 自适应脉冲生成方法、装置、类脑芯片和电子设备
CN116030535B (zh) * 2023-03-24 2023-06-20 深圳时识科技有限公司 手势识别方法及装置、芯片和电子设备

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180181861A1 (en) * 2016-12-28 2018-06-28 Intel Corporation Neuromorphic circuits for storing and generating connectivity information
US20200242452A1 (en) * 2019-01-25 2020-07-30 Northrop Grumman Systems Corporation Superconducting neuromorphic core
WO2021115262A1 (zh) * 2019-12-09 2021-06-17 南京惟心光电系统有限公司 脉冲卷积神经网络算法、集成电路、运算装置及存储介质
CN113269317A (zh) * 2021-04-14 2021-08-17 南京大学 一种脉冲神经网络计算阵列
CN113902106A (zh) * 2021-12-06 2022-01-07 成都时识科技有限公司 脉冲事件决策装置、方法、芯片及电子设备

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102621383B (zh) * 2012-03-23 2014-08-27 重庆大学 一种纳秒级超窄脉冲测量方法及系统
CN106934457B (zh) * 2017-03-08 2019-12-06 杭州领芯电子有限公司 一种可灵活时分复用的脉冲神经元实现架构
US10956811B2 (en) * 2017-07-31 2021-03-23 Intel Corporation Variable epoch spike train filtering
EP3874411A4 (en) * 2018-11-01 2022-08-03 Brainchip, Inc. NEURAL NETWORK WITH IMPROVED SPIKING
CN110210563B (zh) * 2019-06-04 2021-04-30 北京大学 基于Spike cube SNN的图像脉冲数据时空信息学习及识别方法
CN112232440B (zh) * 2020-11-10 2022-11-11 北京理工大学 一种运用特异化神经元团实现脉冲神经网络信息记忆和区分的方法
CN113408714B (zh) * 2021-05-14 2023-04-07 杭州电子科技大学 基于stdp法则的全数字脉冲神经网络硬件系统及方法
CN113449864B (zh) * 2021-07-21 2023-08-25 北京大学 用于图像数据分类的反馈型脉冲神经网络模型训练方法

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180181861A1 (en) * 2016-12-28 2018-06-28 Intel Corporation Neuromorphic circuits for storing and generating connectivity information
US20200242452A1 (en) * 2019-01-25 2020-07-30 Northrop Grumman Systems Corporation Superconducting neuromorphic core
WO2021115262A1 (zh) * 2019-12-09 2021-06-17 南京惟心光电系统有限公司 脉冲卷积神经网络算法、集成电路、运算装置及存储介质
CN113269317A (zh) * 2021-04-14 2021-08-17 南京大学 一种脉冲神经网络计算阵列
CN113902106A (zh) * 2021-12-06 2022-01-07 成都时识科技有限公司 脉冲事件决策装置、方法、芯片及电子设备

Also Published As

Publication number Publication date
US20240086690A1 (en) 2024-03-14
CN113902106A (zh) 2022-01-07
CN113902106B (zh) 2022-02-22

Similar Documents

Publication Publication Date Title
WO2023103149A1 (zh) 脉冲事件决策装置、方法、芯片及电子设备
JP4780921B2 (ja) 並列パルス信号処理装置、及びその制御方法
CN112699956A (zh) 一种基于改进脉冲神经网络的神经形态视觉目标分类方法
Yousefzadeh et al. On practical issues for stochastic STDP hardware with 1-bit synaptic weights
US10848709B2 (en) Artificial intelligence based image data processing method and image processing device
US9460385B2 (en) Apparatus and methods for rate-modulated plasticity in a neuron network
Zhuang et al. Multichannel pulse-coupled-neural-network-based color image segmentation for object detection
Camunas-Mesa et al. A 32$\,\times\, $32 Pixel Convolution Processor Chip for Address Event Vision Sensors With 155 ns Event Latency and 20 Meps Throughput
US20230334632A1 (en) Image recognition method and device, and computer-readable storage medium
US11443514B2 (en) Recognizing minutes-long activities in videos
US20200372254A1 (en) Method for outputting a signal from an event-based sensor, and event-based sensor using such method
CN113287122A (zh) 脉冲神经网络
US20230087292A1 (en) Data annotation method and apparatus, and fine-grained recognition method and apparatus
CN113239801B (zh) 基于多尺度特征学习和多级域对齐的跨域动作识别方法
CN112037269A (zh) 一种基于多域协同特征表达的视觉运动目标跟踪方法
US11689824B2 (en) Event filtering method for operating an event-based image sensor
CN112130118A (zh) 基于snn的超宽带雷达信号处理系统及处理方法
CN112487874A (zh) 一种基于事件相机消除背景噪声的方法及系统
Marchisio et al. R-snn: An analysis and design methodology for robustifying spiking neural networks against adversarial attacks through noise filters for dynamic vision sensors
WO2024067042A1 (zh) 确定屏幕漏光的跌落深度的方法、装置及电子设备
US20240013521A1 (en) Sequence processing for a dataset with frame dropping
CN110121055A (zh) 用于对象识别的方法和设备
Shuqair et al. Incremental learning in time-series data using reinforcement learning
US20220351496A1 (en) Image content classification
CN107330915B (zh) 用于aer图像传感器的目标追踪方法

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 18270888

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22902594

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE