WO2015112261A1 - Configuring neural network for low spiking rate - Google Patents

Configuring neural network for low spiking rate Download PDF

Info

Publication number
WO2015112261A1
WO2015112261A1 PCT/US2014/068417 US2014068417W WO2015112261A1 WO 2015112261 A1 WO2015112261 A1 WO 2015112261A1 US 2014068417 W US2014068417 W US 2014068417W WO 2015112261 A1 WO2015112261 A1 WO 2015112261A1
Authority
WO
WIPO (PCT)
Prior art keywords
neuron model
neuron
model
firing rate
neural network
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/US2014/068417
Other languages
English (en)
French (fr)
Inventor
Sachin Subhash Talathi
David Jonathan Julian
Venkata Sreekanta Reddy Annapureddy
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Qualcomm Inc
Original Assignee
Qualcomm Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Qualcomm Inc filed Critical Qualcomm Inc
Priority to EP14819207.3A priority Critical patent/EP3097516A1/en
Priority to JP2016547861A priority patent/JP2017509953A/ja
Priority to CN201480071680.4A priority patent/CN105900115A/zh
Publication of WO2015112261A1 publication Critical patent/WO2015112261A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0499—Feedforward networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning

Definitions

  • FIGURE 3 illustrates an exemplary diagram 300 of a synaptic weight change as a function of relative timing of presynaptic and postsynaptic spikes in accordance with the STDP.
  • a presynaptic neuron fires before a postsynaptic neuron, then a corresponding synaptic weight may be increased, as illustrated in a portion 302 of the graph 300.
  • This weight increase can be referred to as an LTP of the synapse.
  • the reverse order of firing may reduce the synaptic weight, as illustrated in a portion 304 of the graph 300, causing an LTD of the synapse.
  • the regime and the coupling p may be computed upon events.
  • the regime and coupling (transformation) variables may be defined based on the state at the time of the last (prior) event.
  • the regime and coupling variable may be defined based on the state at the time of the next (current) event.
  • the neurons 912 of the hidden layer 904 may, for example, be configured in accordance with a neuron model such as a Leaky Integrate and Fire (LIF) neuron model, a quadratic integrate and Fire (QIF) model, a Linear Threshold (LT) neuron model, and a Cold neuron Model, or the like.
  • a neuron model such as a Leaky Integrate and Fire (LIF) neuron model, a quadratic integrate and Fire (QIF) model, a Linear Threshold (LT) neuron model, and a Cold neuron Model, or the like.
  • the regularization term ( ⁇ ) of Equation 24 may be selected to reduce or even minimize the number of effective neurons for a given value of expected desired performance. That is, the regularization term ( ⁇ ) may be selected for example, based on an error rate, a number of neurons, other system and/or performance consideration or metrics. For example, for Method 1, a performance target (e.g., in terms of MSE (mean square error)) may be set and used to find the reduced number of effective neurons to meet the desired level of performance target by varying ⁇ . The error tolerance can be user selected, and then the regularization term can be set accordingly.
  • MSE mean square error
  • LASSO processing may be performed to estimate or learn the postsynaptic weights represented by the decoding vectors (e.g., 1008). Applying the LI regularization, some of the estimated or learned decoding vectors weights may be set to zero, while the remainder may be set to a non-zero value.
  • the decoding vectors having a zero weight may be removed thereby generating a reduced set or more sparse set of decoding vectors, which may be referred to as a sparse-set M ⁇ N non-zero decoding vectors 1008a, ..., 1008m (which may be collectively referred to as non-zero decoding vectors 1008).
  • FIGURE 10B illustrates a reduced set of model neurons generated according to Method 2 described above.
  • FIGURE 10B shows a modified hidden layer 1006 of a neural network including M effective model neurons generated according to Method 1 (and shown in FIGURE 10A) and a new set of N-M model neurons 1010.
  • the weights of the presynaptic connections for the M effective model neurons (1006) may be fixed, while the weights of the presynaptic connections for new set of N- M model neurons may be random.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Image Analysis (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Feedback Control In General (AREA)
PCT/US2014/068417 2014-01-23 2014-12-03 Configuring neural network for low spiking rate Ceased WO2015112261A1 (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
EP14819207.3A EP3097516A1 (en) 2014-01-23 2014-12-03 Configuring neural network for low spiking rate
JP2016547861A JP2017509953A (ja) 2014-01-23 2014-12-03 低スパイキングレートのためのニューラルネットワークを構成すること
CN201480071680.4A CN105900115A (zh) 2014-01-23 2014-12-03 配置用于实现低尖峰发放率的神经网络

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US201461930849P 2014-01-23 2014-01-23
US201461930858P 2014-01-23 2014-01-23
US61/930,849 2014-01-23
US61/930,858 2014-01-23
US201461939537P 2014-02-13 2014-02-13
US61/939,537 2014-02-13
US14/449,092 US20150206050A1 (en) 2014-01-23 2014-07-31 Configuring neural network for low spiking rate
US14/449,092 2014-07-31

Publications (1)

Publication Number Publication Date
WO2015112261A1 true WO2015112261A1 (en) 2015-07-30

Family

ID=53545086

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/US2014/068417 Ceased WO2015112261A1 (en) 2014-01-23 2014-12-03 Configuring neural network for low spiking rate
PCT/US2014/068449 Ceased WO2015112262A1 (en) 2014-01-23 2014-12-03 Configuring sparse neuronal networks

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/US2014/068449 Ceased WO2015112262A1 (en) 2014-01-23 2014-12-03 Configuring sparse neuronal networks

Country Status (6)

Country Link
US (2) US10339447B2 (enExample)
EP (2) EP3097519A1 (enExample)
JP (2) JP2017509953A (enExample)
CN (2) CN105874477A (enExample)
TW (2) TW201531966A (enExample)
WO (2) WO2015112261A1 (enExample)

Families Citing this family (47)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016187500A1 (en) * 2015-05-21 2016-11-24 Cory Merkel Method and apparatus for training memristive learning systems
CN107545303B (zh) * 2016-01-20 2021-09-07 中科寒武纪科技股份有限公司 用于稀疏人工神经网络的计算装置和运算方法
JP6453785B2 (ja) * 2016-01-21 2019-01-16 日本電信電話株式会社 回帰分析装置、回帰分析方法および回帰分析プログラム
WO2017136665A1 (en) * 2016-02-05 2017-08-10 Google Inc. Augmenting neural networks with sparsely-accessed external memory
US10643125B2 (en) 2016-03-03 2020-05-05 International Business Machines Corporation Methods and systems of neuron leaky integrate and fire circuits
US11222263B2 (en) 2016-07-28 2022-01-11 Samsung Electronics Co., Ltd. Neural network method and apparatus
CN107239825B (zh) * 2016-08-22 2021-04-09 赛灵思电子科技(北京)有限公司 考虑负载均衡的深度神经网络压缩方法
EP3520038A4 (en) 2016-09-28 2020-06-03 D5A1 Llc LEARNING TRAINER FOR MACHINE LEARNING SYSTEM
CN110298443B (zh) * 2016-09-29 2021-09-17 中科寒武纪科技股份有限公司 神经网络运算装置及方法
US20180164866A1 (en) * 2016-12-13 2018-06-14 Qualcomm Incorporated Low-power architecture for sparse neural network
US10679118B2 (en) 2016-12-20 2020-06-09 Intel Corporation Solving matrix inverse problems using neuromorphic computing
US10871964B2 (en) * 2016-12-29 2020-12-22 Qualcomm Incorporated Architecture for sparse neural network acceleration
US11551074B2 (en) * 2017-01-20 2023-01-10 Tsinghua University Self-adaptive threshold neuron information processing method, self-adaptive leakage value neuron information processing method, system computer device and readable storage medium
KR102390379B1 (ko) * 2017-03-06 2022-04-26 삼성전자주식회사 뉴럴 네트워크 프로세서, 뉴럴 네트워크 프로세서의 동작 방법, 및 뉴럴 네트워크 장치
WO2018175098A1 (en) 2017-03-24 2018-09-27 D5Ai Llc Learning coach for machine learning system
CN110914829B (zh) * 2017-04-07 2025-08-12 英特尔公司 使用改进的卷积神经网络用于图像处理的方法和系统
US10795836B2 (en) * 2017-04-17 2020-10-06 Microsoft Technology Licensing, Llc Data processing performance enhancement for neural networks using a virtualized data iterator
CN108734279B (zh) * 2017-04-20 2021-04-23 上海寒武纪信息科技有限公司 一种运算装置和方法
EP3699826A1 (en) 2017-04-20 2020-08-26 Shanghai Cambricon Information Technology Co., Ltd Operation device and related products
CN109146069B (zh) * 2017-06-16 2020-10-13 上海寒武纪信息科技有限公司 运算装置、运算方法和芯片
WO2018231708A2 (en) * 2017-06-12 2018-12-20 D5Ai Llc Robust anti-adversarial machine learning
KR102601604B1 (ko) 2017-08-04 2023-11-13 삼성전자주식회사 뉴럴 네트워크의 파라미터들을 양자화하는 방법 및 장치
CN107609641B (zh) * 2017-08-30 2020-07-03 清华大学 稀疏神经网络架构及其实现方法
TWI653584B (zh) 2017-09-15 2019-03-11 中原大學 利用非揮發性記憶體完成類神經網路訓練的方法
US10366322B2 (en) * 2017-10-06 2019-07-30 DeepCube LTD. System and method for compact and efficient sparse neural networks
US11580194B2 (en) 2017-11-01 2023-02-14 Nec Corporation Information processing apparatus, information processing method, and program
CN109902812B (zh) 2017-12-11 2020-10-09 中科寒武纪科技股份有限公司 板卡和神经网络运算方法
US10803379B2 (en) 2017-12-12 2020-10-13 Amazon Technologies, Inc. Multi-memory on-chip computational network
US10846621B2 (en) * 2017-12-12 2020-11-24 Amazon Technologies, Inc. Fast context switching for computational networks
CN108268938B (zh) * 2018-01-24 2020-04-21 清华大学 神经网络及其信息处理方法、信息处理系统
US11321612B2 (en) 2018-01-30 2022-05-03 D5Ai Llc Self-organizing partially ordered networks and soft-tying learned parameters, such as connection weights
CN110197272B (zh) * 2018-02-27 2020-08-25 上海寒武纪信息科技有限公司 集成电路芯片装置及相关产品
EP3576019B1 (en) * 2018-05-29 2024-10-09 Nokia Technologies Oy Artificial neural networks
KR102675781B1 (ko) * 2018-06-27 2024-06-18 삼성전자주식회사 뉴로모픽 프로세서 및 그것의 동작 방법
US20200019838A1 (en) * 2018-07-11 2020-01-16 The Board Of Trustees Of The Leland Stanford Junior University Methods and apparatus for spiking neural network computing based on randomized spatial assignments
US11537856B2 (en) * 2018-08-08 2022-12-27 Applied Brain Research Inc. Digital circuits for evaluating neural engineering framework style neural networks
US11537855B2 (en) * 2018-09-24 2022-12-27 International Business Machines Corporation Low spike count ring buffer mechanism on neuromorphic hardware
US11922314B1 (en) 2018-11-30 2024-03-05 Ansys, Inc. Systems and methods for building dynamic reduced order physical models
US11556778B2 (en) * 2018-12-07 2023-01-17 Microsoft Technology Licensing, Llc Automated generation of machine learning models
CN110619905A (zh) * 2019-08-09 2019-12-27 上海集成电路研发中心有限公司 一种基于rram忆阻器单元的集合模块及其形成方法
KR20210023401A (ko) * 2019-08-23 2021-03-04 삼성전자주식회사 뉴럴 네트워크 연산 방법 및 이를 포함하는 시스템
CN110708341B (zh) * 2019-11-15 2020-09-29 中国科学院信息工程研究所 一种基于远程桌面加密网络流量模式差异的用户行为检测方法及系统
US11556384B2 (en) 2020-03-13 2023-01-17 Cisco Technology, Inc. Dynamic allocation and re-allocation of learning model computing resources
EP4107671A1 (en) * 2020-06-05 2022-12-28 DeepMind Technologies Limited System and method for training a sparse neural network whilst maintaining sparsity
DE102020209538A1 (de) * 2020-07-29 2022-02-03 Robert Bosch Gesellschaft mit beschränkter Haftung Vorrichtung und Verfahren zum Ermitteln einer physikalischen Eigenschaft eines physikalischen Objekts
CN114511062A (zh) * 2021-11-22 2022-05-17 北京中科凡语科技有限公司 可解释神经解码模型的生成方法、电子设备和存储介质
CN116842993B (zh) * 2023-06-15 2025-07-25 北京工业大学 一种基于多属性二型模糊神经网络的总氮去除量检测方法

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030105409A1 (en) 2001-11-14 2003-06-05 Donoghue John Philip Neurological signal decoding
MY138544A (en) 2003-06-26 2009-06-30 Neuramatix Sdn Bhd Neural networks with learning and expression capability
JP4848492B2 (ja) 2006-08-03 2011-12-28 パナソニック電工Ｓｕｎｘ株式会社 信号識別方法及び信号識別装置
US8095210B2 (en) 2007-01-19 2012-01-10 California Institute Of Technology Prosthetic devices and methods and systems related thereto
EP2263165A4 (en) 2008-03-14 2011-08-24 Hewlett Packard Development Co NEUROMORPHIC CIRCUIT
US8630966B2 (en) 2009-01-27 2014-01-14 Salk Institute For Biological Studies Temporally dynamic artificial neural networks
US8615476B2 (en) 2009-04-15 2013-12-24 University Of Southern California Protecting military perimeters from approaching human and vehicle using biologically realistic neural network
US8250010B2 (en) 2009-05-21 2012-08-21 International Business Machines Corporation Electronic learning synapse with spike-timing dependent plasticity using unipolar memory-switching elements
US10223632B2 (en) 2009-07-27 2019-03-05 International Business Machines Corporation Modeling states of an entity
US20120117012A1 (en) 2010-04-08 2012-05-10 Neurosciences Research Foundation, Inc. Spike-timing computer modeling of working memory
US20120023051A1 (en) * 2010-07-22 2012-01-26 Ramin Pishehvar Signal coding with adaptive neural network
US8606732B2 (en) 2010-08-04 2013-12-10 Qualcomm Incorporated Methods and systems for reward-modulated spike-timing-dependent-plasticity
US8577820B2 (en) 2011-03-04 2013-11-05 Tokyo Electron Limited Accurate and fast neural network training for library-based critical dimension (CD) metrology
US8521670B2 (en) 2011-05-25 2013-08-27 HGST Netherlands B.V. Artificial neural network application for magnetic core width prediction and modeling for magnetic disk drive manufacture
US9092726B2 (en) 2011-08-11 2015-07-28 Greenray Industries, Inc. Neural network frequency control
KR101912165B1 (ko) 2011-12-09 2018-10-29 삼성전자주식회사 스파이킹 뉴런 기반 작업 기억 장치
US9111225B2 (en) 2012-02-08 2015-08-18 Qualcomm Incorporated Methods and apparatus for spiking neural computation
US9208431B2 (en) 2012-05-10 2015-12-08 Qualcomm Incorporated Method and apparatus for strategic synaptic failure and learning in spiking neural networks
US20130325767A1 (en) 2012-05-30 2013-12-05 Qualcomm Incorporated Dynamical event neuron and synapse models for learning spiking neural networks
US9367798B2 (en) 2012-09-20 2016-06-14 Brain Corporation Spiking neuron network adaptive control apparatus and methods
US9367519B2 (en) * 2013-08-30 2016-06-14 Microsoft Technology Licensing, Llc Sparse matrix data structure

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
J. C. TAPSON ET AL: "Synthesis of neural networks for spatio-temporal spike pattern recognition and processing", FRONTIERS IN NEUROSCIENCE, vol. 7, 153, 30 August 2013 (2013-08-30), XP055192522, DOI: 10.3389/fnins.2013.00153 *
K. D. CARLSON ET AL: "An efficient automated parameter tuning framework for spiking neural networks", FRONTIERS IN NEUROSCIENCE, vol. 8, 10, 17 January 2014 (2014-01-17), XP055192436, DOI: 10.3389/fnins.2014.00010 *
M. A. FARRIES, A. L. FAIRHALL: "Reinforcement learning with modulated spike timing-dependent synaptic plasticity", JOURNAL OF NEUROPHYSIOLOGY, vol. 98, no. 6, 10 October 2007 (2007-10-10), pages 3648 - 3665, XP055192453, DOI: 10.1152/jn.00364.2007 *
T. STEWART ET AL: "Python scripting in the Nengo simulator", FRONTIERS IN NEUROINFORMATICS, vol. 3, 7, 10 October 2008 (2008-10-10), XP055192431, DOI: 10.3389/neuro.11.007.2009 *

Also Published As

Publication number Publication date
EP3097516A1 (en) 2016-11-30
TW201531965A (zh) 2015-08-16
CN105900115A (zh) 2016-08-24
JP2017509953A (ja) 2017-04-06
US20150206048A1 (en) 2015-07-23
TW201531966A (zh) 2015-08-16
US20150206050A1 (en) 2015-07-23
WO2015112262A1 (en) 2015-07-30
US10339447B2 (en) 2019-07-02
CN105874477A (zh) 2016-08-17
JP2017509951A (ja) 2017-04-06
EP3097519A1 (en) 2016-11-30

Similar Documents

Publication Publication Date Title
US10339447B2 (en) Configuring sparse neuronal networks
US9542643B2 (en) Efficient hardware implementation of spiking networks
EP3170126A1 (en) Decomposing convolution operation in neural networks
EP3123404A2 (en) Differential encoding in neural networks
WO2015088774A2 (en) Neuronal diversity in spiking neural networks and pattern classification
EP3129921A2 (en) Modulating plasticity by global scalar values in a spiking neural network
US20150278685A1 (en) Probabilistic representation of large sequences using spiking neural network
WO2015167765A2 (en) Temporal spike encoding for temporal learning
WO2015127130A2 (en) Dynamic spatial target selection
US20140310216A1 (en) Method for generating compact representations of spike timing-dependent plasticity curves
US20150213356A1 (en) Method for converting values into spikes
EP3058518A2 (en) Congestion avoidance in networks of spiking neurons
US9542645B2 (en) Plastic synapse management
US9342782B2 (en) Stochastic delay plasticity
WO2015057305A1 (en) Dynamically assigning and examining synaptic delay
WO2014197175A2 (en) Efficient implementation of neural population diversity in neural system
US20150242742A1 (en) Imbalanced cross-inhibitory mechanism for spatial target selection

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14819207

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
REEP Request for entry into the european phase

Ref document number: 2014819207

Country of ref document: EP

WWE Wipo information: entry into national phase

Ref document number: 2014819207

Country of ref document: EP

ENP Entry into the national phase

Ref document number: 2016547861

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE