JP6986569B2 - ニューラル・ネットワークの更新管理のためのコンピュータ実装方法、コンピュータ・プログラム、およびコンピュータ処理システム - Google Patents
ニューラル・ネットワークの更新管理のためのコンピュータ実装方法、コンピュータ・プログラム、およびコンピュータ処理システム Download PDFInfo
- Publication number
- JP6986569B2 JP6986569B2 JP2019554891A JP2019554891A JP6986569B2 JP 6986569 B2 JP6986569 B2 JP 6986569B2 JP 2019554891 A JP2019554891 A JP 2019554891A JP 2019554891 A JP2019554891 A JP 2019554891A JP 6986569 B2 JP6986569 B2 JP 6986569B2
- Authority
- JP
- Japan
- Prior art keywords
- computer
- stochastic
- implementation method
- deterministic
- scaling
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F7/00—Methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F7/38—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation
- G06F7/48—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices
- G06F7/544—Methods or arrangements for performing computations using exclusively denominational number representation, e.g. using binary, ternary, decimal representation using non-contact-making devices, e.g. tube, solid state device; using unspecified devices for evaluating functions by calculation
- G06F7/5443—Sum of products
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
- G06F17/10—Complex mathematical operations
- G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F5/00—Methods or arrangements for data conversion without changing the order or content of the data handled
- G06F5/01—Methods or arrangements for data conversion without changing the order or content of the data handled for shifting, e.g. justifying, scaling, normalising
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
- G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
- G06N3/065—Analogue means
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2207/00—Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F2207/38—Indexing scheme relating to groups G06F7/38 - G06F7/575
- G06F2207/48—Indexing scheme relating to groups G06F7/48 - G06F7/575
- G06F2207/4802—Special implementations
- G06F2207/4814—Non-logic devices, e.g. operational amplifiers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2207/00—Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F2207/38—Indexing scheme relating to groups G06F7/38 - G06F7/575
- G06F2207/48—Indexing scheme relating to groups G06F7/48 - G06F7/575
- G06F2207/4802—Special implementations
- G06F2207/4818—Threshold devices
- G06F2207/4824—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2207/00—Indexing scheme relating to methods or arrangements for processing data by operating upon the order or content of the data handled
- G06F2207/38—Indexing scheme relating to groups G06F7/38 - G06F7/575
- G06F2207/48—Indexing scheme relating to groups G06F7/48 - G06F7/575
- G06F2207/4802—Special implementations
- G06F2207/4828—Negative resistance devices, e.g. tunnel diodes, gunn effect devices
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Mathematical Physics (AREA)
- Computing Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Software Systems (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- General Health & Medical Sciences (AREA)
- Evolutionary Computation (AREA)
- Computational Linguistics (AREA)
- Mathematical Analysis (AREA)
- Computational Mathematics (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Neurology (AREA)
- Algebra (AREA)
- Databases & Information Systems (AREA)
- Complex Calculations (AREA)
- Machine Translation (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Description
wij=wij+η(xi×δj)
ここで、wijはi番目の行とj番目の列との接続の重みを表し、ηは学習率(スカラ値)、xiは入力ニューロンにおける活動、δjは出力ニューロンによって計算される誤差を表す。
η←BLΔwminC2
η←BLΔwminC2
Pi=Cxxi
Pj=Cδδj
Claims (17)
- ニューラル・ネットワークの更新管理のためのコンピュータ実装方法であって、
抵抗型処理ユニット(RPU)を使用して前記ニューラル・ネットワークに対して、乗算の被乗数と乗数とを使用する等方的更新プロセスを実行するステップを含み、
前記実行するステップは、前記被乗数と前記乗数とを同じ桁を有するようにスケーリングするステップを含み、
1つまたは複数の確率的変換器によって、前記ニューラル・ネットワークのニューロンに対応する数値の組を確率的ビット・ストリームに変換することを含み、
スケーリング係数が、前記1つまたは複数の確率的変換器の増幅係数に適用されることを含む、
コンピュータ実装方法。 - 前記被乗数および前記乗数が、前記スケーリングの前と後とで同じ積を維持するようにスケーリングされる、請求項1に記載のコンピュータ実装方法。
- 前記スケーリングするステップは、前記等方的更新プロセスの入力に適用される入力調整プロセスにおいて実行される、請求項1に記載のコンピュータ実装方法。
- 前記入力調整プロセスは、前記被乗数と前記乗数との桁の相違の結果として生じる偽空間相関を解消する、請求項3に記載のコンピュータ実装方法。
- 前記等方的更新プロセスは、単一の更新サイクルのみを使用して実行される、請求項1に記載のコンピュータ実装方法。
- RPUアレイがアナログ・ベクトル行列乗算を実行するように構成された、請求項1に記載のコンピュータ実装方法。
- 前記等方的更新プロセスが、1つまたは複数の所定の基準を満たすビット長を有する前記確率的ビット・ストリームのそれぞれに対して実行される、請求項1に記載のコンピュータ実装方法。
- 前記1つまたは複数の所定の基準が最小ビット・ストリーム長を含む、請求項7に記載のコンピュータ実装方法。
- 前記1つまたは複数の確率的変換器が第1の確率的変換器と第2の確率的変換器とを含み、前記スケーリングするステップが、前記第1の確率的変換器の増幅係数に前記スケーリング係数を乗じることと、前記第2の確率的変換器の増幅係数を前記スケーリング係数で割ることとを含む、請求項1に記載のコンピュータ実装方法。
- 1つまたは複数の決定論的変換器によって、前記ニューラル・ネットワークのニューロンに対応する数値の組を決定論的ビット・ストリームに変換することをさらに含む、請求項1に記載のコンピュータ実装方法。
- 前記等方的更新プロセスが、1つまたは複数の所定の基準を満たすビット長を有する前記決定論的ビット・ストリームのそれぞれに対して実行される、請求項10に記載のコンピュータ実装方法。
- 前記1つまたは複数の所定の基準が最小ビット・ストリーム長を含む、請求項11に記載のコンピュータ実装方法。
- 前記スケーリング係数が、前記1つまたは複数の決定論的変換器の増幅係数に適用される、請求項1に記載のコンピュータ実装方法。
- 前記1つまたは複数の決定論的変換器が第1の決定論的変換器と第2の決定論的変換器とを含み、前記スケーリングするステップが、前記第1の決定論的変換器の増幅係数に前記スケーリング係数を乗じることと、前記第2の決定論的変換器の増幅係数を前記スケーリング係数で割ることとを含む、請求項10に記載のコンピュータ実装方法。
- 請求項1〜14の何れか1項に記載の方法の各ステップをコンピュータ・ハードウェアによる手段として構成した、コンピュータ処理システム。
- 請求項1〜14の何れか1項に記載の方法の各ステップをコンピュータに実行させる、コンピュータ・プログラム。
- 請求項16に記載の前記コンピュータ・プログラムをコンピュータ可読記録媒体に記録した、コンピュータ可読記録媒体。
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US15/487,701 | 2017-04-14 | ||
US15/487,701 US10783432B2 (en) | 2017-04-14 | 2017-04-14 | Update management for RPU array |
PCT/IB2018/051644 WO2018189600A1 (en) | 2017-04-14 | 2018-03-13 | Update management for rpu array |
Publications (3)
Publication Number | Publication Date |
---|---|
JP2020517002A JP2020517002A (ja) | 2020-06-11 |
JP2020517002A5 JP2020517002A5 (ja) | 2020-07-27 |
JP6986569B2 true JP6986569B2 (ja) | 2021-12-22 |
Family
ID=63790739
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
JP2019554891A Active JP6986569B2 (ja) | 2017-04-14 | 2018-03-13 | ニューラル・ネットワークの更新管理のためのコンピュータ実装方法、コンピュータ・プログラム、およびコンピュータ処理システム |
Country Status (6)
Country | Link |
---|---|
US (2) | US10783432B2 (ja) |
JP (1) | JP6986569B2 (ja) |
CN (1) | CN110506282B (ja) |
DE (1) | DE112018000723T5 (ja) |
GB (1) | GB2576275A (ja) |
WO (1) | WO2018189600A1 (ja) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10303998B2 (en) * | 2017-09-28 | 2019-05-28 | International Business Machines Corporation | Floating gate for neural network inference |
US11651231B2 (en) | 2019-03-01 | 2023-05-16 | Government Of The United States Of America, As Represented By The Secretary Of Commerce | Quasi-systolic processor and quasi-systolic array |
US11562249B2 (en) * | 2019-05-01 | 2023-01-24 | International Business Machines Corporation | DNN training with asymmetric RPU devices |
CN110750231B (zh) * | 2019-09-27 | 2021-09-28 | 东南大学 | 一种面向卷积神经网络的双相系数可调模拟乘法计算电路 |
US11501148B2 (en) * | 2020-03-04 | 2022-11-15 | International Business Machines Corporation | Area and power efficient implementations of modified backpropagation algorithm for asymmetric RPU devices |
US11501023B2 (en) | 2020-04-30 | 2022-11-15 | International Business Machines Corporation | Secure chip identification using resistive processing unit as a physically unclonable function |
US11366876B2 (en) | 2020-06-24 | 2022-06-21 | International Business Machines Corporation | Eigenvalue decomposition with stochastic optimization |
US11568217B2 (en) * | 2020-07-15 | 2023-01-31 | International Business Machines Corporation | Sparse modifiable bit length deterministic pulse generation for updating analog crossbar arrays |
US11443171B2 (en) * | 2020-07-15 | 2022-09-13 | International Business Machines Corporation | Pulse generation for updating crossbar arrays |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5258934A (en) | 1990-05-14 | 1993-11-02 | California Institute Of Technology | Charge domain bit serial vector-matrix multiplier and method thereof |
JPH04153827A (ja) * | 1990-10-18 | 1992-05-27 | Fujitsu Ltd | ディジタル乗算器 |
US20040083193A1 (en) | 2002-10-29 | 2004-04-29 | Bingxue Shi | Expandable on-chip back propagation learning neural network with 4-neuron 16-synapse |
EP1508872A1 (en) * | 2003-08-22 | 2005-02-23 | Semeion | An algorithm for recognising relationships between data of a database and a method for image pattern recognition based on the said algorithm |
US9715655B2 (en) | 2013-12-18 | 2017-07-25 | The United States Of America As Represented By The Secretary Of The Air Force | Method and apparatus for performing close-loop programming of resistive memory devices in crossbar array based hardware circuits and systems |
US9466362B2 (en) | 2014-08-12 | 2016-10-11 | Arizona Board Of Regents On Behalf Of Arizona State University | Resistive cross-point architecture for robust data representation with arbitrary precision |
US20170061279A1 (en) * | 2015-01-14 | 2017-03-02 | Intel Corporation | Updating an artificial neural network using flexible fixed point representation |
US10748064B2 (en) | 2015-08-27 | 2020-08-18 | International Business Machines Corporation | Deep neural network training with native devices |
US10325006B2 (en) | 2015-09-29 | 2019-06-18 | International Business Machines Corporation | Scalable architecture for analog matrix operations with resistive devices |
US10387778B2 (en) | 2015-09-29 | 2019-08-20 | International Business Machines Corporation | Scalable architecture for implementing maximization algorithms with resistive devices |
CN105488565A (zh) * | 2015-11-17 | 2016-04-13 | 中国科学院计算技术研究所 | 加速深度神经网络算法的加速芯片的运算装置及方法 |
-
2017
- 2017-04-14 US US15/487,701 patent/US10783432B2/en active Active
- 2017-12-14 US US15/842,724 patent/US11062208B2/en active Active
-
2018
- 2018-03-13 JP JP2019554891A patent/JP6986569B2/ja active Active
- 2018-03-13 GB GB1916146.2A patent/GB2576275A/en not_active Withdrawn
- 2018-03-13 CN CN201880023799.2A patent/CN110506282B/zh active Active
- 2018-03-13 WO PCT/IB2018/051644 patent/WO2018189600A1/en active Application Filing
- 2018-03-13 DE DE112018000723.5T patent/DE112018000723T5/de active Pending
Also Published As
Publication number | Publication date |
---|---|
GB201916146D0 (en) | 2019-12-18 |
JP2020517002A (ja) | 2020-06-11 |
US20180300627A1 (en) | 2018-10-18 |
US20180300622A1 (en) | 2018-10-18 |
CN110506282A (zh) | 2019-11-26 |
WO2018189600A1 (en) | 2018-10-18 |
US10783432B2 (en) | 2020-09-22 |
US11062208B2 (en) | 2021-07-13 |
CN110506282B (zh) | 2023-04-28 |
DE112018000723T5 (de) | 2019-10-24 |
GB2576275A (en) | 2020-02-12 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP6986569B2 (ja) | ニューラル・ネットワークの更新管理のためのコンピュータ実装方法、コンピュータ・プログラム、およびコンピュータ処理システム | |
US10733390B2 (en) | Processing text sequences using neural networks | |
JP6676165B2 (ja) | 畳み込みゲート制御再帰型ニューラルネットワーク | |
US10664745B2 (en) | Resistive processing units and neural network training methods | |
US11373092B2 (en) | Training of artificial neural networks | |
KR102672586B1 (ko) | 인공신경망의 훈련 방법 및 장치 | |
JP2019511033A5 (ja) | ||
US10387778B2 (en) | Scalable architecture for implementing maximization algorithms with resistive devices | |
KR20180072562A (ko) | 인공 뉴럴 네트워크 클래스-기반 프루닝 | |
US20170091620A1 (en) | Scalable architecture for analog matrix operations with resistive devices | |
US11257592B2 (en) | Architecture for machine learning model to leverage hierarchical semantics between medical concepts in dictionaries | |
JP2018537788A (ja) | 外部メモリを用いたニューラルネットワークの拡張 | |
US11263516B2 (en) | Neural network based acoustic models for speech recognition by grouping context-dependent targets | |
US20240005166A1 (en) | Minimum Deep Learning with Gating Multiplier | |
US20180341856A1 (en) | Balancing memory consumption of multiple graphics processing units in deep learning | |
US20180060730A1 (en) | Leveraging class information to initialize a neural network langauge model | |
CN116882469B (zh) | 用于情感识别的脉冲神经网络部署方法、装置及设备 | |
CN113841164B (zh) | Rpu阵列的噪声和信号管理 | |
US11704542B2 (en) | Convolutional dynamic Boltzmann Machine for temporal event sequence | |
WO2020261509A1 (ja) | 機械学習装置、機械学習プログラム、及び機械学習方法 | |
WO2020262316A1 (ja) | データ分析システム、データ分析方法及びプログラム | |
JP7332193B2 (ja) | 因子グラフ上での確率伝搬 | |
US20210133556A1 (en) | Feature-separated neural network processing of tabular data | |
US20200151569A1 (en) | Warping sequence data for learning in neural networks | |
US11836613B2 (en) | Neural programmer interpreters with modeled primitives |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20200525 |
|
A621 | Written request for application examination |
Free format text: JAPANESE INTERMEDIATE CODE: A621 Effective date: 20200826 |
|
A977 | Report on retrieval |
Free format text: JAPANESE INTERMEDIATE CODE: A971007 Effective date: 20210811 |
|
A131 | Notification of reasons for refusal |
Free format text: JAPANESE INTERMEDIATE CODE: A131 Effective date: 20210812 |
|
A521 | Request for written amendment filed |
Free format text: JAPANESE INTERMEDIATE CODE: A523 Effective date: 20211104 |
|
TRDD | Decision of grant or rejection written | ||
A01 | Written decision to grant a patent or to grant a registration (utility model) |
Free format text: JAPANESE INTERMEDIATE CODE: A01 Effective date: 20211124 |
|
A61 | First payment of annual fees (during grant procedure) |
Free format text: JAPANESE INTERMEDIATE CODE: A61 Effective date: 20211129 |
|
R150 | Certificate of patent or registration of utility model |
Ref document number: 6986569 Country of ref document: JP Free format text: JAPANESE INTERMEDIATE CODE: R150 |