JP7087079B2 - 深層学習アプリケーションのための堅牢な勾配重み圧縮方式 - Google Patents

深層学習アプリケーションのための堅牢な勾配重み圧縮方式 Download PDF

Info

Publication number
JP7087079B2
JP7087079B2 JP2020529245A JP2020529245A JP7087079B2 JP 7087079 B2 JP7087079 B2 JP 7087079B2 JP 2020529245 A JP2020529245 A JP 2020529245A JP 2020529245 A JP2020529245 A JP 2020529245A JP 7087079 B2 JP7087079 B2 JP 7087079B2
Authority
JP
Japan
Prior art keywords
gradient
residual
vector
current
weights
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2020529245A
Other languages
English (en)
Japanese (ja)
Other versions
JP2021505993A (ja
JP2021505993A5 (enExample
Inventor
チェン、チア－ユ
アグラワル、アンカー
ブラント、ダニエル
ゴパラクリシュナン、カイラッシュ
チェ、ジンウク
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of JP2021505993A publication Critical patent/JP2021505993A/ja
Publication of JP2021505993A5 publication Critical patent/JP2021505993A5/ja
Application granted granted Critical
Publication of JP7087079B2 publication Critical patent/JP7087079B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Electrically Operated Instructional Devices (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
JP2020529245A 2017-12-04 2018-11-30 深層学習アプリケーションのための堅牢な勾配重み圧縮方式 Active JP7087079B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US15/830,170 2017-12-04
US15/830,170 US11295208B2 (en) 2017-12-04 2017-12-04 Robust gradient weight compression schemes for deep learning applications
PCT/IB2018/059516 WO2019111118A1 (en) 2017-12-04 2018-11-30 Robust gradient weight compression schemes for deep learning applications

Publications (3)

Publication Number Publication Date
JP2021505993A JP2021505993A (ja) 2021-02-18
JP2021505993A5 JP2021505993A5 (enExample) 2021-04-01
JP7087079B2 true JP7087079B2 (ja) 2022-06-20

Family

ID=66659264

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2020529245A Active JP7087079B2 (ja) 2017-12-04 2018-11-30 深層学習アプリケーションのための堅牢な勾配重み圧縮方式

Country Status (5)

Country Link
US (1) US11295208B2 (enExample)
JP (1) JP7087079B2 (enExample)
DE (1) DE112018006189T5 (enExample)
GB (1) GB2582232A (enExample)
WO (1) WO2019111118A1 (enExample)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11195096B2 (en) * 2017-10-24 2021-12-07 International Business Machines Corporation Facilitating neural network efficiency
US11216719B2 (en) * 2017-12-12 2022-01-04 Intel Corporation Methods and arrangements to quantize a neural network with machine learning
US11961000B2 (en) * 2018-01-22 2024-04-16 Qualcomm Incorporated Lossy layer compression for dynamic scaling of deep neural network processing
US10698766B2 (en) * 2018-04-18 2020-06-30 EMC IP Holding Company LLC Optimization of checkpoint operations for deep learning computing
CN108665067B (zh) * 2018-05-29 2020-05-29 北京大学 用于深度神经网络频繁传输的压缩方法及系统
WO2020248423A1 (zh) * 2019-06-12 2020-12-17 上海寒武纪信息科技有限公司 一种神经网络的量化参数确定方法及相关产品
CN112149706B (zh) * 2019-06-28 2024-03-15 北京百度网讯科技有限公司 模型训练方法、装置、设备和介质
CN114080613A (zh) * 2019-07-02 2022-02-22 交互数字Ce专利控股公司 对深度神经网络进行编码的系统和方法
US11402233B2 (en) * 2019-07-23 2022-08-02 Mapsted Corp. Maintaining a trained neural network in magnetic fingerprint based indoor navigation
GB2581546B (en) * 2019-08-22 2021-03-31 Imagination Tech Ltd Methods and systems for converting weights of a deep neural network from a first number format to a second number format
CN110659725B (zh) * 2019-09-20 2023-03-31 字节跳动有限公司 神经网络模型的压缩与加速方法、数据处理方法及装置
US11461645B2 (en) * 2019-12-02 2022-10-04 International Business Machines Corporation Initialization of memory networks
CN110995488B (zh) * 2019-12-03 2020-11-03 电子科技大学 一种基于分层参数服务器的多机构协同学习系统及方法
KR102899210B1 (ko) * 2019-12-16 2025-12-10 삼성전자주식회사 뉴럴 프로세싱 장치 및 뉴럴 프로세싱 장치에서 뉴럴 네트워크를 처리하는 방법
CN113297128B (zh) * 2020-02-24 2023-10-31 中科寒武纪科技股份有限公司 数据处理方法、装置、计算机设备和存储介质
US11875256B2 (en) 2020-07-09 2024-01-16 International Business Machines Corporation Dynamic computation in decentralized distributed deep learning training
US11977986B2 (en) 2020-07-09 2024-05-07 International Business Machines Corporation Dynamic computation rates for distributed deep learning
US11886969B2 (en) * 2020-07-09 2024-01-30 International Business Machines Corporation Dynamic network bandwidth in distributed deep learning training
US11900640B2 (en) 2020-07-15 2024-02-13 Tencent America LLC Method and apparatus for substitutional neural residual compression
CN114077889A (zh) * 2020-08-13 2022-02-22 华为技术有限公司 一种神经网络处理器和数据处理方法
CN114519423B (zh) * 2020-11-20 2025-10-24 澜起科技股份有限公司 用于压缩神经网络的方法和装置
US20230397172A1 (en) * 2020-12-29 2023-12-07 Qualcomm Incorporated Signaling of gradient vectors for federated learning in a wireless communications system
US12022098B2 (en) * 2021-03-04 2024-06-25 Lemon Inc. Neural network-based in-loop filter with residual scaling for video coding
US20220292348A1 (en) * 2021-03-15 2022-09-15 Smart Engines Service, LLC Distance-based pairs generation for training metric neural networks
CN114782977B (zh) * 2021-04-28 2024-07-05 河南大学 一种基于拓扑信息和亲和度信息引导行人重识别方法
CN113193999B (zh) * 2021-04-29 2023-12-26 东北大学 一种基于深度确定性策略梯度的虚拟网络映射方法
CN113780461B (zh) * 2021-09-23 2022-08-05 中国人民解放军国防科技大学 基于特征匹配的鲁棒神经网络训练方法
US20240104346A1 (en) * 2022-09-15 2024-03-28 Huawei Technologies Co., Ltd. Method and device for compressing generative pre-trained language models via quantization
CN118052260B (zh) * 2024-04-01 2024-08-02 兰州交通大学 一种神经网络模型动态分层梯度压缩方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598972A (zh) 2015-01-22 2015-05-06 清华大学 一种大规模数据回归神经网络快速训练方法
US20170098171A1 (en) 2015-10-02 2017-04-06 International Business Machines Corporation Asynchronous stochastic gradient descent

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5926804A (en) 1994-07-01 1999-07-20 The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations Discriminant neural networks
KR100364753B1 (ko) * 1999-11-19 2002-12-16 엘지전자 주식회사 칼라 히스토그램의 빈값 양자화 방법
US9235799B2 (en) 2011-11-26 2016-01-12 Microsoft Technology Licensing, Llc Discriminative pretraining of deep neural networks
US9483728B2 (en) 2013-12-06 2016-11-01 International Business Machines Corporation Systems and methods for combining stochastic average gradient and hessian-free optimization for sequence training of deep neural networks
US20180075347A1 (en) * 2016-09-15 2018-03-15 Microsoft Technology Licensing, Llc Efficient training of neural networks
US12190231B2 (en) * 2016-10-19 2025-01-07 Samsung Electronics Co., Ltd Method and apparatus for neural network quantization
US10685285B2 (en) * 2016-11-23 2020-06-16 Microsoft Technology Licensing, Llc Mirror deep neural networks that regularize to linear networks
US11062215B2 (en) * 2017-03-17 2021-07-13 Microsoft Technology Licensing, Llc Using different data sources for a predictive model

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598972A (zh) 2015-01-22 2015-05-06 清华大学 一种大规模数据回归神经网络快速训练方法
US20170098171A1 (en) 2015-10-02 2017-04-06 International Business Machines Corporation Asynchronous stochastic gradient descent

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHEN Chia-Yu et al.，AdaComp : Adaptive Residual Gradient Compression for Data-Parallel Distributed Training，arXiv.org [online]，2017年12月07日，[2022年5月25日検索], インターネット<URL : https://arxiv.org/abs/1712.02679v1>

Also Published As

Publication number Publication date
US11295208B2 (en) 2022-04-05
GB2582232A (en) 2020-09-16
DE112018006189T5 (de) 2020-09-03
JP2021505993A (ja) 2021-02-18
US20190171935A1 (en) 2019-06-06
WO2019111118A1 (en) 2019-06-13
GB202009717D0 (en) 2020-08-12

Similar Documents

Publication Publication Date Title
JP7087079B2 (ja) 深層学習アプリケーションのための堅牢な勾配重み圧縮方式
US11681914B2 (en) Determining multivariate time series data dependencies
US11263052B2 (en) Determining optimal compute resources for distributed batch based optimization applications
CN114667507B (zh) 使用基于应用的剖析的机器学习工作负载的弹性执行
JP7695017B2 (ja) ゼロショット言語間転移学習
US20180240062A1 (en) Collaborative algorithm development, deployment, and tuning platform
US11915121B2 (en) Simulator-assisted training for interpretable generative models
CN113826120B (zh) 神经网络的数据集相关的低秩分解
US11409564B2 (en) Resource allocation for tuning hyperparameters of large-scale deep learning workloads
CN107169573A (zh) 利用复合机器学习模型来执行预测的方法及系统
WO2020148588A1 (en) Machine learning framework for finding materials with desired properties
US12217166B2 (en) Markov processes using analog crossbar arrays
CN112446498A (zh) 在线部分奖励学习
CN116583854A (zh) Rpu交叉开关阵列的最坏情况噪声和界限管理
JP7695012B2 (ja) アナログ・クロスバー・アーキテクチャを使用した行列スケッチング
CN116267023A (zh) 用于门控递归神经网络的门控单元
WO2024002753A1 (en) Thermal and performance management
US11574196B2 (en) Dynamic management of weight update bit length
CN116490871A (zh) 在数据分析中自动调整数据访问策略
US20250061360A1 (en) Quantum oracle decomposition for simulating quantum computing systems and applications
CN117808079A (zh) 一种基于少学习参数的神经网络训练方法及装置
US12488170B1 (en) Scalable noise metrology for quantum circuits
US20230385599A1 (en) Parallel and distributed processing of propositional logical neural networks
US20240020528A1 (en) Training a neural network using an accelerated gradient with shuffling
JP2023099321A (ja) 方法、コンピュータプログラムおよび装置（ニューラルネットワークのためのモデル非依存入力変換）

Legal Events

Date Code Title Description
A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20210128

A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20210423

RD04 Notification of resignation of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: A7424

Effective date: 20220502

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20220531

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20220608

R150 Certificate of patent or registration of utility model

Ref document number: 7087079

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150