GB2582232A - Robust Gradient weight compression schemes for deep learning applications - Google Patents

Robust Gradient weight compression schemes for deep learning applications Download PDF

Info

Publication number
GB2582232A
GB2582232A GB2009717.6A GB202009717A GB2582232A GB 2582232 A GB2582232 A GB 2582232A GB 202009717 A GB202009717 A GB 202009717A GB 2582232 A GB2582232 A GB 2582232A
Authority
GB
United Kingdom
Prior art keywords
vector
gradient
current
bins
residue vector
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB2009717.6A
Other languages
English (en)
Other versions
GB202009717D0 (en
Inventor
Chen Chia-Yu
Agrawal Ankur
Brand Daniel
Gopalakrishnan Kailash
Choi Jungwook
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of GB202009717D0 publication Critical patent/GB202009717D0/en
Publication of GB2582232A publication Critical patent/GB2582232A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Molecular Biology (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Electrically Operated Instructional Devices (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
GB2009717.6A 2017-12-04 2018-11-30 Robust Gradient weight compression schemes for deep learning applications Withdrawn GB2582232A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US15/830,170 US11295208B2 (en) 2017-12-04 2017-12-04 Robust gradient weight compression schemes for deep learning applications
PCT/IB2018/059516 WO2019111118A1 (en) 2017-12-04 2018-11-30 Robust gradient weight compression schemes for deep learning applications

Publications (2)

Publication Number Publication Date
GB202009717D0 GB202009717D0 (en) 2020-08-12
GB2582232A true GB2582232A (en) 2020-09-16

Family

ID=66659264

Family Applications (1)

Application Number Title Priority Date Filing Date
GB2009717.6A Withdrawn GB2582232A (en) 2017-12-04 2018-11-30 Robust Gradient weight compression schemes for deep learning applications

Country Status (5)

Country Link
US (1) US11295208B2 (enExample)
JP (1) JP7087079B2 (enExample)
DE (1) DE112018006189T5 (enExample)
GB (1) GB2582232A (enExample)
WO (1) WO2019111118A1 (enExample)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11195096B2 (en) * 2017-10-24 2021-12-07 International Business Machines Corporation Facilitating neural network efficiency
US11216719B2 (en) * 2017-12-12 2022-01-04 Intel Corporation Methods and arrangements to quantize a neural network with machine learning
US11961000B2 (en) * 2018-01-22 2024-04-16 Qualcomm Incorporated Lossy layer compression for dynamic scaling of deep neural network processing
US10698766B2 (en) * 2018-04-18 2020-06-30 EMC IP Holding Company LLC Optimization of checkpoint operations for deep learning computing
CN108665067B (zh) * 2018-05-29 2020-05-29 北京大学 用于深度神经网络频繁传输的压缩方法及系统
KR102609719B1 (ko) * 2019-06-12 2023-12-04 상하이 캠브리콘 인포메이션 테크놀로지 컴퍼니 리미티드 신경망의 양자화 파라미터 확정방법 및 관련제품
CN112149706B (zh) * 2019-06-28 2024-03-15 北京百度网讯科技有限公司 模型训练方法、装置、设备和介质
WO2021001687A1 (en) * 2019-07-02 2021-01-07 Interdigital Ce Patent Holdings, Sas Systems and methods for encoding a deep neural network
US11402233B2 (en) * 2019-07-23 2022-08-02 Mapsted Corp. Maintaining a trained neural network in magnetic fingerprint based indoor navigation
GB2581546B (en) * 2019-08-22 2021-03-31 Imagination Tech Ltd Methods and systems for converting weights of a deep neural network from a first number format to a second number format
CN110659725B (zh) * 2019-09-20 2023-03-31 字节跳动有限公司 神经网络模型的压缩与加速方法、数据处理方法及装置
US11461645B2 (en) * 2019-12-02 2022-10-04 International Business Machines Corporation Initialization of memory networks
CN110995488B (zh) * 2019-12-03 2020-11-03 电子科技大学 一种基于分层参数服务器的多机构协同学习系统及方法
KR102899210B1 (ko) * 2019-12-16 2025-12-10 삼성전자주식회사 뉴럴 프로세싱 장치 및 뉴럴 프로세싱 장치에서 뉴럴 네트워크를 처리하는 방법
CN113297128B (zh) * 2020-02-24 2023-10-31 中科寒武纪科技股份有限公司 数据处理方法、装置、计算机设备和存储介质
US11875256B2 (en) 2020-07-09 2024-01-16 International Business Machines Corporation Dynamic computation in decentralized distributed deep learning training
US11977986B2 (en) 2020-07-09 2024-05-07 International Business Machines Corporation Dynamic computation rates for distributed deep learning
US11886969B2 (en) * 2020-07-09 2024-01-30 International Business Machines Corporation Dynamic network bandwidth in distributed deep learning training
US11900640B2 (en) * 2020-07-15 2024-02-13 Tencent America LLC Method and apparatus for substitutional neural residual compression
CN114077889A (zh) * 2020-08-13 2022-02-22 华为技术有限公司 一种神经网络处理器和数据处理方法
CN114519423B (zh) * 2020-11-20 2025-10-24 澜起科技股份有限公司 用于压缩神经网络的方法和装置
WO2022141034A1 (en) * 2020-12-29 2022-07-07 Qualcomm Incorporated Signaling of gradient vectors for federated learning in a wireless communications system
US12022098B2 (en) * 2021-03-04 2024-06-25 Lemon Inc. Neural network-based in-loop filter with residual scaling for video coding
US20220292348A1 (en) * 2021-03-15 2022-09-15 Smart Engines Service, LLC Distance-based pairs generation for training metric neural networks
CN114782977B (zh) * 2021-04-28 2024-07-05 河南大学 一种基于拓扑信息和亲和度信息引导行人重识别方法
CN113193999B (zh) * 2021-04-29 2023-12-26 东北大学 一种基于深度确定性策略梯度的虚拟网络映射方法
CN113780461B (zh) * 2021-09-23 2022-08-05 中国人民解放军国防科技大学 基于特征匹配的鲁棒神经网络训练方法
US20240104346A1 (en) * 2022-09-15 2024-03-28 Huawei Technologies Co., Ltd. Method and device for compressing generative pre-trained language models via quantization
CN118052260B (zh) * 2024-04-01 2024-08-02 兰州交通大学 一种神经网络模型动态分层梯度压缩方法

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598972A (zh) * 2015-01-22 2015-05-06 清华大学 一种大规模数据回归神经网络快速训练方法
US20170098171A1 (en) * 2015-10-02 2017-04-06 International Business Machines Corporation Asynchronous stochastic gradient descent

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5926804A (en) 1994-07-01 1999-07-20 The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations Discriminant neural networks
KR100364753B1 (ko) * 1999-11-19 2002-12-16 엘지전자 주식회사 칼라 히스토그램의 빈값 양자화 방법
US9235799B2 (en) 2011-11-26 2016-01-12 Microsoft Technology Licensing, Llc Discriminative pretraining of deep neural networks
US9483728B2 (en) 2013-12-06 2016-11-01 International Business Machines Corporation Systems and methods for combining stochastic average gradient and hessian-free optimization for sequence training of deep neural networks
US20180075347A1 (en) * 2016-09-15 2018-03-15 Microsoft Technology Licensing, Llc Efficient training of neural networks
US12190231B2 (en) * 2016-10-19 2025-01-07 Samsung Electronics Co., Ltd Method and apparatus for neural network quantization
US10685285B2 (en) * 2016-11-23 2020-06-16 Microsoft Technology Licensing, Llc Mirror deep neural networks that regularize to linear networks
US11062215B2 (en) * 2017-03-17 2021-07-13 Microsoft Technology Licensing, Llc Using different data sources for a predictive model

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN104598972A (zh) * 2015-01-22 2015-05-06 清华大学 一种大规模数据回归神经网络快速训练方法
US20170098171A1 (en) * 2015-10-02 2017-04-06 International Business Machines Corporation Asynchronous stochastic gradient descent

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Strom, 2015, 'Scalable Distributed DNN Training Using Commodity GPU Cloud Computing' in Sixteenth Annual Conference of the International Speech Communication Association, available at: https://www.isca-speech.org/archive/interspeech_2015/papers/i15_1488.pdf, [accessed 3 August 2020] *

Also Published As

Publication number Publication date
DE112018006189T5 (de) 2020-09-03
US11295208B2 (en) 2022-04-05
WO2019111118A1 (en) 2019-06-13
US20190171935A1 (en) 2019-06-06
JP7087079B2 (ja) 2022-06-20
GB202009717D0 (en) 2020-08-12
JP2021505993A (ja) 2021-02-18

Similar Documents

Publication Publication Date Title
GB2582232A (en) Robust Gradient weight compression schemes for deep learning applications
JP2021505993A5 (enExample)
CN105184362B (zh) 基于参数量化的深度卷积神经网络的加速与压缩方法
CN110222821B (zh) 基于权重分布的卷积神经网络低位宽量化方法
US11461515B2 (en) Optimization apparatus, simulation system and optimization method for semiconductor design
JP6706326B2 (ja) リカレントニューラルネットワークモデルの圧縮
US20200134473A1 (en) Data discriminator training method, data discriminator training apparatus, non-transitory computer readable medium, and training method
WO2017031630A1 (zh) 基于参数量化的深度卷积神经网络的加速与压缩方法
WO2022006919A1 (zh) 基于激活定点拟合的卷积神经网络训练后量化方法及系统
WO2019157251A1 (en) Neural network compression
CN112020724A (zh) 学习可压缩的特征
US12032523B2 (en) Compressed sensing using neural networks
CN108229652B (zh) 神经网络模型迁移方法和系统、电子设备、程序和介质
KR102732517B1 (ko) 뉴럴 네트워크에서 파라미터를 처리하는 방법 및 장치
GB2600055A (en) Dataset dependent low rank decomposition of neural networks
US10482157B2 (en) Data compression apparatus and data compression method and storage medium
US11531884B2 (en) Separate quantization method of forming combination of 4-bit and 8-bit data of neural network
WO2020126692A1 (en) Implementing a multi-layer neural network using a single physical layer of analog neurons
Samui Prediction of pile bearing capacity using support vector machine
CN114503439A (zh) 压缩表现混合压缩性的数据
KR102494095B1 (ko) 인공 신경망 학습 장치 및 방법
US20220004908A1 (en) Information processing apparatus, information processing system, information processing method, and non-transitory computer readable medium storing program
Chen et al. Estimation of line-of-sight velocities of individual galaxies using neural networks–I. Modelling redshift–space distortions at large scales
Mozaffari et al. SLiM: One-shot Quantization and Sparsity with Low-rank Approximation for LLM Weight Compression
US20220215203A1 (en) Storage medium, information processing apparatus, and determination model generation method

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)