JP2021505993A5 - - Google Patents
Download PDFInfo
- Publication number
- JP2021505993A5 JP2021505993A5 JP2020529245A JP2020529245A JP2021505993A5 JP 2021505993 A5 JP2021505993 A5 JP 2021505993A5 JP 2020529245 A JP2020529245 A JP 2020529245A JP 2020529245 A JP2020529245 A JP 2020529245A JP 2021505993 A5 JP2021505993 A5 JP 2021505993A5
- Authority
- JP
- Japan
- Prior art keywords
- residual
- gradient
- vector
- current
- weights
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000013528 artificial neural network Methods 0.000 claims 19
- 238000000034 method Methods 0.000 claims 10
- 238000013139 quantization Methods 0.000 claims 9
- 230000006835 compression Effects 0.000 claims 5
- 238000007906 compression Methods 0.000 claims 5
- 238000013135 deep learning Methods 0.000 claims 5
- 230000003044 adaptive effect Effects 0.000 claims 3
- 238000004590 computer program Methods 0.000 claims 2
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US15/830,170 US11295208B2 (en) | 2017-12-04 | 2017-12-04 | Robust gradient weight compression schemes for deep learning applications |
| US15/830,170 | 2017-12-04 | ||
| PCT/IB2018/059516 WO2019111118A1 (en) | 2017-12-04 | 2018-11-30 | Robust gradient weight compression schemes for deep learning applications |
Publications (3)
| Publication Number | Publication Date |
|---|---|
| JP2021505993A JP2021505993A (ja) | 2021-02-18 |
| JP2021505993A5 true JP2021505993A5 (enExample) | 2021-04-01 |
| JP7087079B2 JP7087079B2 (ja) | 2022-06-20 |
Family
ID=66659264
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2020529245A Active JP7087079B2 (ja) | 2017-12-04 | 2018-11-30 | 深層学習アプリケーションのための堅牢な勾配重み圧縮方式 |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US11295208B2 (enExample) |
| JP (1) | JP7087079B2 (enExample) |
| DE (1) | DE112018006189T5 (enExample) |
| GB (1) | GB2582232A (enExample) |
| WO (1) | WO2019111118A1 (enExample) |
Families Citing this family (29)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US11195096B2 (en) * | 2017-10-24 | 2021-12-07 | International Business Machines Corporation | Facilitating neural network efficiency |
| US11216719B2 (en) * | 2017-12-12 | 2022-01-04 | Intel Corporation | Methods and arrangements to quantize a neural network with machine learning |
| US11961000B2 (en) * | 2018-01-22 | 2024-04-16 | Qualcomm Incorporated | Lossy layer compression for dynamic scaling of deep neural network processing |
| US10698766B2 (en) * | 2018-04-18 | 2020-06-30 | EMC IP Holding Company LLC | Optimization of checkpoint operations for deep learning computing |
| CN108665067B (zh) * | 2018-05-29 | 2020-05-29 | 北京大学 | 用于深度神经网络频繁传输的压缩方法及系统 |
| KR102609719B1 (ko) * | 2019-06-12 | 2023-12-04 | 상하이 캠브리콘 인포메이션 테크놀로지 컴퍼니 리미티드 | 신경망의 양자화 파라미터 확정방법 및 관련제품 |
| CN112149706B (zh) * | 2019-06-28 | 2024-03-15 | 北京百度网讯科技有限公司 | 模型训练方法、装置、设备和介质 |
| WO2021001687A1 (en) * | 2019-07-02 | 2021-01-07 | Interdigital Ce Patent Holdings, Sas | Systems and methods for encoding a deep neural network |
| US11402233B2 (en) * | 2019-07-23 | 2022-08-02 | Mapsted Corp. | Maintaining a trained neural network in magnetic fingerprint based indoor navigation |
| GB2581546B (en) * | 2019-08-22 | 2021-03-31 | Imagination Tech Ltd | Methods and systems for converting weights of a deep neural network from a first number format to a second number format |
| CN110659725B (zh) * | 2019-09-20 | 2023-03-31 | 字节跳动有限公司 | 神经网络模型的压缩与加速方法、数据处理方法及装置 |
| US11461645B2 (en) * | 2019-12-02 | 2022-10-04 | International Business Machines Corporation | Initialization of memory networks |
| CN110995488B (zh) * | 2019-12-03 | 2020-11-03 | 电子科技大学 | 一种基于分层参数服务器的多机构协同学习系统及方法 |
| US11410410B2 (en) * | 2019-12-16 | 2022-08-09 | Samsung Electronics Co., Ltd. | Method and apparatus with neural network processing |
| CN113297128B (zh) * | 2020-02-24 | 2023-10-31 | 中科寒武纪科技股份有限公司 | 数据处理方法、装置、计算机设备和存储介质 |
| US11977986B2 (en) | 2020-07-09 | 2024-05-07 | International Business Machines Corporation | Dynamic computation rates for distributed deep learning |
| US11875256B2 (en) | 2020-07-09 | 2024-01-16 | International Business Machines Corporation | Dynamic computation in decentralized distributed deep learning training |
| US11886969B2 (en) * | 2020-07-09 | 2024-01-30 | International Business Machines Corporation | Dynamic network bandwidth in distributed deep learning training |
| US11900640B2 (en) * | 2020-07-15 | 2024-02-13 | Tencent America LLC | Method and apparatus for substitutional neural residual compression |
| CN114077889A (zh) * | 2020-08-13 | 2022-02-22 | 华为技术有限公司 | 一种神经网络处理器和数据处理方法 |
| CN114519423B (zh) * | 2020-11-20 | 2025-10-24 | 澜起科技股份有限公司 | 用于压缩神经网络的方法和装置 |
| CN116711249A (zh) * | 2020-12-29 | 2023-09-05 | 高通股份有限公司 | 用于无线通信系统中的联合学习的梯度向量的信令 |
| US12022098B2 (en) * | 2021-03-04 | 2024-06-25 | Lemon Inc. | Neural network-based in-loop filter with residual scaling for video coding |
| US20220292348A1 (en) * | 2021-03-15 | 2022-09-15 | Smart Engines Service, LLC | Distance-based pairs generation for training metric neural networks |
| CN114782977B (zh) * | 2021-04-28 | 2024-07-05 | 河南大学 | 一种基于拓扑信息和亲和度信息引导行人重识别方法 |
| CN113193999B (zh) * | 2021-04-29 | 2023-12-26 | 东北大学 | 一种基于深度确定性策略梯度的虚拟网络映射方法 |
| CN113780461B (zh) * | 2021-09-23 | 2022-08-05 | 中国人民解放军国防科技大学 | 基于特征匹配的鲁棒神经网络训练方法 |
| US20240104346A1 (en) * | 2022-09-15 | 2024-03-28 | Huawei Technologies Co., Ltd. | Method and device for compressing generative pre-trained language models via quantization |
| CN118052260B (zh) * | 2024-04-01 | 2024-08-02 | 兰州交通大学 | 一种神经网络模型动态分层梯度压缩方法 |
Family Cites Families (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5926804A (en) | 1994-07-01 | 1999-07-20 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | Discriminant neural networks |
| KR100364753B1 (ko) * | 1999-11-19 | 2002-12-16 | 엘지전자 주식회사 | 칼라 히스토그램의 빈값 양자화 방법 |
| US9235799B2 (en) | 2011-11-26 | 2016-01-12 | Microsoft Technology Licensing, Llc | Discriminative pretraining of deep neural networks |
| US9483728B2 (en) | 2013-12-06 | 2016-11-01 | International Business Machines Corporation | Systems and methods for combining stochastic average gradient and hessian-free optimization for sequence training of deep neural networks |
| CN104598972A (zh) | 2015-01-22 | 2015-05-06 | 清华大学 | 一种大规模数据回归神经网络快速训练方法 |
| US10628740B2 (en) | 2015-10-02 | 2020-04-21 | International Business Machines Corporation | Asynchronous stochastic gradient descent |
| US20180075347A1 (en) * | 2016-09-15 | 2018-03-15 | Microsoft Technology Licensing, Llc | Efficient training of neural networks |
| US12190231B2 (en) * | 2016-10-19 | 2025-01-07 | Samsung Electronics Co., Ltd | Method and apparatus for neural network quantization |
| US10685285B2 (en) * | 2016-11-23 | 2020-06-16 | Microsoft Technology Licensing, Llc | Mirror deep neural networks that regularize to linear networks |
| US11062215B2 (en) * | 2017-03-17 | 2021-07-13 | Microsoft Technology Licensing, Llc | Using different data sources for a predictive model |
-
2017
- 2017-12-04 US US15/830,170 patent/US11295208B2/en active Active
-
2018
- 2018-11-30 JP JP2020529245A patent/JP7087079B2/ja active Active
- 2018-11-30 WO PCT/IB2018/059516 patent/WO2019111118A1/en not_active Ceased
- 2018-11-30 DE DE112018006189.2T patent/DE112018006189T5/de active Pending
- 2018-11-30 GB GB2009717.6A patent/GB2582232A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP2021505993A5 (enExample) | ||
| GB2582232A (en) | Robust Gradient weight compression schemes for deep learning applications | |
| CN105184362B (zh) | 基于参数量化的深度卷积神经网络的加速与压缩方法 | |
| KR102732517B1 (ko) | 뉴럴 네트워크에서 파라미터를 처리하는 방법 및 장치 | |
| US11531932B2 (en) | Systems and methods for compression and distribution of machine learning models | |
| KR101899101B1 (ko) | 인공 신경망 기반 예측 모델 생성 장치 및 방법 | |
| TWI638272B (zh) | 用於對類神經網路執行類神經網路計算之系統與方法及相關正規化電路 | |
| WO2022006919A1 (zh) | 基于激活定点拟合的卷积神经网络训练后量化方法及系统 | |
| WO2017031630A1 (zh) | 基于参数量化的深度卷积神经网络的加速与压缩方法 | |
| CN112955907A (zh) | 量化训练的长短期记忆神经网络 | |
| TW201915839A (zh) | 對人工神經網路及浮點神經網路進行量化的方法及裝置 | |
| JP2022502733A (ja) | ニューラル・ネットワーク・コアにおける動的精度のためのデータ表現 | |
| JP2021504836A5 (enExample) | ||
| EP3494522A1 (en) | Communication efficient federated learning | |
| TW201820174A (zh) | 整體式學習預測裝置與方法、以及非暫存電腦可讀的儲存媒介 | |
| CN111461445B (zh) | 短期风速预测方法、装置、计算机设备及存储介质 | |
| US20110276150A1 (en) | Neural network optimizing sliding mode controller | |
| WO2021086861A1 (en) | Quantized architecture search for machine learning models | |
| CN108334945A (zh) | 深度神经网络的加速与压缩方法及装置 | |
| US20220058472A1 (en) | Implementing a multi-layer neural network using a single physical layer of analog neurons | |
| CN110213784A (zh) | 一种流量预测方法及装置 | |
| CN112384935A (zh) | 分布式神经网络核心网络中的层级并行 | |
| KR20200049422A (ko) | 시뮬레이션-가이드된 반복적 프루닝을 사용하는 효율적인 네트워크 압축 | |
| CN110222816B (zh) | 深度学习模型的建立方法、图像处理方法及装置 | |
| Yvinec et al. | Nupes: Non-uniform post-training quantization via power exponent search |