JP7623398B2 - メモリ要件が低減されたデュアルモーメンタム勾配最適化 - Google Patents

メモリ要件が低減されたデュアルモーメンタム勾配最適化 Download PDF

Info

Publication number
JP7623398B2
JP7623398B2 JP2022561510A JP2022561510A JP7623398B2 JP 7623398 B2 JP7623398 B2 JP 7623398B2 JP 2022561510 A JP2022561510 A JP 2022561510A JP 2022561510 A JP2022561510 A JP 2022561510A JP 7623398 B2 JP7623398 B2 JP 7623398B2
Authority
JP
Japan
Prior art keywords
momentum values
momentum
format
gradient
values
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2022561510A
Other languages
English (en)
Japanese (ja)
Other versions
JP2023521975A (ja
Inventor
シー，ジンウェン
プディペディ，バラドワージ
トレンブレイ，マーク
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of JP2023521975A publication Critical patent/JP2023521975A/ja
Application granted granted Critical
Publication of JP7623398B2 publication Critical patent/JP7623398B2/ja
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/044—Recurrent networks, e.g. Hopfield networks
    • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0495—Quantised networks; Sparse networks; Compressed networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/098—Distributed learning, e.g. federated learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N5/00—Computing arrangements using knowledge-based models
    • G06N5/04—Inference or reasoning models
    • G06N5/046—Forward inferencing; Production systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Mathematical Physics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Neurology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Medical Informatics (AREA)
  • Complex Calculations (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)
  • Advance Control (AREA)
  • Feedback Control In General (AREA)
JP2022561510A 2020-04-17 2021-02-09 メモリ要件が低減されたデュアルモーメンタム勾配最適化 Active JP7623398B2 (ja)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US16/851,847 US11651228B2 (en) 2020-04-17 2020-04-17 Dual-momentum gradient optimization with reduced memory requirements
US16/851,847 2020-04-17
PCT/US2021/017215 WO2021211193A1 (en) 2020-04-17 2021-02-09 Dual-momentum gradient optimization with reduced memory requirements

Publications (2)

Publication Number Publication Date
JP2023521975A JP2023521975A (ja) 2023-05-26
JP7623398B2 true JP7623398B2 (ja) 2025-01-28

Family

ID=74856929

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2022561510A Active JP7623398B2 (ja) 2020-04-17 2021-02-09 メモリ要件が低減されたデュアルモーメンタム勾配最適化

Country Status (6)

Country Link
US (2) US11651228B2 (de)
EP (1) EP4136587A1 (de)
JP (1) JP7623398B2 (de)
KR (1) KR102856047B1 (de)
CN (1) CN115398449A (de)
WO (1) WO2021211193A1 (de)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP7840037B2 (ja) * 2022-01-28 2026-04-03 国立大学法人北海道大学 最適化装置及び最適化方法並びに最適化用プログラム
CN116007597B (zh) * 2022-12-19 2024-06-11 北京工业大学 基于动量梯度下降法对框架柱的垂直度测量方法及装置
KR102842262B1 (ko) * 2024-04-03 2025-08-04 울산과학기술원 정방행렬화 알고리즘을 이용한 메모리 효율적인 심층신경망 최적화 장치 및 방법

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190034784A1 (en) 2017-07-28 2019-01-31 Beijing Deephi Intelligence Technology Co., Ltd. Fixed-point training method for deep neural networks based on dynamic fixed-point conversion scheme
JP2019512126A (ja) 2016-02-29 2019-05-09 アリババ グループ ホウルディング リミテッド 機械学習システムをトレーニングする方法及びシステム

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10572800B2 (en) 2016-02-05 2020-02-25 Nec Corporation Accelerating deep neural network training with inconsistent stochastic gradient descent
CN107526709A (zh) * 2016-06-15 2017-12-29 辉达公司 使用低精度格式的张量处理
US11276002B2 (en) 2017-12-20 2022-03-15 Salesforce.Com, Inc. Hybrid training of deep networks
US10372991B1 (en) * 2018-04-03 2019-08-06 Google Llc Systems and methods that leverage deep learning to selectively store audiovisual content
US11645493B2 (en) * 2018-05-04 2023-05-09 Microsoft Technology Licensing, Llc Flow for quantized neural networks
KR102732517B1 (ko) * 2018-07-04 2024-11-20 삼성전자주식회사 뉴럴 네트워크에서 파라미터를 처리하는 방법 및 장치
US11586904B2 (en) * 2018-09-13 2023-02-21 Google Llc Adaptive optimization with improved convergence
US20200380369A1 (en) * 2019-05-31 2020-12-03 Nvidia Corporation Training a neural network using selective weight updates
US10769528B1 (en) * 2019-06-07 2020-09-08 Sas Institute Inc. Deep learning model training system
KR20190098107A (ko) * 2019-08-02 2019-08-21 엘지전자 주식회사 딥 러닝을 위한 신경망 학습 장치 및 그 방법
US12175359B2 (en) * 2019-09-03 2024-12-24 International Business Machines Corporation Machine learning hardware having reduced precision parameter components for efficient parameter update

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2019512126A (ja) 2016-02-29 2019-05-09 アリババ グループ ホウルディング リミテッド 機械学習システムをトレーニングする方法及びシステム
US20190034784A1 (en) 2017-07-28 2019-01-31 Beijing Deephi Intelligence Technology Co., Ltd. Fixed-point training method for deep neural networks based on dynamic fixed-point conversion scheme

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
Sebastian Ruder，An overview of gradient descent optimization algorithms，arxiv.org, [online]，2017年06月15日，[検索日 2024.09.09], Retrieved from the Internet: <URL: https://arxiv.org/pdf/1609.04747>

Also Published As

Publication number Publication date
CN115398449A (zh) 2022-11-25
KR102856047B1 (ko) 2025-09-04
US11651228B2 (en) 2023-05-16
JP2023521975A (ja) 2023-05-26
KR20230006815A (ko) 2023-01-11
EP4136587A1 (de) 2023-02-22
US20230244945A1 (en) 2023-08-03
WO2021211193A1 (en) 2021-10-21
US20210326711A1 (en) 2021-10-21

Similar Documents

Publication Publication Date Title
CN114402293B (zh) 具有持续且异步更新的流水线式神经网络处理
US20230244945A1 (en) Dual-momentum gradient optimization with reduced memory requirements
US20200082269A1 (en) Memory efficient neural networks
CN112836787B (zh) 通过高效混合并行化减少深度神经网络训练次数
WO2020167480A1 (en) Adjusting activation compression for neural network training
WO2020142192A1 (en) Neural network activation compression with narrow block floating-point
US20170061279A1 (en) Updating an artificial neural network using flexible fixed point representation
CN113273082A (zh) 具有异常块浮点的神经网络激活压缩
CN114626516A (zh) 一种基于对数块浮点量化的神经网络加速系统
WO2020176248A1 (en) Neural network layer processing with scaled quantization
EP3931756A1 (de) Verarbeitung neuronaler netzwerkschichten mit normalisierung und umwandlung von daten
CN113508402A (zh) 从量化的固件神经网络层得出一致的软件神经网络层
US20240220572A1 (en) Pipeline-parallel-dataflow artificial intelligence system for accelerating self-attention computations
CN119884332B (zh) 一种应答信息生成方法、设备、介质及计算机程序产品
US12353984B2 (en) Hardware-assisted gradient optimization using streamed gradients
CN119005265A (zh) 面向高性能数据并行dnn训练的稀疏化压缩方法及装置
JP2024529835A (ja) 二重指数バウンディングボックス浮動小数点プロセッサ
EP4196919A1 (de) Verfahren und system zur quantisierung eines neuronalen netzes
CN117808079A (zh) 一种基于少学习参数的神经网络训练方法及装置
He et al. Research on Efficient CNN Acceleration Through Mixed Precision Quantization: A Comprehensive Methodology.
US20250377940A1 (en) Hardware acceleration for generative models
RU2795887C2 (ru) Матрично-векторный умножитель с набором регистров для хранения векторов, содержащим многопортовую память
Huang et al. A Data-Efficient Deep Reinforcement Learning Algorithm and FPGA Accelerator for Real-Time Robot Motion Control Applications
Xie Hardware Accelerator for LSTM Neural Networks using High-Level Synthesis
CN121523860A (zh) 一种基于多gpu编码的共现矩阵构建方法、设备及介质

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20240125

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20240917

A521 Request for written amendment filed

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20241206

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20241217

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20250116

R150 Certificate of patent or registration of utility model

Ref document number: 7623398

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150

RD02 Notification of acceptance of power of attorney

Free format text: JAPANESE INTERMEDIATE CODE: R3D02