CN111160531A - 神经网络模型的分布式训练方法、装置及电子设备 - Google Patents
神经网络模型的分布式训练方法、装置及电子设备 Download PDFInfo
- Publication number
- CN111160531A CN111160531A CN201911403480.5A CN201911403480A CN111160531A CN 111160531 A CN111160531 A CN 111160531A CN 201911403480 A CN201911403480 A CN 201911403480A CN 111160531 A CN111160531 A CN 111160531A
- Authority
- CN
- China
- Prior art keywords
- neural network
- learning rate
- network model
- preset
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003062 neural network model Methods 0.000 title claims abstract description 299
- 238000012549 training Methods 0.000 title claims abstract description 284
- 238000000034 method Methods 0.000 title claims abstract description 71
- 238000004422 calculation algorithm Methods 0.000 claims description 36
- 238000004364 calculation method Methods 0.000 claims description 32
- 230000002238 attenuated effect Effects 0.000 claims description 23
- 238000012545 processing Methods 0.000 claims description 20
- 238000004590 computer program Methods 0.000 claims description 8
- 238000013528 artificial neural network Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 21
- 230000008859 change Effects 0.000 description 9
- 238000013135 deep learning Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 230000006870 function Effects 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000002474 experimental method Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Life Sciences & Earth Sciences (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Image Analysis (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
Abstract
Description
Claims (12)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911403480.5A CN111160531B (zh) | 2019-12-30 | 2019-12-30 | 神经网络模型的分布式训练方法、装置及电子设备 |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
CN201911403480.5A CN111160531B (zh) | 2019-12-30 | 2019-12-30 | 神经网络模型的分布式训练方法、装置及电子设备 |
Publications (2)
Publication Number | Publication Date |
---|---|
CN111160531A true CN111160531A (zh) | 2020-05-15 |
CN111160531B CN111160531B (zh) | 2023-09-22 |
Family
ID=70559806
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CN201911403480.5A Active CN111160531B (zh) | 2019-12-30 | 2019-12-30 | 神经网络模型的分布式训练方法、装置及电子设备 |
Country Status (1)
Country | Link |
---|---|
CN (1) | CN111160531B (zh) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111753997A (zh) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | 分布式训练方法、系统、设备及存储介质 |
CN111898424A (zh) * | 2020-06-19 | 2020-11-06 | 贝壳技术有限公司 | 文字识别模型训练方法、装置、电子设备及存储介质 |
CN112926495A (zh) * | 2021-03-19 | 2021-06-08 | 高新兴科技集团股份有限公司 | 一种基于多级卷积特征级联的车辆检测方法 |
CN113326939A (zh) * | 2021-06-21 | 2021-08-31 | 商汤国际私人有限公司 | 网络训练、行人重识别方法及装置、电子设备和存储介质 |
CN113327598A (zh) * | 2021-06-30 | 2021-08-31 | 北京有竹居网络技术有限公司 | 模型的训练方法、语音识别方法、装置、介质及设备 |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170228645A1 (en) * | 2016-02-05 | 2017-08-10 | Nec Laboratories America, Inc. | Accelerating deep neural network training with inconsistent stochastic gradient descent |
JP2018018422A (ja) * | 2016-07-29 | 2018-02-01 | 株式会社デンソーアイティーラボラトリ | 予測装置、予測方法および予測プログラム |
CN108629369A (zh) * | 2018-04-19 | 2018-10-09 | 中南大学 | 一种基于Trimmed SSD的尿沉渣有形成分自动识别方法 |
DE102018110947A1 (de) * | 2017-05-05 | 2018-11-08 | Jonah Alben | Verlustskalierung für tiefes neuronales netzwerktraining mit verringerter präzision |
CN108876702A (zh) * | 2018-06-21 | 2018-11-23 | 北京邮电大学 | 一种加速分布式深度神经网络的训练方法及装置 |
CN108875903A (zh) * | 2018-01-02 | 2018-11-23 | 北京迈格威科技有限公司 | 图像检测的方法、装置、系统及计算机存储介质 |
CN109949304A (zh) * | 2018-03-29 | 2019-06-28 | 北京昆仑医云科技有限公司 | 图像检测学习网络的训练和获取方法、图像检测装置和介质 |
US20190311479A1 (en) * | 2018-04-10 | 2019-10-10 | Sun Yat-Sen University Cancer Center | Method and device for identifying pathological picture |
US20190378016A1 (en) * | 2018-06-07 | 2019-12-12 | International Business Machines Corporation | Distributed computing architecture for large model deep learning |
-
2019
- 2019-12-30 CN CN201911403480.5A patent/CN111160531B/zh active Active
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170228645A1 (en) * | 2016-02-05 | 2017-08-10 | Nec Laboratories America, Inc. | Accelerating deep neural network training with inconsistent stochastic gradient descent |
JP2018018422A (ja) * | 2016-07-29 | 2018-02-01 | 株式会社デンソーアイティーラボラトリ | 予測装置、予測方法および予測プログラム |
DE102018110947A1 (de) * | 2017-05-05 | 2018-11-08 | Jonah Alben | Verlustskalierung für tiefes neuronales netzwerktraining mit verringerter präzision |
CN108875903A (zh) * | 2018-01-02 | 2018-11-23 | 北京迈格威科技有限公司 | 图像检测的方法、装置、系统及计算机存储介质 |
CN109949304A (zh) * | 2018-03-29 | 2019-06-28 | 北京昆仑医云科技有限公司 | 图像检测学习网络的训练和获取方法、图像检测装置和介质 |
US20190311479A1 (en) * | 2018-04-10 | 2019-10-10 | Sun Yat-Sen University Cancer Center | Method and device for identifying pathological picture |
CN108629369A (zh) * | 2018-04-19 | 2018-10-09 | 中南大学 | 一种基于Trimmed SSD的尿沉渣有形成分自动识别方法 |
US20190378016A1 (en) * | 2018-06-07 | 2019-12-12 | International Business Machines Corporation | Distributed computing architecture for large model deep learning |
CN108876702A (zh) * | 2018-06-21 | 2018-11-23 | 北京邮电大学 | 一种加速分布式深度神经网络的训练方法及装置 |
Non-Patent Citations (3)
Title |
---|
NITISH SHIRISH KESKAR等人: "ON LARGE-BATCH TRAINING FOR DEEP LEARNING: GENERALIZATION GAP AND SHARP MINIMA" * |
李兴建;谭光明;郭子昱;: "GPU机群系统上加速EMAN(英文)" * |
熬冉,谭光明,陈明宇: "基于多核平台无关属性的程序并行度分析工具" * |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN111898424A (zh) * | 2020-06-19 | 2020-11-06 | 贝壳技术有限公司 | 文字识别模型训练方法、装置、电子设备及存储介质 |
CN111753997A (zh) * | 2020-06-28 | 2020-10-09 | 北京百度网讯科技有限公司 | 分布式训练方法、系统、设备及存储介质 |
CN112926495A (zh) * | 2021-03-19 | 2021-06-08 | 高新兴科技集团股份有限公司 | 一种基于多级卷积特征级联的车辆检测方法 |
CN113326939A (zh) * | 2021-06-21 | 2021-08-31 | 商汤国际私人有限公司 | 网络训练、行人重识别方法及装置、电子设备和存储介质 |
CN113327598A (zh) * | 2021-06-30 | 2021-08-31 | 北京有竹居网络技术有限公司 | 模型的训练方法、语音识别方法、装置、介质及设备 |
CN113327598B (zh) * | 2021-06-30 | 2023-11-14 | 北京有竹居网络技术有限公司 | 模型的训练方法、语音识别方法、装置、介质及设备 |
Also Published As
Publication number | Publication date |
---|---|
CN111160531B (zh) | 2023-09-22 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CN111160531A (zh) | 神经网络模型的分布式训练方法、装置及电子设备 | |
US20230252327A1 (en) | Neural architecture search for convolutional neural networks | |
JP7146952B2 (ja) | データ処理方法、装置、コンピュータデバイス、及び記憶媒体 | |
JP7146955B2 (ja) | データ処理方法、装置、コンピュータデバイス、及び記憶媒体 | |
CN110991652A (zh) | 神经网络模型训练方法、装置及电子设备 | |
CN111738098B (zh) | 一种车辆识别方法、装置、设备及存储介质 | |
TW201915839A (zh) | 對人工神經網路及浮點神經網路進行量化的方法及裝置 | |
CN111079899A (zh) | 神经网络模型压缩方法、系统、设备及介质 | |
CN111105017B (zh) | 神经网络量化方法、装置及电子设备 | |
CN111401550A (zh) | 神经网络模型量化方法、装置及电子设备 | |
CN110796253A (zh) | 生成对抗网络的训练方法及设备 | |
US20190138929A1 (en) | System and method for automatic building of learning machines using learning machines | |
CN111695624A (zh) | 数据增强策略的更新方法、装置、设备及存储介质 | |
CN114168318A (zh) | 存储释放模型的训练方法、存储释放方法及设备 | |
CN115860100A (zh) | 一种神经网络模型训练方法、装置及计算设备 | |
CN115437795A (zh) | 一种异构gpu集群负载感知的显存重计算优化方法及系统 | |
US8589852B1 (en) | Statistical corner extraction using worst-case distance | |
CN108509179B (zh) | 用于检测人脸的方法、用于生成模型的装置 | |
CN109783769B (zh) | 一种基于用户项目评分的矩阵分解方法和装置 | |
TWI758223B (zh) | 具有動態最小批次尺寸之運算方法,以及用於執行該方法之運算系統及電腦可讀儲存媒體 | |
CN112766492A (zh) | 模型处理方法、装置、电子设备及存储介质 | |
JP2024043504A (ja) | ニューラルネットワークモデル推論の加速方法、装置、電子機器及び媒体 | |
US20220405561A1 (en) | Electronic device and controlling method of electronic device | |
CN115552424A (zh) | 用于训练包含量化参数的人工神经网络的方法 | |
CN112561050A (zh) | 一种神经网络模型训练方法及装置 |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PB01 | Publication | ||
PB01 | Publication | ||
SE01 | Entry into force of request for substantive examination | ||
SE01 | Entry into force of request for substantive examination | ||
CB03 | Change of inventor or designer information |
Inventor after: Zhong Zhi Hu Inventor after: Tan Guangming Inventor after: Yao Chengji Inventor after: Tian Zhongbo Inventor after: Zhu Hongrui Inventor after: Zhang Xiaoyang Inventor after: Xiao Junmin Inventor before: Zhong Zhi Hu Inventor before: Yao Chengji Inventor before: Tian Zhongbo Inventor before: Zhu Hongrui Inventor before: Zhang Xiaoyang Inventor before: Xiao Junmin |
|
CB03 | Change of inventor or designer information | ||
GR01 | Patent grant | ||
GR01 | Patent grant |