GB2593070A - Decentralized distributed deep learning - Google Patents

Decentralized distributed deep learning Download PDF

Info

Publication number
GB2593070A
GB2593070A GB2105988.6A GB202105988A GB2593070A GB 2593070 A GB2593070 A GB 2593070A GB 202105988 A GB202105988 A GB 202105988A GB 2593070 A GB2593070 A GB 2593070A
Authority
GB
United Kingdom
Prior art keywords
node
thread
list
neighbor
nodes
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
GB2105988.6A
Other languages
English (en)
Other versions
GB202105988D0 (en
Inventor
Zhang Wei
Zhang Li
Alfons Finkler Ulrich
Cho Minsik
Kung David
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
International Business Machines Corp
Original Assignee
International Business Machines Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by International Business Machines Corp filed Critical International Business Machines Corp
Publication of GB202105988D0 publication Critical patent/GB202105988D0/en
Publication of GB2593070A publication Critical patent/GB2593070A/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/54—Interprogram communication
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F17/00—Digital computing or data processing equipment or methods, specially adapted for specific functions
    • G06F17/10—Complex mathematical operations
    • G06F17/16—Matrix or vector computation, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • G06F18/2148—Generating training patterns; Bootstrap methods, e.g. bagging or boosting characterised by the process organisation or structure, e.g. boosting cascade
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/22—Matching criteria, e.g. proximity measures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/46—Multiprogramming arrangements
    • G06F9/48—Program initiating; Program switching, e.g. by interrupt
    • G06F9/4806—Task transfer initiation or dispatching
    • G06F9/4843—Task transfer initiation or dispatching by program, e.g. task dispatcher, supervisor, operating system
    • G06F9/4881—Scheduling strategies for dispatcher, e.g. round robin, multi-level priority queues
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/082—Learning methods modifying the architecture, e.g. adding, deleting or silencing nodes or connections
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/084—Backpropagation, e.g. using gradient descent
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L67/00—Network arrangements or protocols for supporting network services or applications
    • H04L67/01—Protocols
    • H04L67/10—Protocols in which an application is distributed across nodes in the network
    • H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/40—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass for recovering from a failure of a protocol instance or entity, e.g. service redundancy protocols, protocol state redundancy or protocol service redirection

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • General Health & Medical Sciences (AREA)
  • Computational Linguistics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Computational Mathematics (AREA)
  • Mathematical Analysis (AREA)
  • Pure & Applied Mathematics (AREA)
  • Mathematical Optimization (AREA)
  • Neurology (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Evolutionary Biology (AREA)
  • Computer Security & Cryptography (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Medical Informatics (AREA)
  • Algebra (AREA)
  • Databases & Information Systems (AREA)
  • Debugging And Monitoring (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Hardware Redundancy (AREA)
GB2105988.6A 2018-11-30 2019-11-05 Decentralized distributed deep learning Withdrawn GB2593070A (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US16/206,274 US11521067B2 (en) 2018-11-30 2018-11-30 Decentralized distributed deep learning
PCT/IB2019/059474 WO2020109891A1 (en) 2018-11-30 2019-11-05 Decentralized distributed deep learning

Publications (2)

Publication Number Publication Date
GB202105988D0 GB202105988D0 (en) 2021-06-09
GB2593070A true GB2593070A (en) 2021-09-15

Family

ID=70850132

Family Applications (1)

Application Number Title Priority Date Filing Date
GB2105988.6A Withdrawn GB2593070A (en) 2018-11-30 2019-11-05 Decentralized distributed deep learning

Country Status (6)

Country Link
US (1) US11521067B2 (zh)
JP (1) JP7308003B2 (zh)
CN (1) CN113168569A (zh)
DE (1) DE112019004076T5 (zh)
GB (1) GB2593070A (zh)
WO (1) WO2020109891A1 (zh)

Families Citing this family (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US11244243B2 (en) * 2018-01-19 2022-02-08 Hypernet Labs, Inc. Coordinated learning using distributed average consensus
US10878482B2 (en) 2018-01-19 2020-12-29 Hypernet Labs, Inc. Decentralized recommendations using distributed average consensus
US11521067B2 (en) * 2018-11-30 2022-12-06 International Business Machines Corporation Decentralized distributed deep learning
JP7230683B2 (ja) * 2019-05-21 2023-03-01 富士通株式会社 演算処理装置、プログラム、及び演算処理装置の制御方法
CN112308233A (zh) * 2019-08-02 2021-02-02 伊姆西Ip控股有限责任公司 用于处理数据的方法、设备和计算机程序产品
KR20190106944A (ko) * 2019-08-30 2019-09-18 엘지전자 주식회사 지능형 냉장고 및 그 제어 방법
US11651293B2 (en) * 2020-07-22 2023-05-16 International Business Machines Corporation Hierarchical decentralized distributed deep learning training
CN111858721B (zh) * 2020-08-03 2023-07-21 南京大学 一种基于优先级编码的分布式计算方法
US20220129746A1 (en) * 2020-10-27 2022-04-28 International Business Machines Corporation Decentralized parallel min/max optimization
US11636280B2 (en) 2021-01-27 2023-04-25 International Business Machines Corporation Updating of statistical sets for decentralized distributed training of a machine learning model
CN113255933A (zh) * 2021-06-01 2021-08-13 上海商汤科技开发有限公司 特征工程和图网络生成方法和装置、分布式系统
CN113692031B (zh) * 2021-09-08 2024-04-19 中国海洋大学 船舶自组织网络中基于多维社区的路由方法
CN113986149B (zh) * 2021-12-27 2022-04-22 苏州浪潮智能科技有限公司 一种系统故障处理方法、装置、设备及存储介质
CN115277689B (zh) * 2022-04-29 2023-09-22 国网天津市电力公司 一种基于分布式联邦学习的云边网络通信优化方法及系统
CN115665174B (zh) * 2022-12-13 2023-03-14 浪潮电子信息产业股份有限公司 一种梯度数据的同步方法、系统、设备及存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170169358A1 (en) * 2015-12-09 2017-06-15 Samsung Electronics Co., Ltd. In-storage computing apparatus and method for decentralized machine learning
JP2018036779A (ja) * 2016-08-30 2018-03-08 株式会社東芝 電子装置、方法及び情報処理システム
CN108304918A (zh) * 2018-01-18 2018-07-20 中兴飞流信息科技有限公司 一种数据并行的深度学习的参数交换方法和系统
US20180253818A1 (en) * 2017-03-03 2018-09-06 International Business Machines Corporation Deep learning via dynamic root solvers
US20180322386A1 (en) * 2017-05-05 2018-11-08 Intel Corporation Fine-grain compute communication execution for deep learning frameworks

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20070192863A1 (en) * 2005-07-01 2007-08-16 Harsh Kapoor Systems and methods for processing data flows
EP1941432A4 (en) * 2005-10-25 2011-04-20 Angoss Software Corp STRATEGY TREES FOR DATA MINING
US20120284067A1 (en) * 2011-05-03 2012-11-08 Intuit Inc. Revenue-based impact analysis using multidimensional models of software offerings
US8856328B2 (en) * 2011-07-29 2014-10-07 Cellco Partnership Fail-open network techniques
US9916538B2 (en) * 2012-09-15 2018-03-13 Z Advanced Computing, Inc. Method and system for feature detection
US9355088B2 (en) 2013-07-12 2016-05-31 Microsoft Technology Licensing, Llc Feature completion in computer-human interactive learning
US9672474B2 (en) 2014-06-30 2017-06-06 Amazon Technologies, Inc. Concurrent binning of machine learning data
US10339465B2 (en) * 2014-06-30 2019-07-02 Amazon Technologies, Inc. Optimized decision tree based models
US9699049B2 (en) * 2014-09-23 2017-07-04 Ebay Inc. Predictive model for anomaly detection and feedback-based scheduling
US20160325680A1 (en) * 2015-05-04 2016-11-10 Kamama, Inc. System and method of vehicle sensor management
US10628740B2 (en) * 2015-10-02 2020-04-21 International Business Machines Corporation Asynchronous stochastic gradient descent
US10586173B2 (en) 2016-01-27 2020-03-10 Bonsai AI, Inc. Searchable database of trained artificial intelligence objects that can be reused, reconfigured, and recomposed, into one or more subsequent artificial intelligence models
US10255282B2 (en) * 2016-03-04 2019-04-09 Microsoft Technology Licensing, Llc Determining key concepts in documents based on a universal concept graph
US10891295B2 (en) * 2017-06-04 2021-01-12 Apple Inc. Methods and systems using linear expressions for machine learning models to rank search results
US20190108486A1 (en) * 2017-10-05 2019-04-11 Microsoft Technology Licensing, Llc System and method for intelligent and automatic electronic communication support and routing
US11521067B2 (en) * 2018-11-30 2022-12-06 International Business Machines Corporation Decentralized distributed deep learning

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20170169358A1 (en) * 2015-12-09 2017-06-15 Samsung Electronics Co., Ltd. In-storage computing apparatus and method for decentralized machine learning
JP2018036779A (ja) * 2016-08-30 2018-03-08 株式会社東芝 電子装置、方法及び情報処理システム
US20180253818A1 (en) * 2017-03-03 2018-09-06 International Business Machines Corporation Deep learning via dynamic root solvers
US20180322386A1 (en) * 2017-05-05 2018-11-08 Intel Corporation Fine-grain compute communication execution for deep learning frameworks
CN108304918A (zh) * 2018-01-18 2018-07-20 中兴飞流信息科技有限公司 一种数据并行的深度学习的参数交换方法和系统

Also Published As

Publication number Publication date
CN113168569A (zh) 2021-07-23
JP7308003B2 (ja) 2023-07-13
JP2022511716A (ja) 2022-02-01
US20200175370A1 (en) 2020-06-04
US11521067B2 (en) 2022-12-06
DE112019004076T5 (de) 2021-05-06
WO2020109891A1 (en) 2020-06-04
GB202105988D0 (en) 2021-06-09

Similar Documents

Publication Publication Date Title
GB2593070A (en) Decentralized distributed deep learning
US11132602B1 (en) Efficient online training for machine learning
CN108009016B (zh) 一种资源负载均衡控制方法及集群调度器
Cheng et al. Event-triggered leader-following tracking control for multivariable multi-agent systems
Hong et al. Distributed output regulation of leader–follower multi‐agent systems
US11531926B2 (en) Method and apparatus for generating machine learning model by using distributed computing framework
Wang et al. Quasi-synchronization for fractional-order delayed dynamical networks with heterogeneous nodes
Summers et al. Topology design for optimal network coherence
Zhang et al. Opinion evolution analysis for short-range and long-range Deffuant–Weisbuch models
Kong et al. Observer design for a class of nonlinear fractional-order systems with unknown input
JP2016540463A5 (zh)
Cai et al. Global synchronization of weighted cellular neural network with time-varying coupling delays
Nik et al. New ultimate bound sets and exponential finite-time synchronization for the complex Lorenz system
US20220327428A1 (en) Executing Machine-Learning Models
Xu et al. Adaptive synchronization of stochastic time-varying delay dynamical networks with complex-variable systems
He et al. Robust exponential synchronization for neutral complex networks with discrete and distributed time‐varying delays: A descriptor model transformation method
Kubo et al. Multiple targets enclosure by robotic swarm
Shayeghi et al. A multi objective vector evaluated improved honey bee mating optimization for optimal and robust design of power system stabilizers
Mwaffo et al. Collective dynamics in the Vicsek and vectorial network models beyond uniform additive noise
KR20200010631A (ko) 클라우드 기반의 인공지능 연산 서비스 방법 및 이를 수행하는 장치
Raja et al. Dynamic analysis of discrete-time BAM neural networks with stochastic perturbations and impulses
EP3756095A1 (en) Workload modeling for cloud systems
Mera-Gómez et al. A debt-aware learning approach for resource adaptations in cloud elasticity management
WO2018135515A1 (ja) 情報処理装置、ニューラルネットワークの設計方法及び記録媒体
Fattahi et al. Distributed consensus of multi-agent systems with fault in transmission of control input and time-varying delays

Legal Events

Date Code Title Description
WAP Application withdrawn, taken to be withdrawn or refused ** after publication under section 16(1)