CN110768841A - Acceleration distributed online optimization method based on condition gradient - Google Patents

Acceleration distributed online optimization method based on condition gradient Download PDF

Info

Publication number
CN110768841A
CN110768841A CN201911045411.1A CN201911045411A CN110768841A CN 110768841 A CN110768841 A CN 110768841A CN 201911045411 A CN201911045411 A CN 201911045411A CN 110768841 A CN110768841 A CN 110768841A
Authority
CN
China
Prior art keywords
individual
optimization
local
information
gradient
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201911045411.1A
Other languages
Chinese (zh)
Inventor
申修宇
李德权
董翘
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Anhui University of Science and Technology
Original Assignee
Anhui University of Science and Technology
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Anhui University of Science and Technology filed Critical Anhui University of Science and Technology
Priority to CN201911045411.1A priority Critical patent/CN110768841A/en
Publication of CN110768841A publication Critical patent/CN110768841A/en
Priority to LU102143A priority patent/LU102143B1/en
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/08—Configuration management of networks or network elements
    • H04L41/0803—Configuration setting
    • H04L41/0823—Configuration setting characterised by the purposes of a change of settings, e.g. optimising configuration for enhancing reliability
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/04—Forecasting or optimisation specially adapted for administrative or management purposes, e.g. linear programming or "cutting stock problem"
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/142—Network analysis or design using statistical or mathematical methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/14—Network analysis or design
    • H04L41/145—Network analysis or design involving simulating, designing, planning or modelling of a network
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
    • H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Business, Economics & Management (AREA)
  • Mathematical Analysis (AREA)
  • Pure & Applied Mathematics (AREA)
  • Probability & Statistics with Applications (AREA)
  • Mathematical Physics (AREA)
  • Mathematical Optimization (AREA)
  • Human Resources & Organizations (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Algebra (AREA)
  • Game Theory and Decision Science (AREA)
  • General Business, Economics & Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Marketing (AREA)
  • Operations Research (AREA)
  • Quality & Reliability (AREA)
  • Tourism & Hospitality (AREA)
  • Development Economics (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

Based on the condition gradient, a method for accelerating the optimization of the distributed online condition gradient is provided, and the problem of high time complexity of a distributed online optimization algorithm can be effectively solved. The method decomposes the network optimization objective function into the sum of local objective functions of each node or individual, each individual only knows the objective function of the individual, and the optimization problem is cooperatively solved through information transfer between the individual and adjacent individuals. According to the method, a regularization term is added into local cost information of each individual, a new time-varying cost function is constructed, and the defect that a traditional conditional gradient algorithm is insensitive to the gradient size is overcome. The method also replaces projection operation with local linear optimization setting, so that the convergence speed of the algorithm is doubled. Finally, various tasks are verified through experiments, and the result shows that the method runs well in practical application and has certain superiority compared with the existing optimization method.

Description

Acceleration distributed online optimization method based on condition gradient
Technical Field
The invention relates to an accelerated distributed online optimization method based on condition gradients, and belongs to the field of machine learning.
Background
Distributed convex optimization has been of great interest to researchers in many fields. The classical problems of distributed tracking, estimation and detection are also optimization problems in nature. The distributed optimization problem is mainly to perform global optimization tasks assigned to each node in the network. Since each node has limited resources or partial information about the task, the nodes cooperate to perform data collection and update local estimates by sharing the collected information. Distributed optimization imposes a lower computational burden on the nodes and the network system remains robust even if the nodes experience local failures, so it can effectively overcome the deficiencies in a single information processing unit in a centralized scenario.
Distributed optimization has been widely applied in the case of time-invariant cost functions. However, in practical applications, distributed network systems are often in a dynamic and uncertain environment. For example, consider the problem of tracking moving objects, whose purpose is to track the position, velocity and acceleration of the object. These problems have been the main focus of online learning in the field of machine learning. Therefore, the online optimization and the distributed optimization are combined, the uncertainty of the multi-agent network system is represented by any variable cost function, and the dynamic data flow of the network nodes can be effectively processed in real time.
With the rapid development of distributed online optimization, many conventional optimization algorithms have been extended into distributed online cases. In recent years, traditional optimization algorithms such as a gradient descent method and a dual-mode averaging method are widely applied to distributed online optimization. The conditional gradient method (also called Frank-Wolfe, FW) is essentially a first-order optimization method, and can theoretically achieve a convergence rate lower than that of other effective optimization algorithms, but a large number of variables exist in various variables at present. High dimensionality practical optimization problems, so using second order information or other super linear operations is practically infeasible. Furthermore, the FW method has proven to be a powerful tool for solving large-scale optimization problems, since it can effectively avoid key problems such as the difficulty of computing orthogonal projections in a first-order optimization method. Therefore, a local linear optimization step is introduced into the original condition gradient algorithm, an accelerated distributed online condition gradient algorithm is provided, and the condition gradient online optimization algorithm is expanded to a distributed condition.
Disclosure of Invention
The technical problem to be solved by the invention is as follows: an acceleration distributed online optimization method based on conditional gradient is provided, and aims to accelerate convergence of a model in a distributed network.
In order to solve the technical problems, the invention adopts the following technical scheme:
in the distributed online convex optimization setting, each node represents an individual, in each iteration, the individual generates decision information, and the individual independently submits the decision information and obtains a corresponding cost function. Each individual has a degree of importance in information communication relative to other individuals, and the information is more valuable by giving higher weight to the individual with high degree of importance through the form of weighted average, so that the error of the whole distributed system is reduced. A local linear optimization step is introduced into an original conditional gradient algorithm, so that the convergence of the whole network model is accelerated.
Drawings
FIG. 1 is a convergence diagram of the method of the present invention at L1 for regularizing a logistic regression model.
FIG. 2 is a convergence diagram of the method of the present invention at L2 for regularizing the logistic regression model.
Detailed Description
The invention solves the problem of distributed optimization on the connected undirected network, and avoids the defect of a single information processing unit in a centralized scene, which causes the overhigh communication cost of the central node.
The method comprises the following specific steps:
step 1: revealing a loss function ft(t)＝fi,t(t)
Step 3: calculating the sub-gradient of the individual-generated information, git∈fi,t(xi,t)
For each individual:
Figure BDA0002254004240000021
in a distributed network, the passing of information by the brokers is performed by weighted averaging (third line in Step 4) to ensure that the information of the important brokers is fully utilized. Our method also introduces a local linear optimization setup
Figure BDA0002254004240000022
The objective is to accelerate the convergence of the entire network model ρ is a parameter for the local linear optimization setup α is the learning rate.
The invention is further described below with reference to the accompanying drawings.
FIG. 1 is a convergence diagram of the method of the present invention at L1 for regularizing a logistic regression model. Consider an online distributed learning environment: our goal is to solve the L1 regularized logistic regression problem, with numerical results as shown in fig. 1 for the synthetic dataset. It can be seen that the accelerated distributed online conditional gradient algorithm is superior to other algorithms. Fig. 1 also shows that the convergence rate of the algorithm is significantly faster at the beginning than other algorithms.
FIG. 2 is a convergence diagram of the method of the present invention at L2 for regularizing the logistic regression model. Experiments were performed on the actual data set with satisfactory results. As can be seen from fig. 2, the algorithm proposed herein achieves an acceleration effect. As can be seen from fig. 2, the loss of the algorithm reaches a small level quickly, and the performance is better than that of other algorithms, which may be more suitable for practical application.

Claims (3)

1. An accelerated distributed online optimization method based on conditional gradients is characterized in that individuals in a distributed network independently submit local information and then obtain a local cost function. The individuals are communicated with each other by a weighted average method. The next iteration direction is found by a local linear optimization step after the individual communication.
2. Individuals in the distributed network of claim 1 independently submit local information and then obtain a local cost function, wherein: in the distributed online convex optimization setting, each node represents an individual, in each iteration, the individual generates decision information, and the individual independently submits the decision information and obtains a corresponding cost function.
3. The method of claim 1 wherein the individuals communicate with each other by weighted averaging, wherein: each individual has a degree of importance in information communication relative to other individuals, and the information is more valuable by giving higher weight to the individual with high degree of importance through the form of weighted average, so that the error of the whole distributed system is reduced.
CN201911045411.1A 2019-10-30 2019-10-30 Acceleration distributed online optimization method based on condition gradient Pending CN110768841A (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201911045411.1A CN110768841A (en) 2019-10-30 2019-10-30 Acceleration distributed online optimization method based on condition gradient
LU102143A LU102143B1 (en) 2019-10-30 2020-10-16 Conditional gradient based method for accelerated distributed online optimization

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201911045411.1A CN110768841A (en) 2019-10-30 2019-10-30 Acceleration distributed online optimization method based on condition gradient

Publications (1)

Publication Number Publication Date
CN110768841A true CN110768841A (en) 2020-02-07

Family

ID=69334538

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201911045411.1A Pending CN110768841A (en) 2019-10-30 2019-10-30 Acceleration distributed online optimization method based on condition gradient

Country Status (2)

Country Link
CN (1) CN110768841A (en)
LU (1) LU102143B1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111580962A (en) * 2020-04-29 2020-08-25 安徽理工大学 Distributed self-adaptive online learning method with weight attenuation

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109149568A (en) * 2018-09-10 2019-01-04 上海交通大学 A kind of interconnection micro-capacitance sensor and scheduling Price optimization method based on distributed agent

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109149568A (en) * 2018-09-10 2019-01-04 上海交通大学 A kind of interconnection micro-capacitance sensor and scheduling Price optimization method based on distributed agent

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
DAN GARBER，ELAD HAZAN: "A Linearly Convergent Conditional Gradient Algorithm with Applications to Online and Stochastic Optimization", 《ARXIV》 *
李德权，董翘，周跃进: "分布式在线条件梯度优化算法", 《计算机科学》 *
王俊雅: "分布式在线随机投影优化", 《阜阳师范学院学报(自然科学版)》 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111580962A (en) * 2020-04-29 2020-08-25 安徽理工大学 Distributed self-adaptive online learning method with weight attenuation

Also Published As

Publication number Publication date
LU102143B1 (en) 2021-04-16

Similar Documents

Publication Publication Date Title
Neglia et al. The role of network topology for distributed machine learning
CN102915347A (en) Distributed data stream clustering method and system
US11281232B2 (en) Systems and methods for multi-agent system control using consensus and saturation constraints
Yuan et al. Removing data heterogeneity influence enhances network topology dependence of decentralized sgd
Najafi et al. Outlier-Robust Multi-Aspect Streaming Tensor Completion and Factorization.
CN113298222A (en) Parameter updating method based on neural network and distributed training platform system
CN105205052A (en) Method and device for mining data
CN111178261A (en) Face detection acceleration method based on video coding technology
Boukhdhir et al. An improved MapReduce Design of Kmeans for clustering very large datasets
Li et al. Model-distributed dnn training for memory-constrained edge computing devices
CN110768841A (en) Acceleration distributed online optimization method based on condition gradient
CN111460368A (en) Parallel Bayesian optimization method
CN111681264A (en) Real-time multi-target tracking method for monitoring scene
Dong et al. A single-loop algorithm for decentralized bilevel optimization
Zhang et al. On the Communication Complexity of Decentralized Bilevel Optimization
CN113342313A (en) Method for asynchronously updating linear classification model parameters in Spark MLlib based on parameter server
Wu et al. Decentralized state feedback adaptive tracking for a class of stochastic nonlinear large-scale systems
Pan et al. GSP Distributed Deep Learning Used for the Monitoring System
Li et al. Chunk Dynamic Updating for Group Lasso with ODEs
Zhu et al. Efficient Gaussian Kernel Microcluster Real-Time Clustering Method for Industrial Internet of Things (IIoT) Streams
Yu et al. Finite time estimation and containment control of second order perturbed directed networks
CN102682100B (en) Task execution sequence optimization method based on teleoperation
Yoon et al. GDFed: Dynamic Federated Learning for Heterogenous Device Using Graph Neural Network
WO2022170654A1 (en) Data encryption learning method suitable for dynamic distributed internet of things system
Wattanakitrungroj et al. Memory-less unsupervised clustering for data streaming by versatile ellipsoidal function

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WD01 Invention patent application deemed withdrawn after publication

Application publication date: 20200207

WD01 Invention patent application deemed withdrawn after publication