KR102555268B1 - Asymmetric centralized training for distributed deep learning - Google Patents

Abstract

A parameter server-based asymmetric distributed learning technique is disclosed. A distributed learning system based on asymmetric communication according to an embodiment performs a weighted average of a global parameter updated based on a learning result calculated as a learning model is learned using local data and a local parameter of a worker to obtain a local a plurality of workers updating parameters; and a parameter server that transmits the updated global parameters based on the learning results calculated in the plurality of workers to the plurality of workers according to a predefined update rule through asymmetric communication.

Description

Parameter server-based asymmetric distributed learning technique {ASYMMETRIC CENTRALIZED TRAINING FOR DISTRIBUTED DEEP LEARNING}

The description below relates to an asymmetric distributed learning technique to solve the server bottleneck problem.

Existing parameter server distributed learning techniques have a problem that the parameter server becomes a bottleneck for learning as the number of workers and model parameters increases. This bottleneck problem with the parameter server is because the worker communicates symmetrically with the parameter server. Symmetric communication between the worker and the parameter server means a communication method in which the worker transmits learning results to the parameter server and receives the latest parameters from the parameter server.

Referring to FIG. 1, it is a diagram for explaining an existing parameter server-based distributed learning method. Figure 1(a) shows a synchronous distributed learning method, and Figure 1(b) shows an asynchronous distributed learning method. Regardless of FIGS. 1(a) and 1(b), each worker transmits the learning result to the parameter server (Send.) and waits for the parameter server to return the latest parameters (Wait. & Recv.).

Due to this symmetric communication, each worker finishes one learning step (Iter. t) and cannot immediately proceed to the next step (Iter. t+1) while waiting for the parameter server. This problem can become more serious as the number of parameters and the number of workers in the learning model increase. In order to confirm this problem of the prior art, analysis of the learning time of the workers may be performed while increasing the number of workers. 2 is an analysis of the time required for each learning step according to the number of workers. It can be seen that as the number of workers increases, the overhead due to symmetric communication increases.

It is possible to provide a new distributed learning method and system based on asymmetric communication to solve the problem by identifying that it is caused by symmetric communication, which is the cause of the parameter server bottleneck problem of existing parameter server-based distributed learning methods. there is.

A parameter update strategy can be provided that minimizes the loss of accuracy of the learning model that can be caused by asymmetric communication.

The asymmetric distributed learning system includes a plurality of workers that update local parameters by weighting averages of updated global parameters and local parameters of the workers based on a learning result calculated by learning a learning model using local data; and a parameter server that transmits the updated global parameters based on the learning results calculated in the plurality of workers to the plurality of workers according to a predefined update rule through asymmetric communication.

The plurality of workers may calculate a gradient as a learning model is trained using the local data, and may update a local parameter for each of the plurality of workers with the calculated gradient.

When the updated global parameter is transmitted from the parameter server, the plurality of workers may update the local parameter of the worker by performing a weighted average of the updated global parameter and the local parameter of the worker.

The plurality of workers may proceed with the next learning process for the learning when the updated global parameter is not transmitted from the parameter server.

The parameter server may collect gradients calculated by the plurality of workers and update a global parameter using the collected gradients.

The parameter server may search for a global parameter that minimizes a loss function in distributed learning using gradients collected for the plurality of workers, and transmit the searched global parameter to the plurality of workers every update period.

An asymmetric distributed learning method performed in a worker includes calculating a learning result by learning a learning model using local data; transmitting the calculated learning result to a parameter server; And receiving the updated global parameter from the parameter server based on a predefined parameter update rule through asymmetric communication, wherein in the receiving step, the updated global parameter and the local parameter of the worker are a weighted average ( weighted average) so that the local parameters of the worker can be updated.

The calculating may include calculating a gradient as the plurality of workers learn a learning model using local data in a first learning process, and updating a local parameter for each of the plurality of workers with the calculated gradient steps may be included.

The receiving step, when receiving that the updated global parameter is transmitted from the parameter server, updates the local parameter of the worker by performing a weighted average of the updated global parameter and the local parameter of the worker, and the parameter server When the updated global parameter is not transmitted from , proceeding with a second learning process may be included.

An asymmetric distributed learning method performed by a parameter server includes: collecting learning results calculated as learning a learning model using local data in a plurality of workers; updating global parameters using the collected learning results; and transmitting the updated global parameter to a plurality of workers through asymmetric communication based on a preset update rule, wherein in the transmitting step, the updated global parameter transmitted to the plurality of workers and the plurality of workers The local parameters of the plurality of workers may be updated by performing a weighted average on the local parameters of the workers.

The updating may include collecting gradients calculated by the plurality of workers and using the collected gradients to search for a global parameter that minimizes a loss function in distributed learning.

The transmitting may include transmitting a searched global parameter to the plurality of workers every update period in order to minimize a loss function in distributed learning using the collected gradients.

Through asymmetric communication-based learning, workers can perform learning without waiting for the parameter server regardless of whether the parameter server is a bottleneck or not. Accordingly, the learning model can be trained with high accuracy and within a short time.

By updating the global parameters based on the parameter update rule, it is possible to prevent performance degradation by minimizing a learning model loss that may occur due to asymmetric computation.

1 is a diagram for explaining an existing parameter server-based distributed learning method.
Figure 2 is a graph analyzing the time required for each learning step according to the number of workers.
3 is a diagram for explaining a learning operation based on asymmetric communication in a distributed learning system according to an embodiment.
4 is an example of an algorithm for explaining an operation of a worker in a distributed learning system according to an embodiment.
5 is an example of an algorithm for explaining the operation of a parameter server in a distributed learning system according to an embodiment.
6 is a diagram for explaining the configuration of a distributed learning system according to an embodiment.
7 and 8 are diagrams for explaining a learning method based on asymmetric communication in a distributed learning system according to an embodiment.

Hereinafter, an embodiment will be described in detail with reference to the accompanying drawings.

3 is a diagram for explaining a learning operation based on asymmetric communication in a distributed learning system according to an embodiment.

Whether synchronously or asynchronously, each worker sends learning results to the parameter server and waits for the parameter server to return updated parameters. At this time, it is important to reduce the idle time of the worker regardless of whether it is synchronous or asynchronous. Accordingly, a learning operation based on asymmetric communication in which learning is performed without waiting by not waiting for reception of an updated parameter from a parameter server through a learning operation based on asymmetric communication between a worker and a parameter will be described.

As shown in FIG. 3, asymmetric communication-based learning is a method in which workers transmit the learning results calculated in each learning process to the parameter server and proceed with the next learning process without waiting for the parameter server. Compared to the synchronous communication-based learning of FIG. 1(a) and the asynchronous communication-based learning of FIG. 1(b), the asymmetric communication-based learning of FIG. (wait-free) learning can proceed.

In addition, to ensure that the difference between the model parameters of each worker and the global parameters of the parameter server does not widen beyond a certain level, we propose a parameter server-driven update strategy in which the parameter server periodically updates the model parameters of the worker. In order to accurately perform server-driven parameter update for each worker, a fundamental problem of distributed deep learning research can be defined, and a rule for updating model parameters of a worker can be defined based on the defined problem.

를 찾는 것으로, 수학식 1과 같이 정의될 수 있다. The definition of a distributed learning problem is a global parameter that minimizes the loss function.

By finding , it can be defined as in Equation 1.

Equation 1:

는 글로벌 파라미터,

는 워커 i의 로컬 파라미터,

는 워커 i가 가지고 있는 데이터,

는 상수이다. At this time, n is the number of workers,

is a global parameter,

is the local parameter of worker i,

is the data that worker i has,

is a constant.

)의 입장에 대해 경사 하강법(gradient desent)을 적용하여 수학식 2를 획득할 수 있다. 수학식 2의 문제 정의를 바탕으로 워커에 대한 파라미터 업데이트 룰(rule)이 정의될 수 있다. Equation 1 is applied to each worker (for example,

Equation 2 can be obtained by applying gradient descent to the position of ). Based on the problem definition of Equation 2, a parameter update rule for workers may be defined.

Equation 2:

라고 하고,

라고 하면, 최종적으로 워커 i에 대한 업데이트 룰을 획득할 수 있다.

say,

, the update rule for worker i can finally be obtained.

Equation 3:

는 워커의 수가 증가할수록

이 감소하도록 설정될 수 있으며, k는 상수, n은 워커의 수이다. Equation 3 means that the worker can update the local parameter by taking a weighted average between the local parameter and the global parameter. For example, in Equation 3, the weight

as the number of workers increases

can be set to decrease, where k is a constant and n is the number of workers.

The parameter server may define a periodic update rule (strategy) that transmits global parameters to workers only after the parameter server collects the gradients of all workers in order to update the local parameters of the workers. For example, if the update period is large, learning is performed using local data of each worker, whereas if the update period is small, local parameters may be updated more frequently by global parameters.

4 is an example of an algorithm for explaining an operation of a worker in a distributed learning system according to an embodiment.

The worker can compute the gradient (line 2). The worker can update its local parameters with the computed gradient (line 3). The worker can send the computed gradient to the parameter server (line 4). When updated global parameters are transmitted from the parameter server, the worker may update the local parameters of the worker by performing a weighted average of the updated global parameters and the local parameters of the worker, and the updated global parameters are transmitted from the parameter server. If not, you can proceed with the following learning (lines 5-7). In other words, the worker can proceed to the next iteration without waiting for the parameter server. Through this asymmetric learning, the worker can train the learning model regardless of whether the parameter server is a bottleneck or not.

5 is an example of an algorithm for explaining the operation of a parameter server in a distributed learning system according to an embodiment.

마다 복수의 워커에게 업데이트된 글로벌 파라미터를 전송할 수 있다(4-6 줄). The parameter server can collect gradients computed from multiple workers (line 2). The parameter server can update the global parameters using the collected gradients (line 3). Parameter server update cycle

Each worker can send updated global parameters to multiple workers (lines 4-6).

In this way, since the worker and the parameter server perform asymmetric communication, the parameter server does not need to send updated parameters from the worker each time. The parameter server can collect the gradients of all workers and update the global parameters at once.

6 is a diagram for explaining the configuration of a distributed learning system in a distributed learning system according to an embodiment.

The distributed learning system 100 may include a parameter server 610 and a plurality of workers 620 . A distributed learning method based on asymmetric communication between the parameter server 610 and a plurality of workers (hereinafter referred to as 'workers') 620 will be described with reference to FIGS. 7 and 8 .

7 is for explaining the case where the worker receives updated global parameters from the parameter server, and FIG. 8 is for explaining the case where the worker does not receive the updated global parameters from the parameter server.

Referring to FIGS. 7 and 8 , a worker 620 may learn a learning model using local data (701). As the worker 620 trains the learning model using local data, a learning result may be calculated. The worker 620 may calculate a gradient as the learning model is trained using local data. The worker 620 may obtain a calculated learning result through learning (702). The worker may transmit the calculated learning result to the parameter server 610 (703).

The parameter server 610 may receive the calculated learning result (704). The parameter server 610 may update the global parameters based on a predefined update rule (705). At this time, the operation of the worker varies depending on whether or not the parameter server 610 transmits the global parameter to the worker.

In other words, the parameter server 610 may update the global parameters based on predefined rules. The parameter server 610 transmits updated global parameters at every update period. At this time, when the update period is reached, the updated global parameters may be transmitted from the parameter server 610 to the worker 620 as shown in FIG. 7 (706). Worker 620 may receive updated global parameters (707). The worker 620 may update the local parameter by performing a weighted average of the updated global parameter and the local parameter (708).

If the update period has not reached, referring to FIG. 8 , the updated global parameters may not be transmitted from the parameter server 610 to the worker 620 . If the global parameter is not transmitted from the parameter server 610 to the worker 620, the worker 620 may proceed with the next learning (801). For example, when the worker 620 is in progress of first learning, it may proceed with second learning.

In order to verify the effect of an asymmetric distributed learning operation in a distributed system according to an embodiment, existing distributed deep learning methods can be compared in terms of model accuracy and learning time.

Table 1:

Table 1 shows the comparison results of accuracy and learning time. The asymmetric distributed learning proposed in the example shows the best results in both the learning model (eg, ResNet-50, VGG-16 two models) and the data set (eg, CIFAR-10, ImageNet-1K) You can check.

In particular, the asymmetric communication-based distributed learning proposed in the embodiment can solve the learning performance degradation problem of the existing parameter server-based distributed deep learning method, and achieves high accuracy in the fastest time compared to existing methods. You can check that the results are visible. In other words, it can be confirmed that the distributed learning based on asymmetric communication proposed in the embodiment is superior to conventional techniques.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. The device can be commanded. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (12)

In a distributed learning system based on asymmetric communication,
a plurality of workers that update local parameters by performing a weighted average of updated global parameters and local parameters of the workers based on a learning result calculated by learning a learning model using local data; and
A parameter server that transmits updated global parameters based on the learning results calculated in the plurality of workers to a plurality of workers according to a predefined update rule through asymmetric communication
including,
The parameter server,
Collect gradients calculated from the plurality of workers, update global parameters using the collected gradients, and search for global parameters that minimize a loss function in distributed learning using the gradients collected for the plurality of workers , Transmitting the searched global parameter to the plurality of workers according to an update cycle,
The plurality of workers,
Depending on whether or not the updated global parameters are transmitted from the parameter server, the worker's local parameters are updated or the next learning process for learning is performed.
Distributed learning system. According to claim 1,
The plurality of workers,
Calculating a gradient as the learning model is trained using the local data, and updating local parameters for each of the plurality of workers with the calculated gradient
Distributed learning system, characterized in that. According to claim 1,
The plurality of workers,
When the updated global parameter is transmitted from the parameter server, updating the local parameter of the worker by performing a weighted average of the updated global parameter and the local parameter of the worker
Distributed learning system, characterized in that. According to claim 1,
The plurality of workers,
If the updated global parameter is not transmitted from the parameter server, proceeding with the next learning process for the learning
Distributed learning system, characterized in that. delete delete In the distributed learning method based on asymmetric communication performed in a worker,
calculating a learning result by learning a learning model using local data;
transmitting the calculated learning result to a parameter server; and
Receiving updated global parameters from the parameter server based on a predefined parameter update rule through asymmetric communication
including,
In the receiving step, the updated global parameter and the local parameter of the worker are weighted average according to whether the updated global parameter is received from the parameter server, so that the local parameter of the worker is updated or the next learning for learning including the process
In the parameter server, gradients calculated from a plurality of workers are collected, global parameters are updated using the collected gradients, and a global loss function is minimized in distributed learning using the gradients collected for the plurality of workers. Parameters are searched, and the searched global parameters are transmitted to the plurality of workers according to an update cycle.
Distributed learning method. According to claim 7,
The calculation step is
Calculating a gradient as the plurality of workers learn a learning model using local data in a first learning process, and updating a local parameter for each of the plurality of workers with the calculated gradient.
Distributed learning method comprising a. According to claim 7,
In the receiving step,
When receiving the transmission of the updated global parameter from the parameter server, the updated global parameter and the local parameter of the worker are weighted average to update the local parameter of the worker, and the global parameter updated from the parameter server If not transmitted, proceeding with the second learning process
Distributed learning method comprising a. In the asymmetric distributed learning method performed by the parameter server,
Collecting learning results calculated by learning a learning model using local data in a plurality of workers;
updating global parameters using the collected learning results; and
Transmitting the updated global parameters to a plurality of workers through asymmetric communication based on a preset update rule.
including,
The updating step is
Collecting gradients calculated by the plurality of workers and using the collected gradients to search for global parameters that minimize a loss function in distributed learning.
including,
The sending step is
Transmitting the searched global parameters to the plurality of workers according to an update cycle in order to minimize a loss function in distributed learning using the collected gradients.
including,
In the plurality of workers, the updated global parameters transmitted to the plurality of workers and the local parameters of the plurality of workers are weighted average according to whether or not the updated global parameters are transmitted from the parameter server, and the plurality of workers When the local parameters of are updated or the next training pass for learning is in progress.
Distributed learning method.
delete delete

Images