KR20220009682A - Method and system for distributed machine learning - Google Patents

Abstract

The present invention relates to a method and system for distributed machine learning. The method comprises the operations of: transmitting, by a central server, a random seed; receiving, by a plurality of learning machines, the random seed and generating a random matrix based on the received random seed; generating, by the plurality of learning machines, local gradient parameters based on training data assigned thereto; generating, by the plurality of learning machines, compression information by compressing the generated local gradient parameters and setting parameters using the random matrix; transmitting, by the plurality of learning machines, the compression information to the central server; generating, by the central server, average compression gradient parameters by collecting the compression information received from the plurality of learning machines; transmitting, by the central server, the average compression gradient parameters to the plurality of learning machines; decompressing, by the plurality of learning machines, the received average compression gradient parameters by using the transpose of the random matrix to obtain average gradient parameters; and updating, the plurality of learning machines, the parameters of an artificial intelligence model based on the obtained average gradient parameters. Therefore, the time required to learn a very large amount of training data can be shortened.

Description

Distributed machine learning method and system {METHOD AND SYSTEM FOR DISTRIBUTED MACHINE LEARNING}

Various embodiments relate to a distributed machine learning method and system.

Artificial intelligence can be defined as a technology that performs intelligent actions such as human learning ability, reasoning ability, perceptual ability, natural language understanding ability, etc. in a computing device.

As the performance of the computing device and the level of application development used in the computing device gradually improve, the level of artificial intelligence implemented in the computing device is also gradually improved.

Recently, artificial intelligence has been implemented using a neural network implemented similar to a human brain and a technique for performing reinforcement learning using the structure (eg, a deep learning learning technique).

The performance of artificial intelligence can be improved through learning, and various learning data are required for learning of artificial intelligence.

On the other hand, there is a limitation in that it takes a lot of time to learn when developing a deep learning model using large data. In order to solve this problem, distributed machine learning has recently been attracting attention.

In the distributed processing of the deep learning learning process, a data parallelism technique and a model parallelism technique are used.

The data parallel processing technique refers to a method in which multiple computers divide the input data set to be learned and perform learning, and the model parallel processing technique divides the deep learning model and divides the multiple computers into the divided deep learning models. It means the method of carrying out each learning.

In particular, since the data parallel processing technique is a method of learning by using some training data in each of a plurality of learning machines for the entire training data, the data is stored in an external storage device connected to the network, and the network learning must be carried out through

In this case, whenever learning is in progress, each of the learning machines must frequently access an external storage device to proceed with learning. .

In addition, in the case of very large data, since the entire data cannot be learned in a specific distributed environment, there is a limit in which the learning efficiency is reduced.

The present invention has been devised to solve the above problems, and an object of the present invention is to provide a distributed machine learning method and system for performing more efficient learning by reducing storage and distribution required in distributed machine learning.

The technical problems to be achieved in this document are not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the description below. There will be.

According to various embodiments of the present disclosure, a distributed machine learning method of an artificial intelligence neural network includes: a central server transmitting a random seed; receiving, by a plurality of learning machines, the random seed, and generating a random matrix based on the received random seed; generating, by the plurality of learning machines, a local gradient parameter based on training data assigned to them; generating compressed information by compressing, by the plurality of learning machines, the generated local gradient parameter and setting parameter using the random matrix; sending, by the plurality of learning machines, the compressed information to the central server; generating, by the central server, the compressed information received from the plurality of learning machines to generate an average compression gradient parameter; sending, by the central server, the average compressed gradient parameter to the plurality of learning machines; decompressing, by the plurality of learning machines, the received average compressed gradient parameter using the transpose of the random matrix to obtain an average gradient parameter; and updating, by the plurality of learning machines, the parameters of the artificial intelligence model based on the obtained average gradient parameters.

The method further includes: updating, by the plurality of learning machines, the setting parameter using the average gradient parameter.

In addition, the operation of the plurality of learning machines generating the compressed information includes multiplying a difference between the local gradient parameter and the set parameter by the random matrix to generate the compressed information.

In addition, the operation of the plurality of learning machines to obtain the average gradient parameter includes: obtaining the average gradient parameter by adding the setting parameter to a value obtained by multiplying the average compressed gradient parameter and the transform of the random matrix .

In addition, the computer-readable storage medium according to another embodiment of the present invention for solving the above technical problem, when executed on a computer, a program for performing the distributed machine learning method may be recorded.

In addition, in a distributed machine learning system according to another embodiment of the present invention for solving the above technical problem, a central server transmits a random seed, a plurality of learning machines receive the random seed, and the received random generate a random matrix based on a seed, the plurality of learning machines generate local gradient parameters based on training data assigned to them, and the plurality of learning machines use the generated local gradient parameters and the setting parameters as the random Compressed information is generated by compression using a matrix, the plurality of learning machines transmits the compressed information to the central server, and the central server aggregates the compressed information received from the plurality of learning machines to obtain an average compression gradient. generating parameters, the central server sends the average compression gradient parameter to the plurality of learning machines, and the plurality of learning machines decompresses the received average compression gradient parameter using the transpose of the random matrix, obtain an average gradient parameter, and the plurality of learning machines update the parameter of the artificial intelligence model based on the obtained average gradient parameter.

Also, the plurality of learning machines use the average gradient parameter to update the setting parameter.

In addition, the plurality of learning machines generate the compressed information by multiplying the difference between the local gradient parameter and the set parameter by the random matrix.

In addition, the plurality of learning machines obtain the average gradient parameter by adding the set parameter to a value obtained by multiplying the average compressed gradient parameter and the transposition of the random matrix.

The present invention has the effect of shortening the learning time of the super-capacity learning data and reducing the communication cost by reducing the network communication amount.

In addition, the present invention has the effect of improving the learning performance by reducing the storage and distribution required in distributed machine learning.

Effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. will be.

1 is a schematic diagram of a system 1000 for distributed learning of large amounts of learning data according to an embodiment of the present invention.
2 is a flowchart of a method for updating parameters in distributed machine learning according to an embodiment of the present invention.
3 to 5 are exemplary diagrams for explaining a process of updating parameters in distributed machine learning according to an embodiment of the present invention.

Hereinafter, various embodiments will be described in detail with reference to the accompanying drawings.

Regardless of the reference numerals, the same or similar components are assigned the same reference numerals, and overlapping descriptions thereof may be omitted.

The suffix 'module' or 'unit' for components used in the following description is given or mixed in consideration of only the ease of writing the specification, and does not have a meaning or role distinct from each other by itself. In addition, 'module' or 'unit' refers to software or hardware components such as field programmable gate array (FPGA) or application specific integrated circuit (ASIC), but is not limited to software or hardware. A 'unit' or 'module' may be configured to reside on an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, 'part' or 'module' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, may include procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functionality provided in one component, 'unit' or 'module' may be combined into a smaller number of components and 'unit' or 'module' or additional components and 'unit' or 'module' can be further separated.

Steps of a method or algorithm described in connection with some embodiments of the present invention may be directly implemented in hardware executed by a processor, a software module, or a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of recording medium known in the art. An exemplary recording medium is coupled to the processor, the processor capable of reading information from, and writing information to, the storage medium. Alternatively, the recording medium may be integral with the processor. The processor and recording medium may reside within an application specific integrated circuit (ASIC). The ASIC may reside within the user terminal.

Terms including an ordinal number such as 1st, 2nd, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being 'connected' or 'connected' to another component, it is understood that it may be directly connected or connected to the other component, but other components may exist in between. it should be On the other hand, when it is mentioned that a certain element is 'directly connected' or 'directly connected' to another element, it should be understood that the other element does not exist in the middle.

First, the terms used in this specification will be briefly described.

Artificial intelligence is a computer system or device equipped with human intelligence, and may refer to artificially implemented human intelligence in a machine or the like. Artificial intelligence also refers to the field of science that studies the methodology or feasibility of creating intelligence.

Artificial neural network is a model or algorithm that implements artificial intelligence. It is a statistical learning algorithm modeled by simulating the neural network of biology in machine learning. Therefore, it can be called a model or learning algorithm with problem-solving ability.

An artificial neural network may include an input layer, an output layer, and one or more hidden layers. Each layer of the artificial neural network includes a plurality of nodes corresponding to neurons of the neural network, and a node in one layer of the artificial neural network and a node in another layer may be connected by a synapse. In one embodiment, an artificial neural network in which all nodes of each layer and all nodes of the next layer are synaptically connected may be referred to as a fully connected artificial neural network.

In the artificial neural network, each node may receive input signals input through a synapse and generate an output value based on an activation function for weights and biases for each input signal.

A deep neural network may collectively refer to an artificial neural network including a plurality of hidden layers between an input layer and an output layer. A deep neural network can model complex nonlinear relationships and can have various structures depending on its purpose. For example, as a deep neural network structure, there may be a convolutional neural network, a recurrent neural network (RNN), a long short term memory (LSTM), and the like.

Convolutional neural networks can be effective in classifying and identifying images and videos by learning by identifying features of structured spatial data such as images, videos, and strings. The recurrent neural network has a cyclic structure inside, so the learning in the past time is multiplied by the weight and reflected in the current learning. The current output result is affected by the output result in the past time, and the hidden layer is a kind of memory perform the function Therefore, it can be effective in learning sequential data to perform classification or prediction.

As a method of learning the characteristics and patterns of data using a large amount of data and predicting the result, there are a gradient descent algorithm and a back propagation algorithm.

Gradient descent algorithm is an optimization algorithm for finding first-order approximations. It is an algorithm that finds the gradient of a function, moves it to a lower gradient, and repeats it until it reaches an extreme value. This may be an algorithm that obtains error information between the answer calculated by the neural network and the correct answer, and adjusts individual weights or parameters constituting the neural network by giving feedback to reduce the error. In this case, the error information used in the backpropagation algorithm can be obtained by the gradient descent algorithm.

Hereinafter, a method of updating artificial neural network parameters by learning a large amount of data using a gradient descent algorithm will be described in more detail.

1 is a schematic diagram of a system 1000 for distributed learning based on data parallel processing on large-capacity learning data according to an embodiment of the present invention.

Referring to FIG. 1 , a system 1000 for distributed learning of large-capacity learning data may include a central server 100 and a plurality of worker machines 200 .

The central server 100 may be a server shared by a plurality of learning machines 200 or a similar device. The plurality of learning machines 200 may be electronic devices that are communicatively connected to the central server 100 through a network, and have the same artificial intelligence neural network model to perform and process distributed learning.

On the other hand, in FIG. 1 , the central server 100 and the learning machine 200 are shown as separate configurations, but this is only an example, and the central server 100 is configured in each of the plurality of learning machines 200 . may be included.

The plurality of learning machines 200 may include a communication unit and a processor for data transmission/reception with the central server 100 .

For example, the plurality of learning machines 200 generate a local gradient parameter learned based on the training data allocated so as not to overlap each, and compress the generated local gradient parameter to form the central server 100 ) can be passed as

The central server 100 may collect and average each local gradient parameter transmitted from the plurality of learning machines 200 , and transmit the averaged information so that the plurality of learning machines 200 share with each other. That is, the plurality of learning machines 200 may use the information transmitted from the central server 100 to update the parameters of the artificial intelligence neural network model that each has and wants to learn.

The distributed learning system 1000 presented in this specification reduces the variance caused by each of the plurality of learning machines 200 using only some training data without using the entire training data in a distributed learning environment by using a setting parameter, The plurality of learning machines 200 compress and transmit the gradient parameters to the central server 100, thereby reducing the amount of transmitted data and collecting the setting parameters from the plurality of learning machines 200 Storage of the central server 100 (storage) can be reduced.

The plurality of learning machines 200 are mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation systems, and slate PCs. , including tablet PCs, ultrabooks, wearable devices, such as smart watch, smart glass, head mounted display (HMD), etc. It will be apparent that information and communication devices and multimedia devices and their application devices may be used. That is, the plurality of learning machines 200 may be electronic devices capable of learning large-capacity learning data more efficiently.

On the other hand, the components 100 and 200 shown in FIG. 1 are only an example of the components constituting the system 1000 for distributed learning of large-capacity learning data, and therefore, to implement the distributed learning described in this specification It will be apparent that additional components may be added for this purpose.

2 is a flowchart of a method of updating a parameter of an artificial neural network model in distributed machine learning according to an embodiment of the present invention. Hereinafter, at least some operations of FIG. 2 will be described with reference to FIGS. 3 to 5 . 3 to 5 are exemplary diagrams for explaining a process of updating parameters in distributed machine learning according to an embodiment of the present invention.

Meanwhile, although each operation in FIG. 2 may be sequentially performed, it is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

Referring to FIG. 2 , the central server 100 may provide the same random seed to each of the plurality of learning machines 200 ( S210 ).

The plurality of learning machines 200 may generate a random matrix R based on the random seed received through operation S210 (S220). The plurality of learning machines 200 may generate the same random matrix R based on the same random number sequence and the same random seed.

)를 생성할 수 있다(S231). 복수의 학습 머신(200)은 랜덤 매트릭스(R)을 이용하여 로컬 그래디언트 파라미터(

)를 압축하여 데이터 크기를 줄일 수 있다.A plurality of learning machines 200 based on the training data assigned to each of the local gradient parameters (

) can be generated (S231). A plurality of learning machines 200 using a random matrix (R) local gradient parameters (

) to reduce the data size.

)를 생성할 수 있다. 이 경우, 복수의 학습 머신(200)에는 학습 데이터가 서로 중복되지 않도록 각각 할당될 수 있다.The plurality of learning machines 200 learn the training data assigned to each using the same artificial intelligence neural network model provided in each, and the local gradient parameter (

) can be created. In this case, each of the plurality of learning machines 200 may be allocated so that the learning data do not overlap with each other.

)와 설정 파라미터(

) 정보를 압축한 압축 정보를 생성할 수 있다(S232).2 and 3 , the plurality of learning machines 200 use the random matrix R generated through operation S220 to determine the local gradient parameter (

) and setting parameters (

) compressed information may be generated (S232).

For example, the compressed information may refer to information in which the amount of data is reduced so that the size or amount of the information is not increased even when the central server 100 collects parameter information of each of the plurality of learning machines 200 . . That is, each of the plurality of learning machines 200 may reduce storage required in distributed learning by generating compressed information. Compressed information may be generated by Equation 1 below.

)와 설정 파라미터(

)의 차이에 랜덤 매트릭스(R)를 곱하여 계산된 함수의 출력 값인 압축 정보를 획득할 수 있다.Referring to Equations 1 and 3, the plurality of learning machines 200 have local gradient parameters (

) and setting parameters (

) multiplied by the random matrix (R) to obtain compressed information that is an output value of the calculated function.

)를 이용하거나, 또는 각각 상이한 설정 파라미터(

)를 이용할 수 있다. 설정 파라미터(

)는 분산 학습에서의 분산을 감소시키기 위해 이용되는 임의의 설정 파라미터를 의미할 수 있다.In one embodiment, the learning machine 200 has all the same setting parameters (

), or using different setting parameters (

) can be used. setting parameters (

) may mean any setting parameter used to reduce variance in distributed learning.

The plurality of learning machines 200 may transmit the compressed information obtained through operation S232 to the central server 100 (S233). In this case, the compressed information transmitted from the plurality of learning machines 200 may have a size acceptable to the central server 100 .

)를 생성할 수 있다(S234).The central server 100 collects the compression information received through operation S233, and the average compression gradient parameter (

) can be generated (S234).

)는 복수 개(예를 들어, n개)의 학습 머신(200)으로부터 전달된 압축 정보의 평균값을 의미하며, 평균 압축 그래디언트 파라미터(

)는 일반적으로 이용되는 평균값 수학식을 통해 산출될 수 있다. 중앙 서버(100)는 다음 수학식 2에 따라 평균 압축 그래디언트 파라미터(

)를 계산할 수 있다.Average compression gradient parameter (

) means an average value of compressed information transmitted from a plurality of (eg, n) learning machines 200, and an average compression gradient parameter (

) can be calculated through a commonly used average value equation. The central server 100 calculates the average compression gradient parameter (

) can be calculated.

)를 산출할 수 있다.Referring to Equation 2, the central server 100 divides the plurality of pieces of compressed information received from the plurality of learning machines 200 by the total number of the learning machines 200 and divides the average compression gradient parameter (

) can be calculated.

)를 복수의 학습 머신(200)으로 전송할 수 있다(S235). 즉, 복수의 학습 머신(200)은 동일한 평균 압축 그래디언트 파라미터(

)를 공유할 수 있다.The central server 100 generates the average compression gradient parameter (

) may be transmitted to the plurality of learning machines 200 (S235). That is, a plurality of learning machines 200 have the same average compressed gradient parameter (

) can be shared.

)를 압축 해제하여(S236), 평균 그래디언트 파라미터(

)를 획득할 수 있다(S237).The plurality of learning machines 200 receive the average compressed gradient parameter (

) by decompressing (S236), the average gradient parameter (

) can be obtained (S237).

)를 이용하여 평균 압축 그래디언트 파라미터(

)를 압축 해제시킴으로써, 이전 크기를 가진 평균 그래디언트 파라미터(

)를 획득할 수 있다.A plurality of learning machines 200 transpose of the random matrix (R) (transpose,

) using the average compression gradient parameter (

), by decompressing the average gradient parameter (

) can be obtained.

)는 랜덤 매트릭스(R)의 전치행렬로서, 랜덤 매트릭스(R)의 행과 열을 바꾼 행렬을 의미할 수 있다. 평균 그래디언트 파라미터(

)는 다음 수학식 3에 의해 산출될 수 있다.Transpose of the random matrix (R) (transpose,

) is a transpose matrix of the random matrix R, and may mean a matrix in which rows and columns of the random matrix R are exchanged. average gradient parameter (

) can be calculated by Equation 3 below.

)에 랜덤 매트릭스(R)의 트랜스포즈(

)를 곱하여 압축 해제하고, 설정 파라미터(

)를 더하여 평균 그래디언트 파라미터(

)를 획득할 수 있다. 여기서, 랜덤 매트릭스(R)과 랜덤 매트릭스의 트랜스포즈(

)의 곱은 단위행렬(identity matrix)로서 정의될 수 있다.Referring to Equation 3, the plurality of learning machines 200 have an average compression gradient (

) to the transpose of the random matrix (R) (

) to decompress, and set parameters (

) by adding the average gradient parameter (

) can be obtained. Here, the random matrix (R) and the random matrix transpose (

) can be defined as an identity matrix.

)에 기초하여 인공지능 모델의 파라미터를 갱신할 수 있다(S238).The plurality of learning machines 200 obtain the average gradient parameter (

), it is possible to update the parameters of the artificial intelligence model based on (S238).

For example, the plurality of learning machines 200 may update parameters of each AI model according to distributed learning, and the updated parameters may be defined by Equation 4 below.

는 훈련 레이트(learning rate)를 의미하고, 바람직하게는 0.01 내지 0.02 값으로 설정될 수 있다.Referring to Equation 4,

denotes a training rate, and may preferably be set to a value of 0.01 to 0.02.

)를 업데이트할 수 있다. 이동 평균법은 평균을 구할 수량을 설정하여 시간이 경과함에 따라 과거 데이터를 제외하고, 신규 데이터를 추가하여 평균화하는 방법으로서, 갑작스러운 데이터의 변동을 방지할 수 있다.In one embodiment, the plurality of learning machines 200 use a moving average method to set parameters (

) can be updated. The moving average method is a method of averaging by adding new data while excluding past data as time elapses by setting the quantity to be averaged, and it is possible to prevent sudden data fluctuations.

)를 기초로 이동 평균법을 이용해 설정 파라미터(

)를 업데이트할 수 있다. 이동 평균법을 이용하여 갱신된 설정 파라미터(

)는 다음 수학식 5에 의해 계산될 수 있다.That is, after the plurality of learning machines 200 are updated with the parameters of the artificial intelligence model, the average gradient parameter (

) using the moving average method based on the setting parameters (

) can be updated. Setting parameters updated using the moving average method (

) can be calculated by Equation 5 below.

는 압축률을 의미하고, 랜덤 매트릭스(R)의 열의 개수를 행의 개수로 나눈 값으로 정의될 수 있다.Referring to Equation 5,

denotes a compression ratio, and may be defined as a value obtained by dividing the number of columns of the random matrix R by the number of rows.

In an embodiment, operations S231 to S238 may be repeatedly performed whenever learning of the artificial intelligence model is in progress.

In the present invention, it is possible to more accurately predict the power demand by using the parameter update method and system in the distributed machine learning described above, and through this, it is possible to establish a more stable and efficient power provision strategy.

As described above, the present invention has the effect of shortening the learning time of the super-capacity learning data and reducing the communication cost by reducing the network communication amount.

In addition, the present invention has the effect of improving the learning performance by reducing the storage and distribution required in distributed machine learning.

Meanwhile, various embodiments described herein may be implemented by hardware, middleware, microcode, software, and/or a combination thereof. For example, various embodiments may include one or more application specific semiconductors (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs). ), processors, controllers, microcontrollers, microprocessors, other electronic units designed to perform the functions presented herein, or a combination thereof.

Also, for example, the various embodiments may be embodied in or encoded on a computer-readable medium comprising instructions. The instructions embodied in or encoded on a computer-readable medium may cause a programmable processor or other processor to perform a method, eg, when the instructions are executed. A storage medium may be any available medium that can be accessed by a computer. For example, such computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage medium, magnetic disk storage medium or other magnetic storage device, or desired program code, containing instructions or may include any other medium that can be used for storage in the form of data structures.

Such hardware, software, firmware, etc. may be implemented in the same device or in separate devices to support the various operations and functions described herein. Additionally, components, units, modules, components, etc. described as “parts” in the present invention may be implemented together or individually as separate but interoperable logic devices. Depictions of different features of modules, units, etc. are intended to emphasize different functional embodiments, and do not necessarily imply that they must be realized by separate hardware or software components. Rather, functionality associated with one or more modules or units may be performed by separate hardware or software components or integrated within common or separate hardware or software components.

Although acts are shown in the drawings in a particular order, it should not be understood that these acts need to be performed in the specific order, or sequential order, shown, or that all shown acts need to be performed to achieve a desired result. . In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the division of various components in the above-described embodiments should not be construed as requiring such division in all embodiments, and that the described components will generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there can be

As described above, an optimal embodiment has been disclosed in the drawings and the specification. Although specific terms are used herein, they are used only for the purpose of describing the present invention, and are not used to limit the meaning or scope of the present invention described in the claims. Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the appended claims.

100: central server 200: learning machine

Claims (9)

the central server sending a random seed;
receiving, by a plurality of learning machines, the random seed, and generating a random matrix based on the received random seed;
generating, by the plurality of learning machines, a local gradient parameter based on training data assigned to them;
generating compressed information by compressing, by the plurality of learning machines, the generated local gradient parameter and setting parameter using the random matrix;
sending, by the plurality of learning machines, the compressed information to the central server;
generating, by the central server, the compressed information received from the plurality of learning machines to generate an average compression gradient parameter;
sending, by the central server, the average compressed gradient parameter to the plurality of learning machines;
decompressing, by the plurality of learning machines, the received average compressed gradient parameter using the transpose of the random matrix to obtain an average gradient parameter; and
and updating, by the plurality of learning machines, the parameters of the artificial intelligence model based on the obtained average gradient parameters.
The method of claim 1,
The distributed machine learning method of an artificial intelligence neural network further comprising the plurality of learning machines updating the setting parameters by using the average gradient parameters.
The method of claim 1,
The operation of the plurality of learning machines generating the compressed information includes:
and generating the compressed information by multiplying the difference between the local gradient parameter and the set parameter by the random matrix.
The method of claim 1,
The operation of the plurality of learning machines to obtain the average gradient parameter comprises:
and obtaining the average gradient parameter by adding the setting parameter to a value obtained by multiplying the average compressed gradient parameter and the transform of the random matrix.
A computer-readable storage medium comprising:
A computer-readable storage medium comprising a computer program that, when executed on a computer, performs the distributed machine learning method according to any one of claims 1 to 4.
The central server sends a random seed,
a plurality of learning machines receive the random seed, and generate a random matrix based on the received random seed;
The plurality of learning machines generate local gradient parameters based on the training data assigned to them,
The plurality of learning machines compress the generated local gradient parameter and the setting parameter using the random matrix to generate compressed information,
the plurality of learning machines send the compressed information to the central server,
the central server aggregates the compression information received from the plurality of learning machines to generate an average compression gradient parameter;
the central server sends the average compressed gradient parameter to the plurality of learning machines;
the plurality of learning machines decompress the received average compressed gradient parameter using the transpose of the random matrix to obtain an average gradient parameter;
A distributed machine learning system of an artificial intelligence neural network, wherein the plurality of learning machines update the parameters of the artificial intelligence model based on the obtained average gradient parameters.
The method of claim 6, wherein the plurality of learning machines,
A distributed machine learning system of an artificial intelligence neural network that updates the set parameter using the average gradient parameter.
The method of claim 6, wherein the plurality of learning machines,
A distributed machine learning system of an artificial intelligence neural network for generating the compressed information by multiplying a difference between the local gradient parameter and the set parameter by the random matrix.
The method of claim 6, wherein the plurality of learning machines,
A distributed machine learning system of an artificial intelligence neural network for obtaining the average gradient parameter by adding the setting parameter to a value obtained by multiplying the average compressed gradient parameter and the transform of the random matrix.

Images