CN117910521B - Gradient compression method, gradient compression device, gradient compression equipment, distributed cluster system and storage medium - Google Patents

Abstract

The present invention discloses a gradient compression method, device, equipment, distributed cluster and storage medium, which belongs to the field of distributed computing, and is used to refer to two indicators, namely, the model performance optimization rate and the current single-step training duration, to adjust the gradient compression degree, and solve the problem that the model performance and the communication overhead cannot be balanced when the gradient compression is performed on the low-speed network. The present invention uses a single training step as the granularity, and after obtaining the gradient data in any training step after the warm-up stage, the gradient compression degree is reduced when the model performance optimization rate does not meet the standard, so as to improve the model performance, and when the model performance optimization rate meets the standard and the current single-step training duration exceeds the standard, the gradient compression degree can be amplified to reduce the communication overhead, and the present invention can dynamically adjust the compression degree of the gradient data in combination with the influence of the network status, so as to reduce the communication overhead as much as possible on the basis of taking into account the model performance and the network status.

Description

Gradient compression method, device, equipment, distributed cluster system and storage medium

Technical Field

The present invention relates to the field of distributed computing, and in particular to a gradient compression method, device, equipment, distributed cluster and storage medium.

Background Art

With the development of large language models (LLM), the training of deep learning models requires more computing resources, so more and more distributed clusters are used for distributed computing. During the model training process, any computing node in the distributed cluster needs to send the gradient data generated during the iterative training of the model to other computing nodes in the distributed cluster through the network, so that each computing node can update the model parameters.

During the model training process of distributed clusters, a huge amount of gradient data needs to be transmitted through the network. In order to cope with some slow and unstable networks, the gradient data can be compressed before transmission through the network. However, although the compression of gradient data can reduce communication overhead, it may affect the model performance. Therefore, how to balance the model performance and communication overhead during the compression of gradient data is a difficult problem that needs to be solved urgently.

Therefore, how to provide a solution to the above technical problems is a problem that technical personnel in this field need to solve at present.

Summary of the invention

The purpose of the present invention is to provide a gradient compression method, device, equipment, distributed cluster and storage medium. With a single training step as the granularity, after obtaining gradient data in any training step after the warm-up stage, when the model performance optimization rate does not meet the standard, the gradient compression degree is reduced to improve the model performance. When the model performance optimization rate meets the standard and the current single-step training time exceeds the standard, the gradient compression degree can be amplified to reduce the communication overhead. The present invention can dynamically adjust the compression degree of the gradient data in combination with the influence of the network status, thereby reducing the communication overhead as much as possible on the basis of taking into account the model performance and the network status.

In order to solve the above technical problems, the present invention provides a gradient compression method, which is applied to any computing node in a distributed cluster, comprising:

For any training step after the warm-up phase of the local model iterative training, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard;

If the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule;

If the current model performance optimization rate meets the standard, determine whether the current single-step training time exceeds the standard;

If the current single-step training duration exceeds the limit, the current gradient compression degree of the preset compression method is amplified according to the second preset rule;

Based on the latest gradient compression degree, compressing the gradient data of the current training step using a preset compression method so as to synchronize the compressed gradient data within the distributed cluster;

Among them, the single-step training duration is related to the degree of gradient compression and the network status.

On the other hand, judging whether the current model performance optimization rate meets the standard includes:

Determine whether the improvement in model performance has reached the target within the previous step sliding window;

If the improvement of model performance meets the standard, it is determined that the current model performance optimization rate meets the standard;

If the improvement of model performance does not meet the standard, it is determined that the current model performance optimization rate does not meet the standard;

Wherein, the step number sliding window includes a preset number of training steps.

On the other hand, judging whether the improvement of model performance in the previous step sliding window has reached the standard includes:

Determine the relationship based on the loss function change rate to determine the loss function change rate of the local model within the previous step sliding window;

If the loss function change rate is less than the preset change rate threshold, it is determined that the improvement in model performance has not reached the target;

If the change rate of the loss function is not less than the preset change rate threshold, it is determined that the improvement of the model performance meets the standard;

The loss function change rate determination relationship includes:

;

in, It represents the rate of change of the loss function L in the sliding window of the previous step, based on the current t-th training step; is the loss function of the local model; It represents the sliding average of the loss function L in the sliding window of the previous step based on the current t-th training step, and τ is the variable in the summation operation; Represents the minimum loss function value in the previous step sliding window, and the step sliding window includes M training steps.

On the other hand, the preset change rate threshold includes:

;

in, is the decay function related to the number of training steps t, is a hyperparameter.

On the other hand, the preset compression method includes a hybrid gradient compression method combining a gradient quantization method and a gradient sparsification method.

On the other hand, reducing the current gradient compression degree of the preset compression method according to the first preset rule includes:

According to the compression degree reduction relation, the current gradient compression degree of the preset compression method is reduced;

The compression reduction relational expression includes:

clip _upper (2Q(t))·λS(t);

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _upper () represents the clip function used to perform the gradient quantization upper limit clipping operation, λ is a preset adjustment parameter, λ＞1.

On the other hand, amplifying the current gradient compression degree of the preset compression method according to the second preset rule includes:

According to the compression degree amplification relation, amplify the current gradient compression degree of the preset compression method;

The compression degree amplification relational expression includes:

;

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _lower () represents the clip function used to perform the gradient quantization lower limit clipping operation, λ is a preset adjustment parameter, λ＞1.

On the other hand, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, judging whether the current model performance optimization rate meets the standard includes:

At any training step after the iterative training of the local model starts, based on the gradient data of the current training step, determine whether compression of the gradient data will cause the iterative training to fail to converge;

If it causes the iterative training to fail to converge, the iterative training is determined to be in the warm-up stage;

If it does not cause the iterative training to fail to converge, the warm-up phase of the iterative training is determined to be over;

For any training step after the warm-up phase of the local model iterative training, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard.

On the other hand, based on the gradient data of the current training step, whether the compression of the gradient data will cause the iterative training to fail to converge includes:

Determine the variance of the gradient data of the current training step;

Determine whether the variance of the gradient data of the current training step is greater than the preset variance threshold;

If it is greater than, it is determined that the compression of gradient data will cause the iterative training to fail to converge;

If it is not greater than, it is determined that the compression of the gradient data will not cause the iterative training to fail to converge.

On the other hand, after determining whether the current model performance optimization rate meets the standard, the gradient compression method further includes:

If the current model performance optimization rate does not meet the standard, the number of consecutive times the optimization rate does not meet the standard is increased by one, and it is determined whether the number of consecutive times the optimization rate does not meet the standard reaches a first preset number threshold;

If it is reached, the control prompter will prompt that the optimization rate has not been reached for too many consecutive times;

If the current model performance optimization rate meets the standard, the number of consecutive times that the optimization rate fails to meet the standard is reset to zero.

On the other hand, after determining whether the current single-step training duration exceeds the standard, the gradient compression method further includes:

If the current single-step training duration exceeds the standard, the number of times the single-step training duration exceeds the standard continuously is increased by one, and it is determined whether the number of times the single-step training duration exceeds the standard continuously reaches a second preset number threshold;

If it is reached, the control prompter will prompt that the single-step training duration has exceeded the limit for too many consecutive times;

If the current single-step training duration does not exceed the standard, the number of times the single-step training duration exceeds the standard continuously is reset to zero.

On the other hand, the gradient compression method further comprises:

In response to the standard modification instruction, the standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration are modified.

On the other hand, judging whether the current single-step training duration exceeds the standard includes:

Determine the training duration of the previous training step as the current single-step training duration;

Determine whether the current single-step training duration is greater than a preset duration threshold;

If it is greater than, it is determined that the current single-step training duration exceeds the limit;

If it is not greater than, it is determined that the current single-step training duration does not exceed the limit.

On the other hand, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, before reducing the current gradient compression degree of the preset compression method according to the first preset rule, the gradient compression method further includes:

Determine whether the gradient distortion of the gradient data of the current training step is exceeded based on the current gradient compression degree;

If the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule, including:

If the current model performance optimization rate does not meet the standard and/or the gradient distortion exceeds the standard, reducing the current gradient compression degree of the preset compression method according to the first preset rule;

If the current model performance optimization rate meets the standard, the following steps are used to determine whether the current single-step training duration exceeds the standard:

If the current model performance optimization rate meets the standard and the gradient distortion does not exceed the standard, determine whether the current single-step training duration exceeds the standard.

On the other hand, judging whether the gradient distortion of the gradient data of the current training step is excessive based on the current gradient compression degree includes:

Determine a relationship based on the gradient distortion, and determine the gradient distortion for compressing the gradient data of the current training step based on the current gradient compression degree;

If the gradient distortion is greater than a preset distortion threshold, it is determined that the gradient distortion exceeds the standard;

If the gradient distortion is not greater than the preset distortion threshold, it is determined that the gradient distortion does not exceed the standard;

The gradient distortion determination relationship includes:

;

Wherein, g _GC is the gradient data of the current training step after compression based on the current gradient compression degree, and g is the gradient data of the current training step without compression. represents the Euclidean distance between g _GC and g.

On the other hand, after determining whether the gradient distortion degree of compressing the gradient data of the current training step based on the current gradient compression degree exceeds the standard, the gradient compression method further includes:

If the gradient distortion exceeds the standard, the number of times the gradient distortion exceeds the standard continuously is increased by one, and it is determined whether the number of times the gradient distortion exceeds the standard continuously reaches a third preset number threshold;

If it is reached, the control prompter will prompt that the gradient distortion has exceeded the limit for too many times in a row;

If the gradient distortion does not exceed the standard, the number of times the gradient distortion exceeds the standard continuously is reset to zero.

In order to solve the above technical problems, the present invention further provides a gradient compression device, which is applied to any computing node in a distributed cluster, comprising:

The first judgment module is used for determining whether the current model performance optimization rate meets the standard for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step. If the current model performance optimization rate does not meet the standard, the first adjustment module is triggered; if the current model performance optimization rate meets the standard, the second judgment module is triggered;

The first adjustment module is used to reduce the current gradient compression degree of the preset compression method according to the first preset rule;

The second judgment module is used to judge whether the current single-step training duration exceeds the standard. If the current single-step training duration exceeds the standard, the second adjustment module is triggered;

The second adjustment module is used to amplify the current gradient compression degree of the preset compression method according to a second preset rule;

A compression module, used to compress the gradient data of the current training step using a preset compression method based on the latest gradient compression degree, so as to synchronize the compressed gradient data within the distributed cluster;

Among them, the single-step training duration is related to the degree of gradient compression and the network status.

In order to solve the above technical problems, the present invention further provides a gradient compression device, comprising:

Memory for storing computer programs;

A processor is used to implement the steps of the gradient compression method described above when executing the computer program.

To solve the above technical problem, the present invention also provides a distributed cluster, comprising a plurality of gradient compression devices as described above.

To solve the above technical problem, the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above gradient compression method are implemented.

Beneficial effects: The present invention provides a gradient compression method. Considering that the gradient data needs to be compressed after any training step, and the degree of gradient compression can respectively affect the optimization speed of model performance and the single-step training time, and the single-step training time is correlated with the network condition, the present invention uses a single training step as the granularity. After obtaining the gradient data in any training step after the warm-up stage, the gradient compression degree is reduced when the model performance optimization rate does not meet the standard, so as to improve the model performance. When the model performance optimization rate meets the standard and the current single-step training time exceeds the standard, the gradient compression degree can be amplified to reduce the communication overhead. The present invention can dynamically adjust the compression degree of the gradient data in combination with the influence of the network condition, so as to reduce the communication overhead as much as possible on the basis of taking into account the model performance and the network condition.

The present invention also provides a gradient compression device, equipment, distributed cluster and computer-readable storage medium, which have the same beneficial effects as the above gradient compression method.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the relevant technologies and the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

FIG1 is a schematic flow chart of a gradient compression method provided by the present invention;

FIG2 is a schematic diagram of the structure of a distributed cluster provided by the present invention;

FIG3 is a schematic flow chart of another gradient compression method provided by the present invention;

FIG4 is a schematic structural diagram of a gradient compression device provided by the present invention;

FIG5 is a schematic structural diagram of a gradient compression device provided by the present invention;

FIG. 6 is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention.

DETAILED DESCRIPTION

The core of the present invention is to provide a gradient compression method, device, equipment, distributed cluster and storage medium. With a single training step as the granularity, after obtaining the gradient data in any training step after the warm-up stage, if the model performance optimization rate does not meet the standard, the gradient compression degree is reduced to improve the model performance. If the model performance optimization rate meets the standard and the current single-step training time exceeds the standard, the gradient compression degree can be amplified to reduce the communication overhead. The present invention can dynamically adjust the compression degree of the gradient data in combination with the influence of the network status, thereby reducing the communication overhead as much as possible on the basis of taking into account the model performance.

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

Please refer to FIG. 1 , which is a flow chart of a gradient compression method provided by the present invention. The gradient compression method is applied to any computing node in a distributed cluster, and includes:

S101: For any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard;

To better illustrate the embodiment of the present invention, please refer to FIG. 2 , which is a schematic diagram of the structure of a distributed cluster provided by the present invention. The gradient compression method in the embodiment of the present invention can be applied to any computing node in the distributed cluster.

Specifically, considering the technical problems in the above background technology, and considering that the gradient data needs to be compressed after any training step, and the degree of gradient compression can respectively affect the optimization speed of model performance and the single-step training time, the present invention intends to use a single training step in the iterative training process of the local model as the granularity, and seek to adjust the degree of gradient compression, so as to achieve dynamic adjustment of the degree of gradient compression, and more timely and flexibly adjust the balance between model performance and communication overhead; at the same time, considering that the "model performance optimization rate" determines the performance of the final local model to a large extent during the iterative training process, based on this, in an embodiment of the present invention, for any training step after the warm-up stage of the iterative training of the local model, after obtaining the gradient data of the current training step, it is possible to determine whether the current model performance optimization rate meets the standard, and use the judgment result as the data basis for subsequent steps, so as to ensure the model performance optimization rate by timely adjusting the degree of gradient compression.

Among them, considering that in the initial stage of iterative training of the local model in the distributed cluster, the gradient data and the model parameters have not formed a good foundation, and gradient compression cannot be directly introduced, which will cause the iterative training of the local model to fail to converge. Therefore, a warm-up stage can be divided in the early stage of the iterative training. In this stage, the original uncompressed gradient data can be transmitted over the network to lay a good foundation for the iterative training of the local model in the distributed cluster. After the warm-up stage, the gradient data can be compressed to save communication overhead. Therefore, in the embodiment of the present invention, "for any training step after the warm-up stage of the iterative training of the local model, after obtaining the gradient data of the current training step, it is determined whether the current model performance optimization rate meets the standard."

S102: If the current model performance optimization rate does not meet the standard, reducing the current gradient compression degree of the preset compression method according to the first preset rule;

Specifically, since the "model performance optimization rate" determines the performance of the final local model to a large extent, and the performance of the local model is an indicator of focus, in an embodiment of the present invention, when the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method can be reduced according to the first preset rule. The reduction in the gradient compression degree represents a reduction in the compression degree of the gradient data, which is conducive to learning more features in the local training data during the model parameter optimization process, thereby improving model performance.

S103: If the current model performance optimization rate meets the standard, determine whether the current single-step training time exceeds the standard;

Specifically, considering that after model performance, communication overhead and "training time proportional to communication overhead" are indicators that need to be focused on, and these two indicators are related to the single-step training time in the local model iterative training process. In addition, fluctuations in network bandwidth will also affect the single-step training time. Therefore, the embodiment of the present invention can determine whether the current single-step training time exceeds the standard when the current model performance optimization rate meets the standard, so as to trigger subsequent actions according to the judgment result, thereby combining the impact of network conditions on the single-step training time to reduce the communication overhead as much as possible.

Specifically, the single-step training duration may include the computation time of a single training step and the communication time consumed by gradient data transmission, and the communication time is affected by factors including the degree of gradient compression and the network status. However, in actual applications, there may be some overlap between the computation time and the communication time.

S104: If the current single-step training duration exceeds the standard, amplify the current gradient compression degree of the preset compression method according to the second preset rule;

Among them, there is a certain theoretical range for the single-step training duration in theory, that is, the "standard" in the exceeding standard in this step. It can be considered that when the single-step training duration exceeds the standard, there is room for downward adjustment, and when the single-step training duration does not exceed the standard, there is no room for downward adjustment. Therefore, in this step, when it is determined that the current single-step training duration exceeds the standard, the current gradient compression degree of the preset compression method can be amplified to reduce the single-step training duration, thereby reducing the training time of the model.

Among them, the current gradient compression degree of the preset compression method can be amplified by the second preset rule, so as to achieve higher flexibility. The first preset rule and the second preset rule can be flexibly and independently set, and the embodiment of the present invention is not limited here.

S105: Based on the latest gradient compression degree, a preset compression method is used to compress the gradient data of the current training step, so as to synchronize the compressed gradient data in the distributed cluster;

Among them, the single-step training duration is related to the degree of gradient compression and the network status.

Specifically, after the adjustment in the above steps, based on the judgment of "model performance optimization rate" and "single-step training duration", the adjustment of the gradient compression degree in the training step is completed. Therefore, in this step, based on the latest gradient compression degree, the preset compression method can be used to compress the gradient data of the current training step, so that the compressed gradient data can be synchronized in the distributed cluster.

The present invention provides a gradient compression method. Considering that gradient data needs to be compressed after any training step, and the degree of gradient compression can respectively affect the optimization speed of model performance and the single-step training time, and the single-step training time is correlated with the network condition, the present invention takes a single training step as the granularity, and after obtaining gradient data in any training step after the warm-up stage, the degree of gradient compression is reduced in the case that the model performance optimization rate does not meet the standard, so as to improve the model performance, and in the case that the model performance optimization rate meets the standard and the current single-step training time exceeds the standard, the degree of gradient compression can be amplified to reduce the communication overhead. The present invention can dynamically adjust the degree of compression of gradient data in combination with the influence of the network condition, so as to reduce the communication overhead as much as possible on the basis of taking into account both the model performance and the network condition.

Based on the above embodiments:

As an optional embodiment, determining whether the current model performance optimization rate meets the standard includes:

Determine whether the improvement in model performance has reached the target within the previous step sliding window;

If the improvement of model performance meets the standard, it is determined that the current model performance optimization rate meets the standard;

If the improvement of model performance does not meet the standard, it is determined that the current model performance optimization rate does not meet the standard;

The step number sliding window includes a preset number of training steps.

Specifically, considering that the improvement of model performance can be evaluated for each training step, and the improvement of model performance in the past several training steps can represent the current model performance optimization rate of the local model, the present invention pre-sets a step sliding window, and can determine whether the improvement of model performance in the previous step sliding window meets the standard. If the improvement of model performance meets the standard, it is determined that the current model performance optimization rate meets the standard; if the improvement of model performance does not meet the standard, it is determined that the current model performance optimization rate does not meet the standard.

The preset number of training steps included in the step number sliding window can be set independently, and the embodiment of the present invention does not limit this.

Of course, in addition to this specific method, other methods may be used to determine whether the current model performance optimization rate meets the standard, which is not limited in the embodiment of the present invention.

As an optional embodiment, judging whether the improvement of the model performance in the previous step number sliding window meets the standard includes:

Determine the relationship based on the loss function change rate to determine the loss function change rate of the local model within the previous step sliding window;

If the loss function change rate is less than the preset change rate threshold, it is determined that the improvement in model performance has not reached the target;

If the loss function change rate is not less than the preset change rate threshold, it is determined that the improvement in model performance has reached the standard;

The loss function change rate determination relationship includes:

;

in, It represents the rate of change of the loss function L in the sliding window of the previous step, based on the current t-th training step; is the loss function of the local model; It represents the sliding average of the loss function L in the sliding window of the previous step based on the current t-th training step, and τ is the variable in the summation operation; Represents the minimum loss function value in the previous step sliding window, and the step sliding window includes M training steps.

Specifically, considering that the loss function can accurately evaluate the changes in model performance between each training step, and the loss function change rate can represent the optimization rate of model performance, the embodiment of the present invention can determine the relationship based on the loss function change rate, determine the loss function change rate of the local model in the previous step sliding window, and compare the loss function change rate with the preset change rate threshold. If the loss function change rate is less than the preset change rate threshold, it is determined that the improvement in model performance does not meet the standard. If the loss function change rate is not less than the preset change rate threshold, it is determined that the improvement in model performance meets the standard.

Specifically, in the above loss function change rate determination formula, the subtraction of the two numerators indicates how much the current minimum loss function value is reduced compared to the "sliding average of the loss function L in the previous step sliding window". Dividing it by the "sliding average of the loss function L in the previous step sliding window" indicates the proportion of the current minimum loss function value reduced compared to the "sliding average of the loss function L in the previous step sliding window". We can pay attention to the change in this proportion to see if it is significant enough (which can be obtained by comparing it with the preset change rate threshold).

Among them, the loss function change rate of the local model in the previous step sliding window can be determined efficiently and accurately through the above loss function change rate determination relationship.

Of course, in addition to the above specific forms, the loss function change rate determination relationship can also be in other specific forms, and the embodiments of the present invention are not limited here.

As an optional embodiment, the preset change rate threshold includes:

;

in, is the decay function related to the number of training steps t, is a hyperparameter.

Specifically, considering that as the local model training stage deepens, the optimization rate of model performance also presents a trend from fast to slow, the preset change rate threshold should theoretically also be in a decaying trend. Therefore, in order to more accurately judge whether the "model performance optimization rate meets the standard" of different training steps, the core of the preset change rate threshold in the embodiment of the present invention is a decay function related to the number of training steps, which is fine-tuned with hyperparameters.

Specifically, the attenuation function may be of various types, such as staged attenuation, exponential attenuation, and cosine attenuation, etc., which is not limited in the embodiment of the present invention.

Of course, in addition to this specific form, the preset change rate threshold may also be in many other forms, which is not limited in the embodiment of the present invention.

As an optional embodiment, the preset compression method includes a hybrid gradient compression method that combines a gradient quantization method with a gradient sparsification method.

Specifically, considering that the essential principle of the gradient quantization method is to reduce the number of bits required to characterize a single communication data (that is, gradient data), and the essential principle of the gradient sparsification method is to reduce the amount of communication data, there is no contradiction between the two methods themselves, and there are conditions for common use. If only one of them is used, the degree of compression and the compression effect are limited (because the quantization method and the sparsification method have different sensitivities to gradient data of different features). Therefore, the preset compression method in the embodiment of the present invention may include a hybrid gradient compression method combining the gradient quantization method and the gradient sparsification method. In specific implementation, the gradient data to be compressed can be first sparsely processed by the gradient sparsification method, and then quantized by the gradient quantization method, etc. The embodiment of the present invention is not limited here.

Among them, the gradient quantization method and the gradient sparsification method can be specifically of multiple types. For example, the quantization range of the gradient quantization method can be flexibly set, for example, from 32 bits to 1 bit, etc., and the specific gradient sparsification method can be the Top-K method (that is, retaining the first K gradient data from a total of N _g gradient data), etc., which is not limited in the embodiments of the present invention.

As an optional embodiment, reducing the current gradient compression degree of the preset compression method according to the first preset rule includes:

According to the compression degree reduction relation, the current gradient compression degree of the preset compression method is reduced;

The compression reduction relationships include:

clip _upper (2Q(t))·λS(t);

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _upper () represents the clip function used to perform the gradient quantization upper limit clipping operation, λ is a preset adjustment parameter, λ＞1.

Specifically, the gradient compression degree can be adjusted appropriately and quickly through the compression degree reduction relationship as above, where clip _upper (2Q(t)) represents the callback of the gradient compression degree corresponding to the gradient quantization strategy function, which is specifically manifested as raising the current quantization accuracy Q(t) to 2Q(t), but the highest quantization accuracy of the system is required as the upper limit, so the Clip function can be used to perform an upper limit clipping operation; λS(t) represents the callback of the gradient compression degree corresponding to the gradient sparsification strategy, which is specifically manifested as amplifying the current sparsity S(t) by λ times.

As an optional embodiment, amplifying the current gradient compression degree of the preset compression method according to the second preset rule includes:

According to the compression degree amplification relation, amplify the current gradient compression degree of the preset compression method;

The compression degree amplification relationship includes:

;

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _lower () represents the clip function used to perform the gradient quantization lower limit clipping operation, λ is a preset adjustment parameter, λ＞1.

Specifically, the gradient compression degree can be quickly and appropriately amplified through the above compression degree amplification relationship. The clip _lower (½Q(t)) represents the amplification of the gradient compression degree corresponding to the gradient quantization strategy function, which is specifically manifested as increasing the current quantization accuracy Q(t) to ½Q(t). However, the system's lowest quantization accuracy is required as the lower limit, so the Clip function can be used to perform a lower limit clipping operation. The second half of the above formula represents the adjustment of the gradient compression degree corresponding to the gradient sparsification strategy, which is specifically manifested as multiplying the current sparsity S(t) by 1/λ.

Of course, in addition to the above compression reduction relational expression and compression enlargement relational expression, the compression reduction relational expression and compression enlargement relational expression may also be in other specific forms, which are not limited in the embodiments of the present invention.

As an optional embodiment, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, determining whether the current model performance optimization rate meets the standard includes:

At any training step after the iterative training of the local model starts, based on the gradient data of the current training step, determine whether compression of the gradient data will cause the iterative training to fail to converge;

If it causes the iterative training to fail to converge, the iterative training is determined to be in the warm-up stage;

If it does not cause the iterative training to fail to converge, the warm-up phase of the iterative training is determined to be over;

For any training step after the warm-up phase of the local model iterative training, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard.

Specifically, in order to more accurately identify the warm-up stage of the iterative training of the local model, considering that the gradient data of a single training step can reflect whether the "gradient data compression action" will cause the iterative training to fail to converge, and then determine whether the iterative training is in the warm-up stage, therefore, in an embodiment of the present invention, at any training step after the start of the iterative training of the local model, according to the gradient data of the current training step, it can be judged whether the compression of the gradient data will cause the iterative training to fail to converge; if it causes the iterative training to fail to converge, it is determined that the iterative training is in the warm-up stage; if it does not cause the iterative training to fail to converge, it is determined that the warm-up stage of the iterative training is over, thereby efficiently and accurately realizing the identification of the warm-up stage.

Of course, in addition to this specific method, the identification of the preheating stage can also be performed in other ways, which are not limited in the embodiment of the present invention.

As an optional embodiment, judging whether compression of gradient data will cause iterative training to fail to converge according to the gradient data of the current training step includes:

Determine the variance of the gradient data of the current training step;

Determine whether the variance of the gradient data of the current training step is greater than the preset variance threshold;

If it is greater than, it is determined that the compression of gradient data will cause the iterative training to fail to converge;

If it is not greater than, it is determined that the compression of the gradient data will not cause the iterative training to fail to converge.

Specifically, considering that the variance of the gradient data can reflect whether the local model is in the warm-up stage at the current moment, that is, whether it is suitable for compressing the gradient data, the embodiment of the present invention can determine whether the variance of the gradient data of the current training step is greater than the preset variance threshold. If it is greater than, it is determined that compression of the gradient data will cause the iterative training to fail to converge. If it is not greater than, it is determined that compression of the gradient data will not cause the iterative training to fail to converge, thereby efficiently and accurately "determining whether compression of the gradient data will cause the iterative training to fail to converge based on the gradient data of the current training step".

In the warm-up phase, the gradient compression degree of the gradient quantization method and the gradient sparsification method can be the lowest, for example, it can be a state where no quantization and sparsification processing is performed.

Of course, in addition to this specific form, "determining whether compression of gradient data will cause iterative training to fail to converge based on the gradient data of the current training step" can also be performed in other ways, which are not limited in the embodiments of the present invention.

As an optional embodiment, after determining whether the current model performance optimization rate meets the standard, the gradient compression method further includes:

If the current model performance optimization rate does not meet the standard, the number of consecutive times the optimization rate does not meet the standard is increased by one, and it is determined whether the number of consecutive times the optimization rate does not meet the standard reaches a first preset number threshold;

If it is reached, the control prompter will prompt that the optimization rate has not been reached for too many consecutive times;

If the current model performance optimization rate meets the standard, the number of consecutive times the optimization rate fails to meet the standard will be reset to zero.

Specifically, considering that in some cases, even after repeated adjustments to the degree of gradient compression, the model performance optimization rate may continue to fail to meet the standard, this state requires staff to conduct inspections and repairs. Therefore, in order to inform the staff as soon as possible, this situation can be monitored in an embodiment of the present invention. Therefore, if the current model performance optimization rate does not meet the standard, the number of consecutive times the optimization rate fails to meet the standard is increased by one, and it is determined whether the number of consecutive times the optimization rate fails to meet the standard reaches a first preset number threshold; if reached, the control prompter prompts that the number of consecutive times the optimization rate fails to meet the standard is too high; if the current model performance optimization rate meets the standard, the number of consecutive times the optimization rate fails to meet the standard is cleared to zero, thereby realizing automatic recording of the consecutive times of "model performance optimization rate fails to meet the standard" and triggering corresponding alarms, which is beneficial to improving system reliability and user experience.

As an optional embodiment, after determining whether the current single-step training duration exceeds the standard, the gradient compression method further includes:

If the current single-step training duration exceeds the standard, the number of times the single-step training duration exceeds the standard continuously is increased by one, and it is determined whether the number of times the single-step training duration exceeds the standard continuously reaches a second preset number threshold;

If it is reached, the control prompter will prompt that the single-step training duration has exceeded the limit for too many consecutive times;

If the current single-step training duration does not exceed the limit, the number of consecutive times the single-step training duration exceeds the limit will be reset to zero.

Specifically, considering that theoretically after several adjustments to the gradient compression degree, the single-step training duration will not exceed the standard continuously, that is, the single-step training duration exceeding the standard for multiple times continuously is an abnormal situation. In order to inform the staff as soon as possible, this situation can be monitored in the embodiment of the present invention. Therefore, if the current single-step training duration exceeds the standard, the number of consecutive single-step training duration exceeding the standard is increased by one, and it is determined whether the number of consecutive single-step training duration exceeding the standard reaches the second preset number threshold; if reached, the control prompter prompts that the number of consecutive single-step training duration exceeding the standard is too high; if the current single-step training duration does not exceed the standard, the number of consecutive single-step training duration exceeding the standard is cleared to zero, thereby realizing automatic recording of the consecutive number of "single-step training duration exceeding the standard" and triggering corresponding alarms, which is beneficial to improving system reliability and user experience.

As an optional embodiment, the gradient compression method further includes:

In response to the standard modification instruction, the standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration are modified.

Specifically, taking into account the staff's need to modify the "standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration", in order to improve work efficiency, a relevant modification interface is provided in the embodiment of the present invention, so that the standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration can be modified in response to standard modification instructions.

As an optional embodiment, determining whether the current single-step training duration exceeds the standard includes:

Determine the training duration of the previous training step as the current single-step training duration;

Determine whether the current single-step training duration is greater than a preset duration threshold;

If it is greater than, it is determined that the current single-step training duration exceeds the limit;

If it is not greater than, it is determined that the current single-step training duration does not exceed the limit.

Specifically, considering that the communication time of the gradient data of the current training step has not been determined before the gradient data of the current training step is compressed and synchronized, the single-step training duration of the current training step cannot be determined, and the single-step training duration of the previous training step is affected by the current gradient compression degree and the current network status. Therefore, the training duration of the previous training step can be determined as the current single-step training duration, so as to determine whether the current single-step training duration exceeds the standard by judging whether the previous single-step training duration is greater than the preset duration threshold.

The preset duration threshold may be set independently, and the embodiment of the present invention does not limit this.

As an optional embodiment, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, before reducing the current gradient compression degree of the preset compression method according to the first preset rule, the gradient compression method further includes:

Determine whether the gradient distortion of the gradient data of the current training step is exceeded based on the current gradient compression degree;

If the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule, including:

If the current model performance optimization rate does not meet the standard and/or the gradient distortion exceeds the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule;

If the current model performance optimization rate meets the standard, the following steps are used to determine whether the current single-step training duration exceeds the standard:

If the current model performance optimization rate meets the standard and the gradient distortion does not exceed the standard, determine whether the current single-step training time exceeds the standard.

To better illustrate the embodiment of the present invention, please refer to FIG. 3 , which is a flow chart of another gradient compression method provided by the present invention, wherein S304 is the same as S102, S305 is the same as S103, S306 is the same as S104, S307 is the same as S105, and S301-S303 include:

S301: For any training step after the warm-up phase of the iterative training of the local model, obtain the gradient data of the current training step;

S302: Whether the current model performance optimization rate meets the standard;

S303: Whether the gradient distortion exceeds the standard.

Specifically, considering that in the process of performing the above adjustment action with the training step as the granularity, "compressing the gradient data of the current training step based on the current gradient compression degree" is likely to cause distortion of the gradient data, thereby affecting the longitude of the local model, therefore, in the embodiment of the present invention, it can be determined whether the gradient distortion degree of compressing the gradient data of the current training step based on the current gradient compression degree exceeds the standard. If it exceeds the standard, the action of "reducing the current gradient compression degree of the preset compression method according to the first preset rule" can also be triggered, thereby avoiding the occurrence of gradient data distortion caused by gradient compression, which is conducive to further improving the model accuracy.

Specifically, the basic idea of the patent is based on "mathematical modeling in a low-speed unstable network environment", and the specific content is as follows:

In a distributed training scenario with unstable network, for a given training task, if the total duration of local model training is recorded as , then we have:

;

in represents the communication time of the t-th training step, represents the computation time of the tth training step, Indicates the total number of steps required for iterative training of the local model. Since there is a certain overlap between communication time and computing time in practice, Approximately equal to the sum of the latter with respect to t.

The part of the scenario that we are really interested in is the communication time of the training step. The communication time of the training step can be further expanded and is proportional to the combination of multiple functions about t:

;

Specifically, GC refers to Gradient Compression, and GC(t) represents the gradient compression strategy for training step t. The first element is Q(t), which represents the gradient quantization strategy function. This is a distribution function about t, which means that the quantization strategy may change at each step in distributed training, so it is a function that changes dynamically about t. The second element is S(t), which represents the gradient pruning (sparseness) strategy function. This is a function about t, which means that the pruning strategy may change at each step in distributed training, so it is a function that changes dynamically about t.

The third item BW(t) represents the bandwidth of the network at the t-th training step in the iterative training. As described above, the present invention is aimed at a low-speed and unstable network environment. Therefore, the network bandwidth changes over time and is also expressed as a function of dynamic changes with t.

In this way, the problem of constructing a dynamic gradient compression strategy is transformed into a problem of optimizing the objective function ( or ), solve the optimal joint compression strategy (quantization + sparsification) The specific mathematical expression is as follows:

;

Wherein, Ng represents the number of gradient data; Lrequired represents _a given model performance requirement (which can be expressed as “model performance optimization rate reaching standard” in the _present invention), Indicates that the right side of the formula The optimal hybrid compression strategy that achieves the minimum value (communication time of the training step) (the strategy varies with time t) is the target optimization problem that needs to be solved.

As an optional embodiment, judging whether the gradient distortion degree of compressing the gradient data of the current training step based on the current gradient compression degree exceeds the standard includes:

Determine a relationship based on the gradient distortion, and determine the gradient distortion for compressing the gradient data of the current training step based on the current gradient compression degree;

If the gradient distortion is greater than the preset distortion threshold, it is determined that the gradient distortion exceeds the standard;

If the gradient distortion is not greater than the preset distortion threshold, it is determined that the gradient distortion is within the standard;

The gradient distortion determination relationship includes:

;

Wherein, g _GC is the gradient data of the current training step after compression based on the current gradient compression degree, and g is the gradient data of the current training step without compression. represents the Euclidean distance between g _GC and g.

Specifically, the gradient distortion determination formula as above can efficiently and accurately determine “the gradient distortion of compressing the gradient data of the current training step based on the current gradient compression degree”.

Of course, in addition to this specific method, the "gradient distortion degree of compressing the gradient data of the current training step based on the current gradient compression degree" may also be determined in other ways, which are not limited in the embodiment of the present invention.

As an optional embodiment, after determining whether the gradient distortion degree of the gradient data of the current training step is compressed based on the current gradient compression degree exceeds the standard, the gradient compression method further includes:

If the gradient distortion exceeds the standard, the number of times the gradient distortion exceeds the standard continuously is increased by one, and it is determined whether the number of times the gradient distortion exceeds the standard continuously reaches a third preset number threshold;

If it is reached, the control prompter will prompt that the gradient distortion has exceeded the limit for too many times in a row;

If the gradient distortion does not exceed the standard, the number of times the gradient distortion exceeds the standard continuously is reset to zero.

Specifically, considering that after continuous adjustment of the gradient compression degree, the gradient distortion degree exceeds the standard and still occurs for multiple times in succession, this is an abnormal situation. In order to monitor this situation, in the embodiment of the present invention, if the gradient distortion degree exceeds the standard, the number of times the gradient distortion degree exceeds the standard continuously is increased by one, and it is determined whether the number of times the gradient distortion degree exceeds the standard continuously reaches a third preset number threshold; if it reaches the third preset number threshold, the prompter is controlled to prompt that the number of times the gradient distortion degree exceeds the standard continuously is too high; if the gradient distortion degree does not exceed the standard, the number of times the gradient distortion degree exceeds the standard continuously is cleared to zero, thereby realizing automatic monitoring of the situation of "the number of times the gradient distortion degree exceeds the standard continuously is too high" and triggering an alarm.

The first preset number threshold, the second preset number threshold and the third preset number threshold can all be set independently, and the embodiment of the present invention does not limit this.

To better illustrate the embodiment of the present invention, please refer to FIG. 4 , which is a schematic diagram of the structure of a gradient compression device provided by the present invention. The gradient compression device is applied to any computing node in a distributed cluster, including:

The first judgment module 41 is used for determining whether the current model performance optimization rate meets the standard for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step. If the current model performance optimization rate does not meet the standard, the first adjustment module 42 is triggered; if the current model performance optimization rate meets the standard, the second judgment module 43 is triggered;

A first adjustment module 42, configured to reduce the current gradient compression degree of the preset compression method according to a first preset rule;

The second judging module 43 is used to judge whether the current single-step training duration exceeds the standard, and if the current single-step training duration exceeds the standard, the second adjusting module 44 is triggered;

A second adjustment module 44, used to amplify the current gradient compression degree of the preset compression method according to a second preset rule;

A compression module 45 is used to compress the gradient data of the current training step using a preset compression method based on the latest gradient compression degree, so as to synchronize the compressed gradient data in the distributed cluster;

Among them, the single-step training duration is related to the degree of gradient compression and the network status.

As an optional embodiment, the first determination module 41 includes:

The first judgment submodule is used to judge whether the improvement of the model performance in the previous step number sliding window meets the standard, and then trigger the first judgment module. If the improvement of the model performance does not meet the standard, then trigger the second judgment module;

The first determination module is used to determine whether the current model performance optimization rate meets the standard;

The second determination module is used to determine that the current model performance optimization rate does not meet the standard;

The step number sliding window includes a preset number of training steps.

As an optional embodiment, the first judgment submodule includes:

A first determination module is used to determine a relationship according to the loss function change rate, and determine the loss function change rate of the local model within the previous step number sliding window;

A third determination module is used to determine that the improvement of model performance does not meet the standard if the loss function change rate is less than a preset change rate threshold;

A fourth determination module is used to determine whether the improvement of model performance meets the standard if the loss function change rate is not less than a preset change rate threshold;

The loss function change rate determination relationship includes:

;

in, It represents the rate of change of the loss function L in the sliding window of the previous step, based on the current t-th training step; is the loss function of the local model; It represents the sliding average of the loss function L in the sliding window of the previous step based on the current t-th training step, and τ is the variable in the summation operation; Represents the minimum loss function value in the previous step sliding window, and the step sliding window includes M training steps.

As an optional embodiment, the preset change rate threshold includes:

;

in, is the decay function related to the number of training steps t, is a hyperparameter.

As an optional embodiment, the preset compression method includes a hybrid gradient compression method that combines a gradient quantization method with a gradient sparsification method.

As an optional embodiment, reducing the current gradient compression degree of the preset compression method according to the first preset rule includes:

According to the compression degree reduction relation, the current gradient compression degree of the preset compression method is reduced;

The compression reduction relationships include:

clip _upper (2Q(t))·λS(t);

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _upper () represents the clip function used to perform the gradient quantization upper limit clipping operation, λ is a preset adjustment parameter, λ＞1.

As an optional embodiment, amplifying the current gradient compression degree of the preset compression method according to the second preset rule includes:

According to the compression degree amplification relation, amplify the current gradient compression degree of the preset compression method;

The compression degree amplification relationship includes:

;

Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _lower () represents the clip function used to perform the gradient quantization lower limit clipping operation, λ is a preset adjustment parameter, λ＞1.

As an optional embodiment, the first determination module 41 includes:

The second judgment submodule is used to judge whether compression of gradient data will cause iterative training to fail to converge at any training step after the iterative training of the local model starts, based on the gradient data of the current training step, and if so, trigger the fifth judgment module; if not, trigger the sixth judgment module;

A fifth determination module, used to determine whether the iterative training is in a warm-up stage;

A sixth determination module, used to determine whether the warm-up phase of the iterative training is over;

The third judgment submodule is used to determine whether the current model performance optimization rate meets the standard after obtaining the gradient data of the current training step for any training step after the warm-up stage of the iterative training of the local model.

As an optional embodiment, the second judgment submodule includes:

The second determination module is used to determine the variance of the gradient data of the current training step;

The fourth judgment submodule judges whether the variance of the gradient data of the current training step is greater than a preset variance threshold. If so, the seventh judgment module is triggered; if not, the eighth judgment module is triggered;

A seventh determination module is used to determine whether compression of gradient data will cause iterative training to fail to converge;

The eighth determination module is used to determine whether compression of gradient data will cause iterative training to fail to converge.

As an optional embodiment, the gradient compression device further includes:

The third judgment module is used for, if the current model performance optimization rate does not meet the standard, adding one to the number of times the optimization rate fails to meet the standard continuously, and judging whether the number of times the optimization rate fails to meet the standard continuously reaches a first preset number threshold, and if so, triggering the first prompt module;

The first prompt module is used to control the prompter to prompt that the number of times the optimization rate fails to meet the standard is too high;

The first clearing module is used to clear the number of consecutive times that the optimization rate fails to meet the standard if the current model performance optimization rate meets the standard.

As an optional embodiment, the gradient compression device further includes:

The fourth judgment module is used for, if the current single-step training duration exceeds the standard, increasing the number of times the single-step training duration exceeds the standard by one, and judging whether the number of times the single-step training duration exceeds the standard continuously reaches a second preset number threshold, and if so, triggering the second prompt module;

The second prompt module is used to control the prompter to prompt that the single-step training duration exceeds the limit for a continuous number of times;

The second clearing module is used to clear the number of times the single-step training duration exceeds the standard if the current single-step training duration does not exceed the standard.

As an optional embodiment, the gradient compression device further includes:

The modification module is used to modify the standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration in response to the standard modification instruction.

As an optional embodiment, the second determination module 43 includes:

A third determination module is used to determine the training duration of the previous training step as the current single-step training duration;

The fifth judgment submodule is used to judge whether the current single-step training duration is greater than a preset duration threshold, and if so, the ninth judgment module is triggered; if not, the tenth judgment module is triggered;

A ninth determination module, used to determine whether the current single-step training duration exceeds the limit;

The tenth determination module is used to determine whether the current single-step training duration has not exceeded the limit.

As an optional embodiment, the gradient compression device further includes:

A fifth judgment module, used to judge whether the gradient distortion degree of the gradient data of the current training step compressed based on the current gradient compression degree exceeds the standard, and if the gradient distortion degree exceeds the standard, triggering the first adjustment module 42;

The triggering conditions of the second judgment module 43 include:

If the current model performance optimization rate meets the standard and the gradient distortion does not exceed the standard.

As an optional embodiment, the fifth judgment module includes:

A fourth determination module is used to determine the gradient distortion degree according to the gradient distortion degree determination relationship, and determine the gradient distortion degree for compressing the gradient data of the current training step based on the current gradient compression degree;

an eleventh determination module, configured to determine that the gradient distortion exceeds a standard if the gradient distortion is greater than a preset distortion threshold;

A twelfth determination module, configured to determine that the gradient distortion degree does not exceed the standard if the gradient distortion degree is not greater than a preset distortion degree threshold;

The gradient distortion determination relationship includes:

;

Wherein, g _GC is the gradient data of the current training step after compression based on the current gradient compression degree, and g is the gradient data of the current training step without compression. represents the Euclidean distance between g _GC and g.

As an optional embodiment, the gradient compression device further includes:

a sixth judgment module, configured to, if the gradient distortion exceeds the standard, increase the number of times the gradient distortion exceeds the standard continuously by one, and judge whether the number of times the gradient distortion exceeds the standard continuously reaches a third preset number threshold, and if so, trigger the third prompt module;

The third prompting module is used to control the prompter to prompt that the gradient distortion degree exceeds the standard for a continuous number of times;

The third clearing module is used to clear the number of times the gradient distortion degree exceeds the standard continuously if the gradient distortion degree does not exceed the standard.

For an introduction to the gradient compression device provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the gradient compression method, and the embodiment of the present invention will not be described in detail here.

To better illustrate the embodiment of the present invention, please refer to FIG. 5 , which is a schematic diagram of the structure of a gradient compression device provided by the present invention. The gradient compression device includes:

A memory 51, used for storing computer programs;

The processor 52 is used to implement the steps of the gradient compression method in the above-mentioned embodiment when executing the computer program.

For an introduction to the gradient compression device provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the gradient compression method, and the embodiment of the present invention will not be described in detail here.

The present invention also provides a distributed cluster, comprising a plurality of gradient compression devices as described in the aforementioned embodiments.

For an introduction to the distributed cluster provided in the embodiment of the present invention, please refer to the aforementioned embodiment of the gradient compression method, and the embodiment of the present invention will not be described in detail here.

To better illustrate an embodiment of the present invention, please refer to FIG. 6 , which is a schematic diagram of the structure of a computer-readable storage medium provided by the present invention. A computer program 61 is stored on the computer-readable storage medium 60. When the computer program 61 is executed by the processor 52, the steps of the gradient compression method in the aforementioned embodiment are implemented.

For an introduction to the computer-readable storage medium provided in an embodiment of the present invention, please refer to the aforementioned embodiment of the gradient compression method, and the embodiment of the present invention will not be described in detail here.

In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should also be noted that in this specification, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "includes", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the element.

The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims (18)

1. A gradient compression method, characterized in that it is applied to any computing node in a distributed cluster, comprising: For any training step after the warm-up phase of the local model iterative training, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard; If the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule; If the current model performance optimization rate meets the standard, determine whether the current single-step training time exceeds the standard; If the current single-step training duration exceeds the limit, the current gradient compression degree of the preset compression method is amplified according to the second preset rule; Based on the latest gradient compression degree, compressing the gradient data of the current training step using a preset compression method so as to synchronize the compressed gradient data within the distributed cluster; Among them, the single-step training duration is related to the degree of gradient compression and network conditions; Judging whether the current model performance optimization rate meets the standard includes: Determine whether the improvement in model performance has reached the target within the previous step sliding window; If the improvement of model performance meets the standard, it is determined that the current model performance optimization rate meets the standard; If the improvement of model performance does not meet the standard, it is determined that the current model performance optimization rate does not meet the standard; Wherein, the step number sliding window includes a preset number of training steps; Determining whether the model performance improvement in the previous step sliding window meets the standard includes: Determine the relationship based on the loss function change rate to determine the loss function change rate of the local model within the previous step sliding window; If the loss function change rate is less than the preset change rate threshold, it is determined that the improvement in model performance has not reached the target; If the change rate of the loss function is not less than the preset change rate threshold, it is determined that the improvement of the model performance meets the standard; The loss function change rate determination relationship includes: ; in, It represents the rate of change of the loss function L in the sliding window of the previous step, based on the current t-th training step; is the loss function of the local model; It represents the sliding average of the loss function L in the sliding window of the previous step based on the current t-th training step, and τ is the variable in the summation operation; Represents the minimum loss function value in the previous step sliding window, and the step sliding window includes M training steps. 2. The gradient compression method according to claim 1, wherein the preset change rate threshold comprises: ; in, is the decay function related to the number of training steps t, is a hyperparameter. 3. The gradient compression method according to claim 1, characterized in that the preset compression method comprises a hybrid gradient compression method combining a gradient quantization method and a gradient sparsification method. 4. The gradient compression method according to claim 3, wherein reducing the current gradient compression degree of the preset compression method according to the first preset rule comprises: According to the compression degree reduction relation, the current gradient compression degree of the preset compression method is reduced; The compression reduction relational expression includes: clip _upper (2Q(t))·λS(t); Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _upper () represents the clip function used to perform the gradient quantization upper limit clipping operation, λ is a preset adjustment parameter, λ＞1. 5. The gradient compression method according to claim 3, wherein amplifying the current gradient compression degree of the preset compression method according to the second preset rule comprises: According to the compression degree amplification relation, amplify the current gradient compression degree of the preset compression method; The compression degree amplification relational expression includes: ; Among them, Q(t) represents the gradient quantization strategy function related to the number of training steps t, S(t) represents the gradient sparsification strategy function related to the number of training steps t, clip _lower () represents the clip function used to perform the gradient quantization lower limit clipping operation, λ is a preset adjustment parameter, λ＞1. 6. The gradient compression method according to claim 1, characterized in that, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, determining whether the current model performance optimization rate meets the standard comprises: At any training step after the iterative training of the local model starts, based on the gradient data of the current training step, determine whether compression of the gradient data will cause the iterative training to fail to converge; If it causes the iterative training to fail to converge, the iterative training is determined to be in the warm-up stage; If it does not cause the iterative training to fail to converge, the warm-up phase of the iterative training is determined to be over; For any training step after the warm-up phase of the local model iterative training, after obtaining the gradient data of the current training step, determine whether the current model performance optimization rate meets the standard. 7. The gradient compression method according to claim 6, wherein judging, based on the gradient data of the current training step, whether compression of the gradient data will cause the iterative training to fail to converge comprises: Determine the variance of the gradient data of the current training step; Determine whether the variance of the gradient data of the current training step is greater than the preset variance threshold; If it is greater than, it is determined that the compression of gradient data will cause the iterative training to fail to converge; If it is not greater than, it is determined that the compression of the gradient data will not cause the iterative training to fail to converge. 8. The gradient compression method according to claim 1, characterized in that after determining whether the current model performance optimization rate meets the standard, the gradient compression method further comprises: If the current model performance optimization rate does not meet the standard, the number of consecutive times the optimization rate does not meet the standard is increased by one, and it is determined whether the number of consecutive times the optimization rate does not meet the standard reaches a first preset number threshold; If it is reached, the control prompter will prompt that the optimization rate has not been reached for too many consecutive times; If the current model performance optimization rate meets the standard, the number of consecutive times that the optimization rate fails to meet the standard is reset to zero. 9. The gradient compression method according to claim 1, characterized in that after determining whether the current single-step training duration exceeds the standard, the gradient compression method further comprises: If the current single-step training duration exceeds the standard, the number of times the single-step training duration exceeds the standard continuously is increased by one, and it is determined whether the number of times the single-step training duration exceeds the standard continuously reaches a second preset number threshold; If it is reached, the control prompter will prompt that the single-step training duration has exceeded the limit for too many consecutive times; If the current single-step training duration does not exceed the standard, the number of times the single-step training duration exceeds the standard continuously is reset to zero. 10. The gradient compression method according to claim 1, characterized in that the gradient compression method further comprises: In response to the standard modification instruction, the standard for meeting the model performance optimization rate and/or the standard for exceeding the single-step training duration are modified. 11. The gradient compression method according to claim 1, wherein determining whether the current single-step training duration exceeds the standard comprises: Determine the training duration of the previous training step as the current single-step training duration; Determine whether the current single-step training duration is greater than a preset duration threshold; If it is greater than, it is determined that the current single-step training duration exceeds the limit; If it is not greater than, it is determined that the current single-step training duration does not exceed the limit. 12. The gradient compression method according to any one of claims 1 to 11, characterized in that, for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step, before reducing the current gradient compression degree of the preset compression method according to the first preset rule, the gradient compression method further comprises: Determine whether the gradient distortion of the gradient data of the current training step is exceeded based on the current gradient compression degree; If the current model performance optimization rate does not meet the standard, the current gradient compression degree of the preset compression method is reduced according to the first preset rule, including: If the current model performance optimization rate does not meet the standard and/or the gradient distortion exceeds the standard, reducing the current gradient compression degree of the preset compression method according to the first preset rule; If the current model performance optimization rate meets the standard, the following steps are used to determine whether the current single-step training duration exceeds the standard: If the current model performance optimization rate meets the standard and the gradient distortion does not exceed the standard, determine whether the current single-step training duration exceeds the standard. 13. The gradient compression method according to claim 12, wherein judging whether the gradient distortion degree of the gradient data of the current training step compressed based on the current gradient compression degree exceeds the standard comprises: Determine a relationship based on the gradient distortion, and determine the gradient distortion for compressing the gradient data of the current training step based on the current gradient compression degree; If the gradient distortion is greater than a preset distortion threshold, it is determined that the gradient distortion exceeds the standard; If the gradient distortion is not greater than the preset distortion threshold, it is determined that the gradient distortion does not exceed the standard; The gradient distortion determination relationship includes: ; Wherein, g _GC is the gradient data of the current training step after compression based on the current gradient compression degree, and g is the gradient data of the current training step without compression. represents the Euclidean distance between g _GC and g. 14. The gradient compression method according to claim 12, characterized in that after determining whether the gradient distortion degree of the gradient data of the current training step is compressed based on the current gradient compression degree exceeds the standard, the gradient compression method further comprises: If the gradient distortion exceeds the standard, the number of times the gradient distortion exceeds the standard continuously is increased by one, and it is determined whether the number of times the gradient distortion exceeds the standard continuously reaches a third preset number threshold; If it is reached, the control prompter will prompt that the gradient distortion has exceeded the limit for too many times in a row; If the gradient distortion does not exceed the standard, the number of times the gradient distortion exceeds the standard continuously is reset to zero. 15. A gradient compression device, characterized in that it is applied to any computing node in a distributed cluster, comprising: The first judgment module is used for determining whether the current model performance optimization rate meets the standard for any training step after the warm-up phase of the iterative training of the local model, after obtaining the gradient data of the current training step. If the current model performance optimization rate does not meet the standard, the first adjustment module is triggered; if the current model performance optimization rate meets the standard, the second judgment module is triggered; The first adjustment module is used to reduce the current gradient compression degree of the preset compression method according to the first preset rule; The second judging module is used to judge whether the current single-step training duration exceeds the standard, and if the current single-step training duration exceeds the standard, trigger the second regulating module; The second adjustment module is used to amplify the current gradient compression degree of the preset compression method according to a second preset rule; A compression module, used to compress the gradient data of the current training step using a preset compression method based on the latest gradient compression degree, so as to synchronize the compressed gradient data within the distributed cluster; Among them, the single-step training duration is related to the degree of gradient compression and network conditions; The first judgment module includes: The first judgment submodule is used to judge whether the improvement of the model performance in the previous step number sliding window meets the standard. If the improvement of the model performance meets the standard, the first judgment module is triggered; if the improvement of the model performance does not meet the standard, the second judgment module is triggered; The first determination module is used to determine whether the current model performance optimization rate meets the standard; The second determination module is used to determine that the current model performance optimization rate does not meet the standard; Wherein, the step number sliding window includes a preset number of training steps; The first judgment submodule includes: A first determination module is used to determine a relationship according to the loss function change rate, and determine the loss function change rate of the local model within the previous step number sliding window; A third determination module is used to determine that the improvement of model performance does not meet the standard if the loss function change rate is less than a preset change rate threshold; A fourth determination module is used to determine whether the improvement of model performance meets the standard if the loss function change rate is not less than a preset change rate threshold; The loss function change rate determination relationship includes: ; in, It represents the rate of change of the loss function L in the sliding window of the previous step, based on the current t-th training step; is the loss function of the local model; It represents the sliding average of the loss function L in the sliding window of the previous step based on the current t-th training step, and τ is the variable in the summation operation; Represents the minimum loss function value in the previous step sliding window, and the step sliding window includes M training steps. 16. A gradient compression device, comprising: Memory for storing computer programs; A processor, configured to implement the steps of the gradient compression method according to any one of claims 1 to 14 when executing the computer program. 17. A distributed cluster system, characterized by comprising a plurality of gradient compression devices as claimed in claim 16. 18. A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the gradient compression method according to any one of claims 1 to 14 are implemented.