CN111695689B - A natural language processing method, device, equipment and readable storage medium - Google Patents

Abstract

The invention discloses a natural language processing method, which comprises the following steps: receiving natural language information to be processed; inputting the natural language information to be processed into a target natural language processing model; wherein, the target natural language processing model is averaged by the model It is obtained through distributed training of the model aggregation algorithm; the language processing model is used to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed. By applying the technical solutions provided by the embodiments of the present invention, the accuracy and processing efficiency of the natural language processing performed by the language processing model are improved. The invention also discloses a natural language processing device, equipment and storage medium, which have corresponding technical effects.

Description

A natural language processing method, device, equipment and readable storage medium

technical field

The present invention relates to the technical field of deep learning, in particular to a natural language processing method, device, equipment and computer-readable storage medium.

Background technique

At present, deep learning methods have been widely used to solve natural language processing problems. With the deepening of research on natural language processing problems, the deep learning models used are becoming more and more complex, and the training data that the deep learning models rely on are becoming more and more complicated. On the other hand, in natural language processing applications such as machine translation and dialogue generation, as the language environment and training data change, frequent incremental training of the model is required to ensure the performance of the language processing model in natural language processing. Accuracy and Validity. Therefore, in the case of using one computing device, these deep learning algorithms take a long time to complete the training of the deep learning model, which cannot meet the needs of high-speed iteration of the model for natural language processing applications. Therefore, it is necessary to train the language processing model through distributed training on multiple computing devices, which can reduce the time spent on training.

One of the key issues in distributed training is to efficiently complete the synchronization of gradients or model parameters, reduce communication overhead, thereby increasing the utilization rate of computing equipment and speeding up the training progress. For the existing distributed training methods, most of them adopt a synchronous gradient synchronization method to maintain the consistency of model parameters on each node. In this way, each distributed node needs to complete gradient transmission and synchronization in each iteration of training. This distributed training method has been widely used in the distributed training of image classification models, and has achieved good results through experimental verification.

In the current research, the trained deep neural network is mainly used to solve computer vision problems. A typical representative is the deep neural network that solves image classification problems. The characteristic of this type of network is that all gradient data can be used with dense matrix To represent. Few studies have attempted distributed training of language models in natural language processing. The characteristic of this type of model is that part of the gradient data is sparse, and they are often represented by sparse matrices in existing distributed deep learning frameworks. Transferring the gradient data represented by the dense matrix between the computing nodes of the cluster can be more efficiently realized by the existing collective communication library, while the gradient data represented by the sparse matrix is difficult to perform efficient data transmission in the cluster. Therefore, for distributed training of sparse models that appear in fields such as natural language processing, it takes a long time for the deep neural network model to converge to the target progress. In the process of natural language processing, the accuracy of the language processing model for natural language processing is low. The processing efficiency is low.

To sum up, how to effectively solve the problems of low accuracy and low processing efficiency of the existing language processing models for natural language processing is an urgent problem for those skilled in the art.

Contents of the invention

The purpose of the present invention is to provide a natural language processing method, which improves the accuracy and processing efficiency of the language processing model for natural language processing; another purpose of the present invention is to provide a natural language processing device, equipment and computer-based Read storage media.

In order to solve the above technical problems, the present invention provides the following technical solutions:

A natural language processing method comprising:

Receive natural language information to be processed;

Inputting the natural language information to be processed into a target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm;

Using the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the to-be-processed natural language information.

In a specific embodiment of the present invention, the process of obtaining the target natural language processing model by performing model training through the model averaging model aggregation algorithm includes:

Preprocessing the original natural language processing model to obtain the update frequency of each model parameter in the original natural language processing model;

performing a grouping operation on each of the model parameters according to each of the update frequencies to obtain each of the model parameter groups;

Respectively determine the synchronization interval of each of the model parameter groups;

During the model iteration process, each of the model parameters in the corresponding model parameter group of the original natural language processing model in each training node is synchronized according to each of the synchronization intervals to obtain the target natural language processing model.

In a specific embodiment of the present invention, the original natural language processing model is preprocessed to obtain the update frequency of each model parameter in the original natural language processing model, including:

selecting a first preset number of data sets from the training data set;

sequentially input each of the data groups into the original natural language processing model, so as to perform the first preset number of model iteration operations on the original natural language processing model;

For each data group, perform forward calculation and backpropagation on the original natural language processing model to obtain the gradient value of each model parameter in the original natural language processing model;

Using an indicator function to perform statistical operations on the non-zero values in each of the gradient values in each model iteration, to obtain the effective number of gradient updates corresponding to each of the model parameters in each model iteration;

Calculating the ratio of each effective number of gradient updates to the preset number of times to obtain update frequencies corresponding to each of the model parameters.

In a specific embodiment of the present invention, each model parameter is grouped according to each update frequency to obtain each model parameter group, including:

inputting each of said model parameters into a preset sortable container;

Using the sorting container to perform a sorting operation on each of the model parameters according to the magnitude of the update frequency corresponding to each of the model parameters, to obtain a sorting result;

Divide each of the model parameters into a second preset number of model parameter groups according to the ranking result.

In a specific implementation manner of the present invention, determining the synchronization intervals of each of the model parameter groups respectively includes:

For each model parameter group, calculate the average update frequency of the update frequencies of the model parameters in the model parameter group;

The respective synchronization intervals of each of the model parameter groups are calculated according to each of the average update frequencies.

In a specific embodiment of the present invention, during the model iteration process, each of the model parameters in the corresponding model parameter group of the original natural language processing model in each training node is synchronized according to each of the synchronization intervals, include:

selecting a target training node from each of the training nodes;

Initializing each model parameter in the original natural language processing model in each of the target training nodes to obtain an initialization result;

using the target training node to broadcast the initialization result to other training nodes except the target training node;

Initialize the number of model iterations;

During the model iteration process, each of the synchronization intervals is used to perform a remainder operation on the accumulation result of each model iteration number;

A synchronization operation is performed on each of the model parameters in the target model parameter group corresponding to the synchronization interval whose remainder is zero.

In a specific embodiment of the present invention, the synchronization operation is performed on each of the model parameters in the target model parameter group corresponding to the synchronization interval whose remainder is zero, including:

Obtaining target model parameter groups in each of the training nodes respectively;

Carrying out mean calculations for each of the model parameters corresponding to each of the target model parameter groups to obtain each of the average model parameters;

Each model parameter in the target model parameter group in each of the training nodes is set as each of the average model parameters.

A natural language processing device, comprising:

An information receiving module, configured to receive natural language information to be processed;

An information input module, configured to input the natural language information to be processed into a target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm;

An information processing module, configured to use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the to-be-processed natural language information.

A natural language processing device, comprising:

memory for storing computer programs;

A processor, configured to implement the steps of the aforementioned natural language processing method when executing the computer program.

A computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the aforementioned natural language processing method are realized.

Apply the method provided by the embodiment of the present invention to receive the natural language information to be processed; input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is distributed through the model aggregation algorithm of model averaging Obtained through training; use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed. The target natural language processing model is obtained by using the model averaging model aggregation algorithm for distributed training. The model averaging model aggregation algorithm is suitable for training language processing models represented by sparse matrices commonly used in distributed deep learning frameworks, greatly shortening The model training time is shortened, and the accuracy and processing efficiency of the language processing model for natural language processing are improved.

Correspondingly, the embodiments of the present invention also provide a natural language processing device, device, and computer-readable storage medium corresponding to the above-mentioned natural language processing method, which have the above-mentioned technical effects, and will not be repeated here.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention. Those skilled in the art can also obtain other drawings based on these drawings without creative work.

Fig. 1 is a kind of implementation flowchart of the natural language processing method in the embodiment of the present invention;

Fig. 2 is another implementation flowchart of the natural language processing method in the embodiment of the present invention;

FIG. 3 is a structural block diagram of a natural language processing device in an embodiment of the present invention;

Fig. 4 is a structural block diagram of a natural language processing device in an embodiment of the present invention.

Detailed ways

In order to enable those skilled in the art to better understand the solution of the present invention, the present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. Apparently, the described embodiments are only some of the embodiments of the present invention, but not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Embodiment one:

Referring to FIG. 1, FIG. 1 is an implementation flowchart of a natural language processing method in an embodiment of the present invention, and the method may include the following steps:

S101: Receive natural language information to be processed.

When natural language processing is required, the natural language information to be processed is sent to the processing center. The processing center receives the natural language information to be processed. The natural language information to be processed may be a sentence to be translated, a sentence to be generated for a dialogue, and the like.

S102: Input the to-be-processed natural language information into a target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm.

In advance, the model aggregation algorithm of model averaging is used to perform distributed training on the original natural language processing model to obtain the target natural language processing model. After receiving the natural language information to be processed, the processing center inputs the natural language information to be processed into the target natural language processing model. Following the above example, when the natural language information to be processed is a sentence to be translated, the original machine translation model is distributedly trained using the model averaging model aggregation algorithm in advance to obtain the target machine translation model, and the sentence to be translated is input to the target machine translation model Middle; when the natural language information to be processed is the sentence of the dialogue to be generated, the original dialogue generation model is distributedly trained by the model aggregation algorithm of model averaging in advance to obtain the target dialogue generation model, and the sentence to be generated is input into the target dialogue Generate a model.

S103: Use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed.

After inputting the natural language information to be processed into the target natural language processing model, the language processing model is used to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed. Following the above example, when the natural language information to be processed is a sentence to be translated, use the target machine translation model to translate the sentence to be translated; when the natural language information to be processed is a sentence to be generated, use the target dialog generation model to translate the sentence statement to perform dialog generation operations. The model aggregation algorithm of model averaging is suitable for training language processing models represented by sparse matrices commonly used in distributed deep learning frameworks. efficiency.

It should be noted that the target natural language processing model is not limited to processing sentences to be translated and dialogues to be generated, but can also be used for information extraction, text sentiment analysis, personalized recommendation, etc., and pre-train corresponding models according to different application scenarios Target natural language processing model.

Apply the method provided by the embodiment of the present invention to receive the natural language information to be processed; input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is distributed through the model aggregation algorithm of model averaging Obtained through training; use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed. The target natural language processing model is obtained by using the model averaging model aggregation algorithm for distributed training. The model averaging model aggregation algorithm is suitable for training language processing models represented by sparse matrices commonly used in distributed deep learning frameworks, greatly shortening The model training time is shortened, and the accuracy and processing efficiency of the language processing model for natural language processing are improved.

It should be noted that, based on the first embodiment above, the embodiment of the present invention also provides a corresponding improvement solution. In subsequent embodiments, the same steps as in the first embodiment above or corresponding steps may be referred to each other, and the corresponding beneficial effects may also be referred to each other, and will not be repeated in the improved embodiments below.

Embodiment two:

Referring to Fig. 2, Fig. 2 is another implementation flowchart of the natural language processing method in the embodiment of the present invention, and the method may include the following steps:

S201: Preprocessing the original natural language processing model to obtain the update frequency of each model parameter in the original natural language processing model.

In the model iteration process, the original natural language processing model is obtained, and the original natural language processing model is preprocessed to obtain the update frequency of each model parameter in the original natural language processing model.

In a specific implementation manner of the present invention, step S201 may include the following steps:

Step 1: selecting a first preset number of data groups from the training data set;

Step 2: Input each data group into the original natural language processing model in turn, so as to perform a first preset number of model iteration operations on the original natural language processing model;

Step 3: For each data group, perform forward calculation and backpropagation on the original natural language processing model to obtain the gradient value of each model parameter in the original natural language processing model;

Step 4: Use the indicator function to perform statistical operations on the non-zero values in each gradient value in each model iteration, and obtain the effective number of gradient updates corresponding to each model parameter in each model iteration.

Step 5: Calculate the ratio of the effective number of update times of each gradient to the preset number of times, and obtain the update frequency corresponding to each model parameter.

For convenience of description, the above five steps are combined for description.

利用指示函数I对各次模型迭代中各梯度值/>

中的非零值进行统计操作，从而获取在各次模型迭代中各模型参数对应的梯度更新是否有效，得到各所述模型参数分别在各次模型迭代中对应的梯度更新有效次数。指示函数I的定义如下：Select the first preset number of data groups in the pre-acquired training data set, such as selecting m data groups, and input each data group into the original natural language processing model in turn, each data group corresponds to the original natural language processing model. One iteration, so that a first preset number of model iterations are performed on the original natural language processing model. For each data group, perform forward calculation and backpropagation on the original natural language processing model to obtain the gradient value of each model parameter in the original natural language processing model

Each gradient value in each model iteration is calculated by using the indicator function I

Statistical operations are performed on the non-zero values in , so as to obtain whether the gradient update corresponding to each model parameter is valid in each model iteration, and obtain the effective number of gradient updates corresponding to each model parameter in each model iteration. The indicator function I is defined as follows:

时，说明对应的梯度更新是有效的，/>

表示第t次迭代后第i个模型参数的梯度值。when

, indicating that the corresponding gradient update is valid, />

Indicates the gradient value of the i-th model parameter after the t-th iteration.

Calculate the ratio of the effective number of update times of each gradient to the preset number of times, and obtain the update frequency corresponding to each model parameter. The ratio of the effective times of each gradient update to the preset times can be calculated by the following formula, and the update frequency α _i corresponding to each model parameter can be obtained:

S202: Group each model parameter according to each update frequency to obtain each model parameter group.

Since the language processing model is often represented by a sparse matrix in the distributed deep learning framework, after obtaining the update frequency of each model parameter in the original natural language processing model, the model parameters are grouped according to each update frequency, and each model parameter is obtained. Model parameter group.

In a specific implementation manner of the present invention, step S202 may include the following steps:

Step 1: Input each model parameter into a preset sortable container;

Step 2: use the sorting container to sort each model parameter according to the update frequency corresponding to each model parameter, and obtain the sorting result;

Step 3: Divide each model parameter into a second preset number of model parameter groups according to the sorting result.

For convenience of description, the above three steps are combined for description.

Deploy the sortable container in advance, input each model parameter into the preset sortable container, and use the sorting _container according to the update frequency corresponding to each model parameter to sort each The model parameters are sorted and the sorted results are obtained. Each model parameter is divided into a second preset number of model parameter groups according to the sorting result. The second preset number is represented by the preset hyperparameter q, and the i-th group of model parameters with similar sparsity is represented by p _i , and the model parameter set can be expressed as:

P={p ₁ ,p ₂ ,...,p _q };

S203: Determine the synchronization interval of each model parameter group respectively.

After each model parameter group is obtained, the synchronization interval of each model parameter group is determined respectively.

In a specific implementation manner of the present invention, step S203 may include the following steps:

Step 1: For each model parameter group, calculate the average update frequency of the update frequencies of each model parameter in the model parameter group;

Step 2: Calculate the respective synchronization intervals of each model parameter group according to each average update frequency.

For convenience of description, the above two steps are combined for description.

根据各平均更新频率计算各模型参数组分别的同步间隔。具体可以根据以下公式计算平均更新频率/>

After obtaining the update frequency of each model parameter and grouping each model parameter to obtain each model parameter group, for each model parameter group, calculate the average update frequency of the update frequency of each model parameter in the model parameter group

The respective synchronization intervals for each model parameter set are calculated according to each average update frequency. Specifically, the average update frequency can be calculated according to the following formula/>

Among them, || represents 1-norm calculation.

After the synchronization intervals of each model parameter group are respectively determined, the respective synchronization intervals k _i of each model parameter group are calculated according to each average update frequency, and the respective synchronization intervals k _i of each model parameter group can be calculated by the following formula:

Among them, _ki represents the synchronization interval corresponding to the i-th model parameter group, and K represents the synchronization interval set composed of the synchronization intervals corresponding to each group of model parameter groups, and K is expressed as:

Wherein, λ is a synchronization interval setting coefficient, which can facilitate the inventors to dynamically adjust the synchronization intervals of different sparse deep neural network models based on prior knowledge.

S204: During the model iteration process, perform synchronous operations on each model parameter in the corresponding model parameter group of the original natural language processing model in each training node according to each synchronization interval, to obtain a target natural language processing model.

During the model iteration process, each model parameter in the corresponding model parameter group of the original natural language processing model in each training node is synchronized according to each synchronization interval to obtain the target natural language processing model.

In a specific implementation manner of the present invention, step S204 may include the following steps:

Step 1: Select the target training node from each training node;

Step 2: Initialize each model parameter in the original natural language processing model in each target training node, and obtain the initialization result;

Step 3: Use the target training node to broadcast the initialization result to other training nodes except the target training node;

Step 4: Initialize the number of model iterations;

Step 5: During the model iteration process, each synchronization interval is used to perform a remainder operation on the cumulative result of each model iteration number;

Step 6: Synchronize each model parameter in the target model parameter group corresponding to the synchronization interval whose remainder is zero, and obtain the target natural language processing model.

For convenience of description, the above six steps are combined for description.

A target training node is selected from each training node, and each model parameter in the original natural language processing model in each target training node is initialized to obtain an initialization result.

The target training node can be any one of the training nodes. For example, each training node can be numbered in advance, and the numbers are 0, 1..., and the training node numbered 0 is selected as the target training node. Initialize each model parameter in the original natural language processing model in each target training node to obtain an initialization result. The target training node is used to broadcast the initialization result to other training nodes except the target training node. Initialize the number of model iterations. For example, before performing iterative training on the original natural language processing model, initialize the number of iterations to 0. During the model iteration process, use each synchronization interval to take the remainder of the accumulated results of each model iteration number t operation, can be calculated by the following formula:

t modk _i ;

The synchronization operation is performed on each model parameter in the target model parameter group corresponding to the synchronization interval whose remainder is zero, to obtain the target natural language processing model.

In a specific embodiment of the present invention, the synchronous operation of each model parameter in the target model parameter group corresponding to the synchronous interval whose remainder is zero may include the following steps:

Step 1: Obtain the target model parameter groups in each training node respectively;

Step 2: Calculate the mean value of each model parameter corresponding to each target model parameter group to obtain each average model parameter;

Step 3: Set each model parameter in the target model parameter group in each training node as each average model parameter to obtain the target natural language processing model.

For convenience of description, the above three steps are combined for description.

Obtain the target model parameter groups in each training node respectively, calculate the mean value of each model parameter corresponding to each target model parameter group, obtain each average model parameter, and set each model parameter in the target model parameter group in each training node is each average model parameter to obtain the target natural language processing model.

Steps S201 to S203 complete the solution of the key parameters P and K, and steps S201 to S203 can be obtained by using only one training node for offline calculation on a single machine. Since step S201 to step S203 do not train the model and do not need to modify the model parameters, only the corresponding gradient values need to be calculated according to different batches of training data, so the time-consuming is much shorter than that of general stand-alone training. A single server equipped with an Intel Xeon CPU and 4 Nvidia RTX 2080Ti GPUs can be selected, and one GPU in the server can be used to complete steps S201 to S203 for the LM1b model, which takes a short time. From experience, the time consumption of steps S201 to S203 is negligible relative to the overall training time.

Compared with the existing distributed training mode of the language processing model, the method of the present invention is used to carry out the distributed training of the language processing model on a single server or two servers, which greatly shortens the training time.

S205: Receive natural language information to be processed.

S206: Input the to-be-processed natural language information into the target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm.

S207: Use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed.

Corresponding to the above method embodiments, an embodiment of the present invention also provides a natural language processing device, and the natural language processing device described below and the natural language processing method described above can be referred to in correspondence.

Referring to FIG. 3, FIG. 3 is a structural block diagram of a natural language processing device in an embodiment of the present invention, which may include:

An information receiving module 31, configured to receive natural language information to be processed;

The information input module 32 is used to input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm;

The information processing module 33 is configured to use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed.

Apply the device provided by the embodiment of the present invention to receive the natural language information to be processed; input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is distributed through the model aggregation algorithm of model averaging Obtained through training; use the language processing model to perform corresponding natural language understanding or natural language generation operations on the natural language information to be processed. The target natural language processing model is obtained by using the model averaging model aggregation algorithm for distributed training. The model averaging model aggregation algorithm is suitable for training language processing models represented by sparse matrices commonly used in distributed deep learning frameworks, greatly shortening The model training time is shortened, and the accuracy and processing efficiency of the language processing model for natural language processing are improved.

In a specific embodiment of the present invention, the device includes a model training module, and the model training module includes:

The parameter update frequency acquisition sub-module is used to preprocess the original natural language processing model to obtain the update frequency of each model parameter in the original natural language processing model;

The parameter group acquisition sub-module is used to group each model parameter according to each update frequency to obtain each model parameter group;

The synchronization interval determination submodule is used to determine the synchronization interval of each model parameter group respectively;

The parameter synchronization sub-module is used for synchronizing each model parameter in the corresponding model parameter group of the original natural language processing model in each training node according to each synchronization interval during the model iteration process to obtain the target natural language processing model.

In a specific embodiment of the present invention, the parameter update frequency obtaining submodule includes:

a data group selection unit, configured to select a first preset number of data groups from the training data set;

A model iteration unit, configured to sequentially input each data group into the original natural language processing model, so as to perform a first preset number of model iteration operations on the original natural language processing model;

The gradient value obtaining unit is used to perform forward calculation and backpropagation on the original natural language processing model for each data group, and obtain the gradient value of each model parameter in the original natural language processing model;

The effective number of times obtaining unit is used to perform statistical operations on the non-zero values in each gradient value in each model iteration by using the indicator function, and obtain the corresponding gradient update effective times of each model parameter in each model iteration;

The update frequency acquisition unit is used to calculate the ratio of the effective number of update times of each gradient to the preset number of times, and obtain the update frequency corresponding to each model parameter.

In a specific embodiment of the present invention, the parameter set obtaining submodule includes:

A parameter input unit, configured to input each model parameter into a preset sortable container;

The sorting result obtaining unit is used to use the sorting container to sort each model parameter according to the size of the update frequency corresponding to each model parameter, and obtain the sorting result;

A parameter group dividing unit, configured to divide each model parameter into a second preset number of model parameter groups according to the sorting result.

In a specific embodiment of the present invention, the synchronization interval determination submodule includes:

An average update frequency calculation unit, for each model parameter group, to calculate the average update frequency of the update frequencies of the model parameters in the model parameter group;

The synchronization interval calculation unit is configured to calculate the respective synchronization intervals of each model parameter group according to each average update frequency.

In a specific embodiment of the present invention, the parameter synchronization submodule includes:

A node selection unit is used to select a target training node from each training node;

The parameter initialization unit is used to initialize each model parameter in the original natural language processing model in each target training node, and obtain the initialization result;

A broadcast unit, configured to use the target training node to broadcast the initialization result to other training nodes except the target training node;

The number of iterations initialization unit is used to initialize the number of iterations of the model;

A remainder unit is used to perform a remainder operation on the accumulation result of each model iteration number by using each synchronization interval during the model iteration process;

The parameter synchronization unit is configured to perform a synchronization operation on each model parameter in the target model parameter group corresponding to the synchronization interval whose remainder is zero.

In a specific implementation manner of the present invention, the parameter synchronization unit includes:

The parameter group obtaining subunit is used to respectively obtain the target model parameter group in each training node;

The parameter average value obtaining subunit is used to calculate the average value of each corresponding model parameter in each target model parameter group to obtain each average model parameter;

The parameter setting subunit is used to set each model parameter in the target model parameter group in each training node as each average model parameter.

Corresponding to the above method embodiment, see FIG. 4, which is a schematic diagram of a natural language processing device provided by the present invention, which may include:

Memory 41, used to store computer programs;

When the processor 42 is used to execute the computer program stored in the memory 41, the following steps can be realized:

Receive the natural language information to be processed; input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is obtained through distributed training through the model averaging model aggregation algorithm; use the target natural language processing model to treat Process natural language information to perform corresponding natural language understanding or natural language generation operations.

For the introduction of the device provided by the present invention, please refer to the above method embodiment, and the present invention will not repeat it here.

Corresponding to the above method embodiments, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the following steps can be implemented:

Receive the natural language information to be processed; input the natural language information to be processed into the target natural language processing model; wherein, the target natural language processing model is obtained through distributed training through the model averaging model aggregation algorithm; use the target natural language processing model to treat Process natural language information to perform corresponding natural language understanding or natural language generation operations.

The computer-readable storage medium may include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc., which can store program codes. medium.

For the introduction of the computer-readable storage medium provided by the present invention, please refer to the foregoing method embodiments, and the present invention will not repeat them here.

Each embodiment in this specification is described in a progressive manner, each embodiment focuses on the difference from other embodiments, and the same or similar parts of each embodiment can be referred to each other. As for the device, equipment and computer-readable storage medium disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and for relevant details, please refer to the description of the method part.

In this paper, specific examples are used to illustrate the principles and implementation methods of the present invention, and the descriptions of the above embodiments are only used to help understand the technical solutions and core ideas of the present invention. It should be pointed out that for those skilled in the art, without departing from the principle of the present invention, some improvements and modifications can be made to the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

Claims (8)

1. A natural language processing method, characterized in that, comprising: Receive natural language information to be processed; Inputting the natural language information to be processed into a target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm; Using the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the to-be-processed natural language information; Wherein, the process of obtaining the target natural language processing model by performing model training through the model averaging model aggregation algorithm includes: Preprocessing the original natural language processing model to obtain the update frequency of each model parameter in the original natural language processing model; performing a grouping operation on each of the model parameters according to each of the update frequencies to obtain each of the model parameter groups; Respectively determine the synchronization interval of each of the model parameter groups; During the model iteration process, perform synchronous operations on each of the model parameters in the corresponding model parameter groups of the original natural language processing model in each training node according to each of the synchronization intervals, to obtain the target natural language processing model; Wherein, each of the model parameters is grouped according to each of the update frequencies to obtain each of the model parameter groups, including: inputting each of said model parameters into a preset sortable container; Using the sorting container to perform a sorting operation on each of the model parameters according to the magnitude of the update frequency corresponding to each of the model parameters, to obtain a sorting result; Divide each of the model parameters into a second preset number of model parameter groups according to the ranking result. 2. The natural language processing method according to claim 1, wherein the original natural language processing model is preprocessed to obtain the update frequency of each model parameter in the original natural language processing model, including: selecting a first preset number of data sets from the training data set; sequentially input each of the data groups into the original natural language processing model, so as to perform the first preset number of model iteration operations on the original natural language processing model; For each data group, perform forward calculation and backpropagation on the original natural language processing model to obtain the gradient value of each model parameter in the original natural language processing model; Using an indicator function to perform statistical operations on the non-zero values in each of the gradient values in each model iteration, to obtain the effective number of gradient updates corresponding to each of the model parameters in each model iteration; Calculating the ratio of each effective number of gradient updates to the preset number of times to obtain update frequencies corresponding to each of the model parameters. 3. the natural language processing method according to claim 1, is characterized in that, determines the synchronous interval of each described model parameter group respectively, comprises: For each model parameter group, calculate the average update frequency of the update frequencies of the model parameters in the model parameter group; The respective synchronization intervals of each of the model parameter groups are calculated according to each of the average update frequencies. 4. according to the natural language processing method described in any one of claim 1 to 3, it is characterized in that, in model iterative process, according to each described synchronization interval to the corresponding model of the original natural language processing model in each training node The model parameters in the parameter group are synchronously operated, including: selecting a target training node from each of the training nodes; Initializing each model parameter in the original natural language processing model in each of the target training nodes to obtain an initialization result; using the target training node to broadcast the initialization result to other training nodes except the target training node; Initialize the number of model iterations; During the model iteration process, each of the synchronization intervals is used to perform a remainder operation on the accumulation result of each model iteration number; A synchronization operation is performed on each of the model parameters in the target model parameter group corresponding to the synchronization interval whose remainder is zero. 5. the natural language processing method according to claim 4, is characterized in that, carry out synchronous operation to each described model parameter in the target model parameter group corresponding to the synchronous interval that remainder is zero, comprising: Obtaining target model parameter groups in each of the training nodes respectively; Carrying out mean calculations for each of the model parameters corresponding to each of the target model parameter groups to obtain each of the average model parameters; Each model parameter in the target model parameter group in each of the training nodes is set as each of the average model parameters. 6. A natural language processing device, characterized in that, comprising: An information receiving module, configured to receive natural language information to be processed; An information input module, configured to input the natural language information to be processed into a target natural language processing model; wherein, the target natural language processing model is obtained by performing distributed training through a model averaging model aggregation algorithm; An information processing module, configured to use the target natural language processing model to perform corresponding natural language understanding or natural language generation operations on the to-be-processed natural language information; Wherein, the device includes a model training module, and the model training module includes: The parameter update frequency acquisition sub-module is used to preprocess the original natural language processing model to obtain the update frequency of each model parameter in the original natural language processing model; The parameter group acquisition sub-module is used to group each model parameter according to each update frequency to obtain each model parameter group; The synchronization interval determination submodule is used to determine the synchronization interval of each model parameter group respectively; The parameter synchronization sub-module is used for synchronizing each model parameter in the corresponding model parameter group of the original natural language processing model in each training node according to each synchronization interval during the model iteration process to obtain the target natural language processing model; Wherein, the parameter group obtaining submodule performs grouping operation on each of the model parameters according to each of the update frequencies to obtain each model parameter group, including: inputting each of the model parameters into a preset sortable container; Using the sorting container to perform a sorting operation on each of the model parameters according to the update frequency corresponding to each of the model parameters to obtain a sorting result; divide each of the model parameters into a second preset number according to the sorting result set of model parameters. 7. A natural language processing device, comprising: memory for storing computer programs; A processor configured to implement the steps of the natural language processing method according to any one of claims 1 to 5 when executing the computer program. 8. A computer-readable storage medium, characterized in that, a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the natural language described in any one of claims 1 to 5 is realized. The steps of the processing method.

Images