CN117688176A - A pseudo-language family clustering method and device based on multi-language pre-trained large model - Google Patents

Abstract

The invention relates to the technical field of text machine translation, and in particular refers to a pseudo-language family clustering method and device based on a large multi-language pre-training model. The method includes: establishing a shared language pool; based on a large multi-language pre-training model, calculating the shared language The Fisher information matrix of the language pairs in the pool is used to obtain the representation results of the language pairs in the shared language pool; the similarity between the language pairs is calculated based on the representation results to obtain the similarity value; based on the similarity value, the language pair is Sort the similarities between them, select auxiliary language pairs that meet the boundary values according to the preset boundary values, and complete pseudo-language family clustering based on the multi-language pre-trained large model. This invention uses the ability of multi-language pre-training itself to characterize language pairs, more effectively selects and clusters auxiliary languages and improves its generalization between different models and data sets, and ultimately improves the multi-language performance of low-resource language pairs. Translation quality under collaborative training.

Description

A pseudo-language family clustering method and device based on multi-language pre-trained large model

Technical field

The present invention relates to the technical field of machine translation, and in particular refers to a pseudo-language family clustering method and device based on a multi-language pre-trained large model.

Background technique

Neural machine translation (NMT) has become the dominant machine translation (MT) paradigm in academic research and commercial use. In recent years, research has found that the NMT framework can naturally integrate multiple languages. As a result, research efforts involving MT systems in multiple languages have increased dramatically. Researchers call NMT systems that handle translation for more than one language pair multilingual NMT (MNMT) systems. The ultimate goal of MNMT research is to develop a single model for translation between as many languages as possible by efficiently utilizing available language resources. Although MNMT brings promising improvements in translation quality, these models rely on massively parallel corpora. Since such corpora only exist for a few language pairs, translation performance falls far short of expectations in most low-resource languages. Relevant research shows that for low-resource language translation, by introducing additional auxiliary language pairs in the fine-tuning stage and performing multi-language collaborative training, it can outperform traditional fine-tuning methods in some cases. However, follow-up research further pointed out that collaborative training does not always bring positive effects, and sometimes may even lead to a decrease in translation quality, depending on the choice of collaborative language pairs.

In recent years, research at home and abroad has shown that by fine-tuning the model using language pairs that are similar to the target language, the translation quality of the target language pair can be improved without using the data of the target language pair, further illustrating that language pairs There is synergy between them. However, not all collaborative training of any language pair can achieve the same effect. Therefore, the screening of collaborative language pairs has become a key step to improve the translation quality of MNMT in low-resource language pairs. Languages in a language family usually have a common geographical and linguistic background, so there will be more similar meanings at the character or word level. From a linguistic perspective, these language pairs will have more identical or similar meanings. Linguistic features such as characters and grammar. Current academic research in this field is mainly divided into two directions: on the one hand, researchers usually integrate different prior knowledge, including language similarity, resource availability, language type and task-specific requirements; on the other hand, researchers Try to apply language embedding, use embedding vectors to represent each language, and cluster them in the embedding space, such as adding a language embedding (LanguageEmbedding) layer to the model, and building embedding vectors for each language pair after multi-language training. Then build a language family through hierarchical clustering to improve the translation quality of language pairs, or embed an adapter structure in the structure of the model while keeping the parameters of the pre-trained model unchanged, and train the language family in downstream tasks. Adapter to improve translation quality.

Although these methods can improve the translation quality of language pairs, they also face certain difficulties in practical application. In particular, training new models or changing the structure of the model will make these methods complicated, and these methods are also difficult to reproduce when the original structure and data of large language models are difficult to obtain.

Contents of the invention

In order to solve the technical problem in the existing technology that training a new model or changing the structure of the model will make these methods complicated and difficult to reproduce when the original structure and data of a large language model are difficult to obtain, the present invention is implemented The example provides a pseudo-language family clustering method and device based on a multi-language pre-trained large model. The technical solutions are as follows:

On the one hand, a pseudo-language family clustering method based on a large multi-language pre-trained model is provided. The method is implemented by a pseudo-language family clustering device based on a large multi-language pre-trained model. The method includes:

S1. Establish a shared language pool;

S2 is based on the large multi-language pre-training model, calculates the Fisher information matrix of the language pairs in the shared language pool, and obtains the representation results of the language pairs in the shared language pool;

S3. Calculate the similarity between language pairs based on the representation results and obtain the similarity value;

S4. Sort the similarities between language pairs according to the similarity value, select auxiliary language pairs that meet the boundary value according to the preset boundary value, and complete pseudo-language family clustering based on the multi-language pre-trained large model.

Optionally, in step S1, establish a shared language pool, including:

Get TED data set;

Extract multiple languages from the TED data set, and use the language pairs translated from multiple languages into English as the basic data set to establish a shared language pool.

Optionally, in step S2, based on the multi-language pre-trained large model, calculate the Fisher information matrix of the language pairs in the shared language pool, and obtain the representation results of the language pairs in the shared language pool, including:

Obtain a parallel corpus corresponding to the language pairs in the shared language pool, and equalize the data in the parallel corpus into j mini-batch data sets;

Input the small batch data sets into the multi-language pre-trained large model in turn, and output the Fisher information matrix of each small batch data set;

After an input round, the average Fisher information matrix of each mini-batch data set is calculated, and the average Fisher information matrix is used as an estimate to obtain the Fisher information weight of each mini-batch data set;

Characterize the distribution of corresponding language pairs in the shared language pool according to Fisher information weights.

Optionally, in step S3, calculate the similarity between the language pairs based on the characterization results to obtain the similarity value, including:

Obtain characterization results;

The mean square error method is used to calculate the distance between the target language pair and the auxiliary language pair. The closer the distance is, the higher the similarity.

Optionally, in step S3, calculate the similarity between the language pairs based on the characterization results to obtain the similarity value, including:

Use the Fisher information matrix to calculate the KL divergence from the auxiliary language to the target language, and obtain the distance between the target language pair and the auxiliary language pair. The closer the distance is, the higher the similarity.

Optionally, in step S3, calculate the similarity between the language pairs based on the characterization results to obtain the similarity value, including:

Create a Fisher information mask by selecting the first K parameters and assigning them the value 1, while the remaining parameters are assigned the value 0;

Based on the number of parameters activated simultaneously and the number of parameters activated in the target direction, the distance between the target language pair and the auxiliary language pair is calculated. The closer the distance is, the higher the similarity.

Optionally, in step S4, the similarities between the language pairs are sorted according to the similarity value, and the auxiliary language pairs that meet the boundary value are selected according to the preset boundary value to complete the pseudo-language family based on the multi-language pre-training large model. Clustering, including:

Traversely calculate the similarity between all language pairs;

Sort in descending order according to similarity between language pairs;

Preset the initial search radius and delineate the boundary range based on the initial search radius;

Integrate the closest language pairs within the boundary into the auxiliary language list;

Update the search radius based on the similarity between the newly added language pair and the target language pair;

The search radius is updated repeatedly until new language pairs are no longer expanded, the clustered pseudo-language family is obtained, and the pseudo-language family clustering based on the multi-language pre-trained large model is completed.

On the other hand, a pseudo-language family clustering device based on a large multi-language pre-trained model is provided. The device is applied to a pseudo-language family clustering method based on a large multi-language pre-trained model. The device includes:

Language pool module, used to establish a shared language pool;

The representation module is used to calculate the Fisher information matrix of the language pairs in the shared language pool based on the multi-language pre-trained large model, and obtain the representation results of the language pairs in the shared language pool;

The similarity calculation module is used to calculate the similarity between language pairs based on the representation results and obtain the similarity value;

The clustering module is used to sort the similarities between language pairs according to the similarity value, select auxiliary language pairs that meet the boundary value according to the preset boundary value, and complete pseudo-language family clustering based on the multi-language pre-trained large model. .

On the other hand, a pseudo-language family clustering device based on a large multi-language pre-training model is provided. The pseudo-language family clustering device based on a large multi-language pre-training model includes: a processor; a memory, and the memory stores There are computer readable instructions. When the computer readable instructions are executed by the processor, any one of the above-mentioned pseudo-language family clustering methods based on multi-language pre-trained large models is implemented.

On the other hand, a computer-readable storage medium is provided. At least one instruction is stored in the storage medium. The at least one instruction is loaded and executed by a processor to implement the above-mentioned pseudo-language family based on a multi-language pre-trained large model. Any of the clustering methods.

The beneficial effects brought by the technical solutions provided by the embodiments of the present invention include at least:

In view of the limitations in the prior art that require additional prior knowledge or the need to modify the model architecture, the present invention provides a method for constructing a more effective language pair clustering method for multi-language collaborative training. The core goal is to use the ability of multi-language pre-training itself to characterize language pairs, more effectively select and cluster auxiliary languages and improve their generalization between different models and data sets, and ultimately improve the performance of low-resource language pairs in multiple languages. Translation quality under language collaborative training.

Description of the drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without exerting creative efforts.

Figure 1 is a flow chart of a pseudo-language family clustering method based on a multi-language pre-trained large model provided by an embodiment of the present invention;

Figure 2 is a schematic diagram of the language pair (XX-en) provided by the embodiment of the present invention;

Figure 3 is a schematic diagram of the distribution of high 40% Fisher information parameters in the model structure provided by the embodiment of the present invention;

Figure 4 is a block diagram of a pseudo-language family clustering device based on a multi-language pre-trained large model provided by an embodiment of the present invention;

Figure 5 is a schematic structural diagram of a pseudo-language family clustering device based on a multi-language pre-trained large model provided by an embodiment of the present invention.

Detailed ways

The technical solutions in the present invention will be described below with reference to the accompanying drawings.

In the embodiments of the present invention, words such as "exemplarily" and "for example" are used to represent examples, illustrations or explanations. Any embodiment or design described as "an example" of the invention is not intended to be construed as preferred or advantageous over other embodiments or designs. Rather, the use of the word example is intended to present a concept in a concrete way. In addition, in the embodiment of the present invention, the meaning expressed by "and/or" may be both, or may be either one of the two.

In the embodiments of the present invention, "image" and "picture" may sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they convey are consistent. "Of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the difference is not emphasized, the meanings they convey are consistent.

In the embodiment of the present invention, sometimes a subscript such as W1 may be mistakenly written in a non-subscript form such as W1. When the difference is not emphasized, the meanings to be expressed are consistent.

In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, a detailed description will be given below with reference to the accompanying drawings and specific embodiments.

The embodiment of the present invention provides a pseudo-language family clustering method based on a large multi-language pre-training model. The method can be implemented by a pseudo-language family clustering device based on a large multi-language pre-training model. The method is based on a large multi-language pre-training model. The pseudo-language family clustering device of the model can be a terminal or a server. As shown in Figure 1, the flow chart of the pseudo-language family clustering method based on a multi-language pre-trained large model is shown. The processing flow of this method can include the following steps:

S101. Establish a shared language pool;

In a feasible implementation, in step S101, establishing a shared language pool includes:

Get TED data set;

Extract multiple languages from the TED data set, and use the language pairs translated from multiple languages into English as the basic data set to establish a shared language pool.

In a feasible implementation, for the study of language pair representation, it is first necessary to establish a shared language pool that contains high-resource and low-resource language pairs and spans multiple language departments. This invention selects the TED data set, and uses data from 17 languages to English (English, referred to as en) as the basic data set of the invention. Together, these language pairs form a shared language pool for selection, which will be used as alternative auxiliary languages to low-resource languages in subsequent selections. These languages span seven different language families: Balto-Slavic, Austronesian, Indo-Iranian, Turkic, Japonic, and Korean ) and Germanic (Germanic), as shown in Figure 2.

S102 is based on the large multi-language pre-training model, calculates the Fisher information matrix of the language pairs in the shared language pool, and obtains the representation results of the language pairs in the shared language pool;

In a feasible implementation, in step S102, based on the multi-language pre-trained large model, the Fisher information matrix of the language pairs in the shared language pool is calculated, and the characterization results of the language pairs in the shared language pool are obtained, including:

Obtain a parallel corpus corresponding to the language pairs in the shared language pool, and equalize the data in the parallel corpus into j mini-batch data sets;

Input the small batch data sets into the multi-language pre-trained large model in turn, and output the Fisher information matrix of each small batch data set;

After an input round, the average Fisher information matrix of each mini-batch data set is calculated, and the average Fisher information matrix is used as an estimate to obtain the Fisher information weight of each mini-batch data set;

Characterize the distribution of corresponding language pairs in the shared language pool according to Fisher information weights.

The present invention chooses to use FIM to calculate the parameters in the pre-training model, and calculates its Fisher information weight for each parameter. The size of the weight indicates its importance, and the parameter with a large weight indicates that it is sensitive in a specific translation direction. Parameter calculation is used to evaluate and select those parameters that are sensitive to a specific translation direction, that is, parameters that require a large number of weight updates during the fine-tuning phase. It is an important indicator to evaluate the importance and potential value of specific parameters. Essentially, it quantifies the variance of the first derivative of the logarithm of the likelihood function. By measuring the magnitude of this metric, one can infer the necessity of fine-tuning specific parameters in subsequent tasks. Its original calculation formula is as follows:

Among them, X and Y represent the input and output of the model respectively; θ represents the parameters of the model; P represents the probability distribution of the output Y when the input is For the i-th parameter, using the diagonal matrix helps estimate the Fisher information matrix:

Although utilizing a diagonal matrix helps estimate FIM, obtaining accurate probability estimates is still a difficult task. Given this, we use the following formula to approximate FIM:

Here, D represents the entire data set, |D| represents the number of data, and the entire data set is divided into j small batches with equal amounts of data and are sequentially input to the model for training. This invention inputs the parallel corpus corresponding to the language into the model and calculates the FIM of each mini-batch in only one round (epoch). During model training, we use formula (3) to accumulate FIM for each mini-batch but do not perform backpropagation. After an epoch is completed, we calculate the average FIM obtained for each mini-batch as the final estimate.

Furthermore, the present invention analyzes the distribution of high Fisher information parameters in the pre-training model structure. The distribution of the high 40% parameters observed in the study is shown in Figure 3. The pre-training model is initially divided into five parts: encoder attention layer (E _a for short), encoder fully connected layer (E _f for short), decoder self-attention (decoder self-attention) , referred to as D _a ), decoder cross attention layer (decoder cross attention layer, referred to as D _c ) and decoder fully connected layer (decoder fully connected layer, referred to as D _f ). More than 60% of the parameters are distributed in feed-forward networks (FFN). Therefore, the present invention selects the FFN layer to calculate the FIM and measure the similarity.

S103. Calculate the similarity between language pairs based on the representation results and obtain the similarity value;

In a feasible implementation, in step S103, the similarity between the language pairs is calculated according to the characterization result, and the similarity value is obtained, including:

Obtain characterization results;

The mean square error method is used to calculate the distance between the language pairs in the shared language pool and the target language pair. The closer the distance is, the higher the similarity.

In a feasible implementation, the mean square error (Mean Square Error, referred to as MSE): the calculation formula is as follows, where t, a are the target language pair and the auxiliary language pair respectively, and S _{(t, a)} is between t and a distance, the closer the distance, the higher the similarity, F is FIM, |F _t | represents the parameter amount.

In a feasible implementation, in step S103, the similarity between the language pairs is calculated according to the characterization result, and the similarity value is obtained, including:

Use the Fisher information matrix to calculate the KL divergence between the language pair in the shared language pool and the target language pair, and obtain the distance between the language pair in the shared language pool and the target language pair. The closer the distance is, the higher the similarity.

In a feasible implementation, KL divergence (Kullback-Leibler divergence, KL for short): directly use FIM to calculate the KL divergence of the language pairs in the shared language pool and the target language pair, thereby representing the language pairs more accurately. the distance between. The calculation formula is as follows, where the symbols are consistent with those described in MSE, and |·| represents the calculated absolute value.

In a feasible implementation, in step S103, the similarity between the language pairs is calculated according to the characterization result, and the similarity value is obtained, including:

Create a Fisher information mask by selecting the first K parameters and assigning them the value 1, while the remaining parameters are assigned the value 0;

Based on the number of parameters activated simultaneously and the number of parameters activated in the target direction, the distance between the target language pair and the auxiliary language pair is calculated. The closer the distance is, the higher the similarity.

In a feasible implementation, Overlap Similarity (Overlap): Different from the first two calculation methods, the Overlap Similarity does not directly use FIM, by selecting the top K parameters and assigning them a value of 1, and the rest The parameter is assigned a value of 0 to create a Fisher information mask (M). The calculation method is as follows, where Overlapping and Activate represent the number of parameters activated simultaneously and the amount of parameters activated in the target direction.

In the Overlap method, the present invention uses 40% as the default parameter of K by default because it achieves the best translation effect.

S104. Sort the similarities between language pairs according to the similarity value, select auxiliary language pairs that meet the boundary value according to the preset boundary value, and complete pseudo-language family clustering based on the multi-language pre-trained large model.

In a feasible implementation, in step S104, the similarities between the language pairs are sorted according to the similarity value, and the auxiliary language pairs that meet the boundary value are selected according to the preset boundary value to complete the multi-language pre-training large model. Pseudo-language family clustering, including:

Traversely calculate the similarity between all language pairs;

Sort in descending order according to similarity between language pairs;

Preset the initial search radius and delineate the boundary range based on the initial search radius;

Integrate the closest language pairs within the boundary into the auxiliary language list;

Update the search radius based on the similarity between the newly added language pair and the target language pair;

The search radius is updated repeatedly until new language pairs are no longer expanded, the clustered pseudo-language family is obtained, and the pseudo-language family clustering based on the multi-language pre-trained large model is completed.

In a feasible implementation, the present invention designs a simple algorithm to select the auxiliary language. First, we use the above similarity measurement method for the target language pair to calculate the similarity of other language pairs with it, sort them according to the similarity between the language pairs, and then set an initial search radius. Within this predefined boundary, the closest language pairs are integrated into the auxiliary language list. The radius is then adjusted based on the similarity measure between the newly added language pair and the target language pair. This process is repeated until no more new language pairs are extended. This invention refers to such clustered language families as pseudo-language families. The algorithm for selecting auxiliary language pairs is as follows:

1. Arrange the similarities in descending order (MSE and KL in ascending order, Overlap in descending order) and create a list L;

2. Initialize Gap to |L[1]-L[0]|, and add the first language to the auxiliary list;

3. Iterate from i=2 to the end of L. For each loop, we choose the language as follows:

a) If |L[i-1]-L[i]|<Gap, add the i-th language to the auxiliary list and update Gap=|L[i-1]-L[i]|;

b) If |L[i-1]-L[i]|<Gap, add the i-th language to the auxiliary list and update

c) If |L[i-1]-L[i]|>Gap, terminate the loop;

4. The language pairs in the auxiliary list that are paired with the target form a pseudo-language family of the target language pair.

In a feasible implementation, in order to evaluate the method of the present invention, the following baseline is designed based on the m2m100_418M model:

1. Pre-trained: directly use the pre-trained model to directly translate the target language pair without any fine-tuning;

2. FT (fine-tune, fine-tuning): Use bilingual data of the target language pair to fine-tune the basic model;

3.LF (langauge family, language family): Use the traditional language family divided in Figure 2 for fine-tuning, use temperature (temperature) sampling and set the temperature to 1.5;

4. LF+FT: Based on the LF method, use the data of the target language pair to further fine-tune.

For the training phase, the batch size is set to 4096; for the method proposed in the present invention, the training data is upsampled to make it the same to ensure that the proportion of each language in each mini-batch is the same, except for Hindi , except for hi), when using the Overlap method, the same sampling method as LF is used. Optimization was performed using Adam optimizer with β ₁ =0.98, β ₂ =0.98 and ∈ =10e ⁻⁶ . The learning rate is lr=3e ^-5 . In the present invention, the BLEU (bilingualevaluation understudy, bilingual replacement evaluation) value is used to evaluate the translation quality.

Table 1 Pseudo-language families under different method selections

Table 2 BLEU scores of each low-resource language pair under different methods

Table 1 shows the pseudo-language families clustered by the method of the present invention for low-resource language pairs in the shared language pool. Table 2 shows Persian (Persian, referred to as fa), Hindi (Hindi, referred to as hi), Bengali Test results on the TED data set from Bengali (bn for short), Indonesian (Indonesian for short), Malay (ms) to English.

Models (1) to (4) represent the baseline in currently commonly used fine-tuning, and models (5) to (7) represent our implementation of this method using different computational methods.

The present invention uses three measurement methods: MSE, KL and Overlap to calculate the distance or similarity between language pairs, thereby clustering pseudo-language families and conducting verification on low-resource language pairs. The evaluation results show that all three calculation methods further improve the BLEU score and the final improvement effect is equivalent. Model (7) achieves the best improvement effect.

Figure 4 is a block diagram of a pseudo-language family clustering device based on a large multi-language pre-trained model according to an exemplary embodiment. The device is used for a pseudo-language family clustering method based on a large multi-language pre-trained model. Referring to FIG. 4 , the device includes a language pool module 410 , a representation module 420 , a similarity calculation module 430 , and a clustering module 440 . For ease of explanation, Figure 4 only shows the main components of the full process visualization device 400:

Language pool module 410, used to establish a shared language pool;

The representation module 420 is used to calculate the Fisher information matrix of the language pairs in the shared language pool based on the multi-language pre-trained large model, and obtain the representation results of the language pairs in the shared language pool;

The similarity calculation module 430 is used to calculate the similarity between language pairs according to the representation results and obtain the similarity value;

The clustering module 440 is used to sort the similarities between language pairs according to the similarity value, select auxiliary language pairs that meet the boundary value according to the preset boundary value, and complete pseudo-language family clustering based on the large multi-language pre-training model. kind.

Optionally, the language pool module 410 is used to obtain the TED data set;

Extract multiple languages from the TED data set, and use the language pairs translated from multiple languages into English as the basic data set to establish a shared language pool.

Optionally, the characterization module 420 is used to obtain a parallel corpus corresponding to the language pairs in the shared language pool, and equalize the data in the parallel corpus into j mini-batch data sets;

Input the small batch data sets into the multi-language pre-trained large model in turn, and output the Fisher information matrix of each small batch data set;

After an input round, the average Fisher information matrix of each mini-batch data set is calculated, and the average Fisher information matrix is used as an estimate to obtain the Fisher information weight of each mini-batch data set;

Characterize the distribution of corresponding language pairs in the shared language pool according to Fisher information weights.

Optionally, the similarity calculation module 430 is used to obtain the characterization results;

The mean square error method is used to calculate the distance between the language pairs in the shared language pool and the target language pair. The closer the distance is, the higher the similarity.

Optionally, the similarity calculation module 430 is used to use the Fisher information matrix to calculate the KL divergence between the language pairs in the shared language pool and the target language pair, and obtain the KL divergence between the language pairs in the shared language pool and the target language pair. The distance between them, the distance and proximity, the higher the similarity.

Optionally, the similarity calculation module 430 is used to select the first K parameters and assign them a value of 1, while the remaining parameters are assigned a value of 0 to create a Fisher information mask;

Based on the number of parameters activated simultaneously and the number of parameters activated in the target direction, the distance between the language pair in the shared language pool and the target language pair is calculated. The closer the distance is, the higher the similarity.

Optionally, the clustering module 440 is used to traversely calculate the similarity between all language pairs;

Sort in descending order according to similarity between language pairs;

Preset the initial search radius and delineate the boundary range based on the initial search radius;

Integrate the closest language pairs within the boundary into the auxiliary language list;

Update the search radius based on the similarity between the newly added language pair and the target language pair;

The search radius is updated repeatedly until new language pairs are no longer expanded, the clustered pseudo-language family is obtained, and the pseudo-language family clustering based on the multi-language pre-trained large model is completed.

In view of the limitations in the existing technology that require additional prior knowledge or the need to modify the model architecture, the present invention provides a method for constructing a more effective language pair clustering method for multi-language collaborative training. The core goal is to use the capabilities of multilingual pre-training to characterize language pairs, more effectively select and cluster auxiliary languages and improve their generalization across different models and data sets, and ultimately improve the performance of low-resource language pairs in multiple languages. Translation quality under language collaborative training.

Figure 5 is a schematic structural diagram of a pseudo-language family clustering device based on a large multi-language pre-trained model provided by an embodiment of the present invention. As shown in Figure 5, the pseudo-language family clustering device based on a large multi-language pre-trained model can It includes the pseudo-language family clustering device based on the multi-language pre-trained large model shown in Figure 4 above. Optionally, the pseudo-language family clustering device 510 based on a multi-language pre-trained large model may include a processor 2001.

Optionally, the pseudo-language family clustering device 510 based on the multi-language pre-trained large model may also include a memory 2002 and a transceiver 2003.

Among them, the processor 2001, the memory 2002 and the transceiver 2003 can be connected through a communication bus.

The following is a detailed introduction to each component of the pseudo-language family clustering device 510 based on a multi-language pre-trained large model with reference to Figure 5:

Among them, the processor 2001 is the control center of the pseudo-language family clustering device 510 based on the multi-language pre-trained large model, and can be a processor or a collective name for multiple processing elements. For example, the processor 2001 is one or more central processing units (CPUs), may also be an application specific integrated circuit (ASIC), or may be one or more processors configured to implement embodiments of the present invention. Integrated circuits, such as one or more microprocessors (digital signal processor, DSP), or one or more field programmable gate arrays (field programmable gate array, FPGA).

Optionally, the processor 2001 can execute various functions of the pseudo-language family clustering device 510 based on the multi-language pre-trained large model by running or executing the software program stored in the memory 2002 and calling the data stored in the memory 2002. Function.

In a specific implementation, as an embodiment, the processor 2001 may include one or more CPUs, such as CPU0 and CPU1 shown in FIG. 4 .

In specific implementation, as an embodiment, the pseudo-language family clustering device 510 based on a multi-language pre-trained large model may also include multiple processors, such as the processor 2001 and the processor 2004 shown in Figure 4 . Each of these processors can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). A processor here may refer to one or more devices, circuits, and/or processing cores for processing data (eg, computer program instructions).

The memory 2002 is used to store the software program for executing the solution of the present invention, and the processor 2001 controls the execution. For specific implementation methods, please refer to the above method embodiments, which will not be described again here.

Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (random access memory, RAM) or a random access memory (RAM) that can store information and instructions. Other types of dynamic storage devices for instructions can also be electrically erasable programmable read-only memory (EEPROM), compactdisc read-only memory (CD-ROM) or other optical disk storage , optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), magnetic disk storage media or other magnetic storage devices, or can be used to carry or store the desired program code in the form of instructions or data structures and Any other media capable of being accessed by a computer, without limitation. The memory 2002 can be integrated with the processor 2001, or can exist independently, and is coupled to the processor 2001 through the interface circuit (not shown in Figure 5) of the pseudo-language family clustering device 510 based on the multi-language pre-trained large model, The embodiment of the present invention does not specifically limit this.

Transceiver 2003 is used to communicate with network equipment or with terminal equipment.

Optionally, the transceiver 2003 may include a receiver and a transmitter (not shown separately in Figure 5). Among them, the receiver is used to implement the receiving function, and the transmitter is used to implement the sending function.

Optionally, the transceiver 2003 can be integrated with the processor 2001, or can exist independently, and communicate with the transceiver 2003 through the interface circuit (not shown in Figure 5) of the pseudo-language family clustering device 510 based on the multi-language pre-trained large model. The processor 2001 is coupled, which is not specifically limited in the embodiment of the present invention.

It should be noted that the structure of the pseudo-language family clustering device 510 based on the multi-language pre-trained large model shown in Figure 5 does not constitute a limitation on the router. The actual knowledge structure recognition device may include more than what is shown in the figure. Or fewer parts, or a combination of certain parts, or a different arrangement of parts.

In addition, the technical effects of the pseudo-language family clustering device 410 based on the multi-language pre-trained large model can be referred to the technical effects of the pseudo-language family clustering method based on the multi-language pre-trained large model described in the above method embodiments, which are not discussed here. Again.

It should be understood that the processor 2001 in the embodiment of the present invention can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits. (application specific integrated circuit, ASIC), off-the-shelf programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

It should also be understood that memory in embodiments of the present invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (erasable PROM, EPROM), electrically removable memory. Erase programmable read-only memory (electrically EPROM, EEPROM) or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of illustration, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (RAM), etc. Access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (doubledata rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous connection dynamic random access memory (synchlink DRAM, SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

The above embodiments may be implemented in whole or in part by software, hardware (such as circuits), firmware, or any other combination. When implemented using software, the above-described embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in accordance with the embodiments of the present invention are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another, e.g., the computer instructions may be transferred from a website, computer, server, or data center Transmit to another website, computer, server or data center through wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that contains one or more sets of available media. The usable media may be magnetic media (eg, floppy disk, hard disk, tape), optical media (eg, DVD), or semiconductor media. The semiconductor medium may be a solid state drive.

It should be understood that the term "and/or" in this article is only an association relationship describing related objects, indicating that there can be three relationships, for example, A and/or B, which can mean: A alone exists, and A and B exist simultaneously. , there are three cases of B alone, where A and B can be singular or plural. In addition, the character "/" in this article generally indicates that the related objects are an "or" relationship, but it may also indicate an "and/or" relationship. For details, please refer to the previous and later contexts for understanding.

In the present invention, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions thereof refers to any combination of these items, including any combination of a single item (items) or a plurality of items (items). For example, at least one of a, b, or c can mean: a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, c can be single or multiple .

It should be understood that in various embodiments of the present invention, the size of the sequence numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its functions and internal logic, and should not be used in the embodiments of the present invention. The implementation process constitutes any limitation.

Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Skilled artisans may implement the described functionality using different methods for each specific application, but such implementations should not be considered to be beyond the scope of the present invention.

Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working processes of the above-described equipment, devices and units can be referred to the corresponding processes in the foregoing method embodiments, and will not be described again here.

In the several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or can be integrated into another device, or some features can be ignored, or not implemented. On the other hand, the coupling or direct coupling or communication connection between each other shown or discussed may be through some interfaces, and the indirect coupling or communication connection of the devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit.

If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk and other media that can store program code. .

The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention. should be covered by the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the claims.

Claims (10)

1. A pseudo language family clustering method based on a multilingual pre-training large model, the method comprising:

s1, establishing a shared language pool;

s2, calculating a Fisher information matrix of the language pairs in the shared language pool based on the multi-language pre-training large model, and obtaining a characterization result of the language pairs in the shared language pool;

s3, calculating the similarity between the language pairs according to the characterization result to obtain a similarity value;

s4, sorting the similarity among the language pairs according to the similarity value, selecting auxiliary language pairs conforming to the boundary value according to a preset boundary value, and completing pseudo language family clustering based on a multilingual pre-training large model.

2. The method according to claim 1, wherein in the step S1, the step of creating the shared language pool includes:

Acquiring a TED data set;

extracting multiple languages in the TED data set, translating the multiple languages into language pairs of English to serve as a basic data set, and establishing a shared language pool.

3. The method according to claim 1, wherein in the step S2, based on the multilingual pre-training large model, a fee-house information matrix of the language pairs in the shared language pool is calculated, and the characterization result of the language pairs in the shared language pool is obtained, including:

acquiring a parallel corpus corresponding to the languages in the shared language pool, and equally dividing the data in the parallel corpus into j small batch data sets;

sequentially inputting the small batch data sets into a multilingual pre-training large model, and outputting a Fisher information matrix of each small batch data set;

calculating an average Fisher information matrix of each small batch data set after one input round, and taking the average Fisher information matrix as an estimated value to obtain Fisher information weight of each small batch data set;

and characterizing the distribution of the corresponding language pairs in the shared language pool according to the Fisher information weight.

4. The method according to claim 3, wherein in the step S3, the step of calculating the similarity between the language pairs according to the characterization result to obtain a similarity value includes:

Obtaining a characterization result; selecting a target language pair;

and calculating the distance between the language pair in the shared language pool and the target language pair by adopting a mean square error method, wherein the distance is similar to the target language pair, and the higher the similarity is.

5. The method according to claim 3, wherein in the step S3, the step of calculating the similarity between the language pairs according to the characterization result to obtain a similarity value includes:

selecting a target language pair;

and calculating the KL divergence of the language pairs in the shared language pool and the target language pairs by using the fee-house information matrix to obtain the distance between the language pairs in the shared language pool, wherein the distance is similar, and the similarity is higher.

6. The method according to claim 3, wherein in the step S3, the step of calculating the similarity between the language pairs according to the characterization result to obtain a similarity value includes:

selecting a target language pair;

selecting and assigning a value of 1 to the parameter of the previous K, and assigning a value of 0 to the remaining parameters to create a fee-house information mask;

and calculating the distance between the language pair in the shared language pool and the target language pair according to the number of the parameters activated simultaneously and the number of the parameters activated in the target direction, wherein the distance is similar, and the higher the similarity is.

7. The method according to claim 4, 5 or 6, wherein in step S4, the similarity between the language pairs is ranked according to the similarity value, the auxiliary language pair conforming to the boundary value is selected according to a preset boundary value, and the completion of the pseudo language family clustering based on the multilingual pre-training large model includes:

traversing and calculating the similarity between all language pairs;

descending order according to the similarity between the language pairs;

presetting an initial searching radius, and defining a boundary range according to the initial searching radius;

integrating the nearest language pair in the boundary range into an auxiliary language list;

updating the search radius according to the similarity between the latest added language pair and the target language pair;

and repeatedly updating the search radius until new language pairs are not expanded, obtaining clustered pseudo language families, and completing pseudo language family clustering based on the multilingual pre-training large model.

8. A pseudo-language family clustering device based on a multilingual pre-training large model, the device comprising:

the language pool module is used for establishing a shared language pool;

the characterization module is used for calculating a Fisher information matrix of the language pairs in the shared language pool based on the multi-language pre-training large model to obtain a characterization result of the language pairs in the shared language pool;

The similarity calculation module is used for calculating the similarity between the language pairs according to the characterization result to obtain a similarity value;

and the clustering module is used for sequencing the similarity among the language pairs according to the similarity value, selecting auxiliary language pairs conforming to the boundary value according to a preset boundary value, and completing pseudo language family clustering based on the multilingual pre-training large model.

9. A pseudo language family clustering device based on a multilingual pre-training large model, the pseudo language family clustering device based on the multilingual pre-training large model comprising:

a processor;

a memory having stored thereon computer readable instructions which, when executed by the processor, implement the method of any of claims 1 to 7.

10. A computer readable storage medium having stored therein program code which is callable by a processor to perform the method of any one of claims 1 to 7.