CN118132672A - Model pre-training method and device, storage medium and electronic equipment - Google Patents

Abstract

The invention provides a model pre-training method, a device, a storage medium and electronic equipment, wherein the method comprises the following steps: compression processing is carried out to obtain compressed data sources under each path of data in each training data; masking the compressed data sources under the data of each path in each training data, and encoding the masking data sources under the corresponding path in each training data to obtain the encoded data sources under the data of each path in each training data; based on the coding data sources under each path of data in each training data and the compression data sources under each path of data in each training data, respectively calculating model loss values under each path of data, respectively optimizing model parameters in an initial language model corresponding to the corresponding path of data according to the model loss values under each path of data, and obtaining an intermediate language model corresponding to each path of data so as to determine a target language model corresponding to each path of data. The embodiment of the invention can conveniently perform model pre-training so as to improve the model pre-training efficiency.

Description

Model pre-training method, device, storage medium and electronic device

Technical Field

The present invention relates to the field of computer technology, and in particular to a model pre-training method, device, storage medium and electronic equipment.

Background technique

At present, pre-trained language models have attracted extensive attention in natural language processing. The so-called pre-trained language models refer to: using a large amount of unsupervised data to pre-train before downstream task training to obtain a better semantic representation; however, the relevant technology has low efficiency in model pre-training (i.e. modeling), especially when modeling long texts. Based on this, there is currently no good solution for how to conveniently pre-train models to improve model pre-training efficiency.

Summary of the invention

In view of this, the embodiments of the present invention provide a model pre-training method, device, storage medium and electronic device to solve the problem of low model pre-training efficiency caused by related technologies; that is, the embodiments of the present invention can conveniently perform model pre-training to improve the model pre-training efficiency, which can effectively improve the speed of model training and reasoning.

According to one aspect of the present invention, a model pre-training method is provided, the method comprising:

Obtain a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, where M is a positive integer;

Performing compression processing on the initial data sources of each data path in each training data included in the training data set to obtain compressed data sources of each data path in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source;

Masking the compressed data sources under each path of data in each of the training data respectively to obtain the masked data sources under each path of data in each of the training data, and respectively calling the initial language models corresponding to each path of data to encode the masked data sources under the corresponding path of data in each of the training data to obtain the encoded data sources under each path of data in each of the training data;

Based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, the model loss value under each data path is calculated respectively, and according to the model loss value under each data path, the model parameters in the initial language model corresponding to the corresponding data path are optimized respectively to obtain the intermediate language model corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path.

According to another aspect of the present invention, a model pre-training device is provided, the device comprising:

An acquisition unit is used to acquire a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, where M is a positive integer;

A processing unit, configured to compress the initial data sources of the respective data paths in the respective training data included in the training data set to obtain compressed data sources of the respective data paths in the respective training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source;

The processing unit is further configured to respectively mask the compressed data sources under the respective paths of data in the respective training data to obtain the masked data sources under the respective paths of data in the respective training data, and respectively call the initial language models corresponding to the respective paths of data to encode the masked data sources under the corresponding paths of data in the respective training data to obtain the encoded data sources under the respective paths of data in the respective training data;

The processing unit is further used to calculate the model loss value of each data path based on the encoded data source of each data path in the each training data and the compressed data source of each data path in the each training data, and optimize the model parameters in the initial language model corresponding to the corresponding data path according to the model loss value of each data path, so as to obtain the intermediate language model corresponding to each data path, and determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path.

According to another aspect of the present invention, an electronic device is provided, the electronic device comprising a processor and a memory storing a program, wherein the program comprises instructions, and when the instructions are executed by the processor, the processor executes the above-mentioned method.

According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method.

After obtaining the training data set, the embodiment of the present invention can compress the initial data sources under each data path in each training data included in the training data set to obtain the compressed data sources under each data path in each training data. One training data includes the initial data sources under each data path of an object in M data paths, and one data path corresponds to one language model, M is a positive integer; the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source. Then, the compressed data sources under each data path in each training data can be masked to obtain the masked data sources under each data path in each training data, and the initial language model corresponding to each data path can be called to encode the masked data sources under the corresponding data path in each training data to obtain the encoded data sources under each data path in each training data. Based on this, the model loss value under each data path can be calculated based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, and the model parameters in the initial language model corresponding to the corresponding data path can be optimized according to the model loss value under each data path to obtain the intermediate language model corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path. It can be seen that the embodiments of the present invention can conveniently perform model pre-training to improve the efficiency of model pre-training, that is, can effectively improve the speed of model training and reasoning.

BRIEF DESCRIPTION OF THE DRAWINGS

Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

FIG1 is a schematic diagram showing a flow chart of a model pre-training method according to an exemplary embodiment of the present invention;

FIG2 shows a schematic diagram of pre-training according to an exemplary embodiment of the present invention;

FIG3 shows a schematic flow chart of another model pre-training method according to an exemplary embodiment of the present invention;

FIG4 shows a schematic diagram of contrastive learning according to an exemplary embodiment of the present invention;

FIG5 shows a schematic block diagram of a model pre-training device according to an exemplary embodiment of the present invention;

FIG. 6 shows a block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention.

Detailed ways

Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and/or in parallel. In addition, the method embodiments may include additional steps and/or omit the steps shown. The scope of the present invention is not limited in this respect.

The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

It should be noted that the execution subject of the model pre-training method provided in the embodiment of the present invention may be one or more electronic devices, and the present invention does not limit this; wherein, the electronic device may be a terminal (i.e., a client) or a server, then when the execution subject includes multiple electronic devices, and the multiple electronic devices include at least one terminal and at least one server, the model pre-training method provided in the embodiment of the present invention may be jointly executed by the terminal and the server. Accordingly, the terminals mentioned here may include but are not limited to: smart phones, tablet computers, laptop computers, desktop computers, intelligent voice interaction devices, and the like. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing (cloud computing), cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, and the like.

Based on the above description, an embodiment of the present invention proposes a model pre-training method, which can be executed by the electronic device (terminal or server) mentioned above; or, the model pre-training method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the model pre-training method as an example; as shown in Figure 1, the model pre-training method may include the following steps S101-S104:

S101, obtaining a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, and M is a positive integer.

Among them, one training data corresponds to one object; then correspondingly, the training data set may include the training data of each object in at least one object, that is, the training data set may include the initial data source of each object under each data channel; optionally, the data source of an object under each data channel may also be called the data source of each data channel in the training data of the corresponding object, that is, one training data may include the initial data source under each data channel, and the data source under each data channel in one training data may refer to: the data source of the object corresponding to the corresponding training data under each data channel. Exemplarily, the compressed data source under each data channel in one training data may refer to the compressed data source of the object corresponding to the corresponding training data under each data channel, the mask data source under each data channel in one training data may refer to the mask data source of the object corresponding to the corresponding training data under each data channel, the encoded data source under each data channel in one training data may refer to the encoded data source of the object corresponding to the corresponding training data under each data channel, and so on.

Optionally, an object may be a user, or a commodity (such as a book), etc., which is not limited in the embodiment of the present invention. Optionally, each object in at least one object may be of the same type, such as all users or all books.

It should be understood that when the value of M is 1, the embodiment of the present invention involves a single-channel data, thereby realizing the following determination of the target language model corresponding to one channel of data; when the value of M is greater than 1, the data can be expanded to multiple channels of data, thereby realizing the following determination of the target language model corresponding to each channel of data, that is, the number of target language models can be multiple, one channel of data corresponds to one target language model, and so on.

Exemplarily, assuming that the value of M is 2 and an object is a user, then one channel of data can be the user's attribute label data (such as gender and age, etc.), that is, the initial data source under one channel of data can be used to describe the user's attribute information; correspondingly, another channel of data can be the user's search text and/or browsing text, etc., that is, the initial data source under another channel of data can be used to describe the user's search content and/or browsing content, and so on.

In the embodiment of the present invention, the above training data set may be obtained in the following ways, including but not limited to:

The first acquisition method: multiple training data are stored in the electronic device's own storage space. In this case, the electronic device can select at least one training data from the multiple training data and add the at least one training data to the training data set so that the training data set includes at least one training data.

The second acquisition method: the electronic device can obtain the training data download link, and add the training data downloaded based on the training data download link to the training data set, so that the training data set includes the training data downloaded based on the training data download link.

The third acquisition method: the electronic device can obtain a training text set, the training text set includes at least one training text, and a training text may include a training sub-text of an object under each data channel. In this case, for any training text in the training text set, the electronic device can perform word segmentation processing on the training sub-text under each data channel in any training text to obtain a word segmentation result under each data channel in any training text, and a word segmentation result may include multiple word segments (i.e., tokens), and the word segmentation results under each data channel in any training text can be vectorized respectively to obtain an initial data source under each data channel in any training text, and the initial data source under any data channel in any training text may include the initial semantic representation of each word segmentation under any data channel in any training text, so that the initial data source under each data channel in any training text can be added to the training data set, so that the initial data source under each data channel in any training text is used as a training data in the training data set, and so on.

It should be noted that one training data may correspond to one training text, and one data path in one training data corresponds to one training sub-text in the corresponding training text. The text length of a training sub-text may be any length, which is not limited in the embodiment of the present invention. The text length of a training sub-text may be the number of segmented words in the corresponding training sub-text. Optionally, when the text length of a text is greater than a preset length threshold, the text may be regarded as a long text; optionally, the preset length threshold may be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention.

Optionally, when the training sub-text in the training text corresponding to each training data is a long text, the embodiment of the present invention can realize long text modeling, which can be seen as follows, and the embodiment of the present invention will not be repeated here.

S102, respectively compressing the initial data sources of each data path in each training data included in the training data set to obtain compressed data sources of each data path in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source.

In an embodiment of the present invention, for any training data in a training data set, and any data in M data paths, the electronic device may compress the initial data source under any data path in any training data to obtain a compressed data source under any data path in any training data; that is, the initial data source under any data path of an object corresponding to any training data may be compressed to obtain a compressed data source under any data path of an object corresponding to any training data. Among them, the compressed data source under any data path in any training data may include: a compressed data source under any data path of an object corresponding to any training data, and a compressed data source may include multiple compressed semantic representations, that is, the compressed data source under any data path in any training data may include: multiple compressed semantic representations under any data path in any training data. Correspondingly, the initial data source under any data path in any training data may include: multiple initial semantic representations under any data path in any training data.

It should be noted that the number of compressed semantic representations under any data path in any training data may be less than the number of initial semantic representations under any data path in any training data; optionally, when the training sub-text corresponding to any data path in any training data is a long text, the number of compressed semantic representations under any data path in any training data may be much smaller than the number of initial semantic representations under any data path in any training data, thereby effectively compressing the text length.

S103, respectively masking the compressed data source under each data path in each training data to obtain the masked data source under each data path in each training data, and respectively calling the initial language model corresponding to each data path, encoding the masked data source under the corresponding data path in each training data to obtain the encoded data source under each data path in each training data.

Among them, a masked data source may include the masked semantic representation of each mask unit in H mask units, that is, the electronic device may mask the compressed semantic representation of each mask unit in a compressed data source to obtain the corresponding masked data source, then correspondingly, the masked semantic representation of each semantic unit other than the H mask units in a masked data source may be the compressed semantic representation of the corresponding semantic unit in the corresponding compressed data source, and H is a positive integer. In addition, an encoded data source may include the encoded semantic representation of each semantic unit, and thus may include the encoded semantic representation of each mask unit in H mask units. Among them, one semantic representation may correspond to one semantic unit, and one semantic unit may be a mask unit or a semantic unit other than a mask unit.

It should be noted that the mask units in any two data sources may be the same or different; that is, the number of mask units in any two data sources may be the same or different, and the embodiment of the present invention is not limited to this; it should be understood that the mask units in the data sources under the same data path in the same training data (such as a compressed data source and an encoded data source, etc.) are the same.

Exemplarily, assuming that a compressed data source includes four compressed semantic representations, and the mask unit in the four compressed semantic representations includes the third semantic unit (i.e., the semantic unit corresponding to the third compressed semantic representation), then the electronic device may mask the third semantic unit in the compressed data source to obtain the masked semantic representation of the third semantic unit, so as to mask the compressed data source and obtain the masked data source corresponding to the compressed data source; wherein the masked data source corresponding to the compressed data source may include: the compressed semantic representation of the first semantic unit, the compressed semantic representation of the second semantic unit, the masked semantic representation of the third semantic unit, and the compressed semantic representation of the fourth semantic unit.

Optionally, a language model can be a FLASH model (an efficient long text model), or a Bert (a deep bidirectional pre-training model), etc.; the embodiment of the present invention is not limited to this. It should be noted that when performing long text modeling, the FLASH model can be preferably used as the language model. Optionally, the model parameters in an initial language model can be randomly generated, set according to experience, or set according to actual needs, which is not limited in the embodiment of the present invention.

S104, based on the encoded data source under each data channel in each training data and the compressed data source under each data channel in each training data, respectively calculate the model loss value under each data channel, and optimize the model parameters in the initial language model corresponding to the corresponding data channel according to the model loss value under each data channel, to obtain the intermediate language model corresponding to each data channel, so as to determine the target language model corresponding to each data channel based on the intermediate language model corresponding to each data channel.

In one implementation, for the m-th data path in the M-path data, the electronic device may determine the encoded semantic representation of each mask unit under the m-th data path in the corresponding training data (that is, the encoded semantic representation of each mask unit of each object under the m-th data path) from the encoded data source under the m-th data path in each training data path, m∈[1,M]. Then, correspondingly, the model loss value under the m-th data path may be calculated based on the encoded semantic representation of each mask unit under the m-th data path in each training data path and the compressed data source under the m-th data path in the corresponding training data path. Specifically, the electronic device can traverse each training data in the training data set, and use the currently traversed training data as the current training data; then, the encoded semantic representation of each mask unit under the m-th data path in the current training data and the compressed data source under the m-th data path in the current training data can be used to calculate the model loss value under the m-th data path in the current training data; after traversing each training data in the training data set, the model loss value under the m-th data path in each training data can be obtained, and the model loss value under the m-th data path in each training data can be weighted and summed to obtain the model loss value under the m-th data path. Optionally, the weight values involved in the weighted summation process of the embodiment of the present invention can be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this; illustratively, when the weight values of the model loss values under the m-th data path in each training data are the same, the model loss values under the m-th data path in each training data can be averaged or summed, and so on.

Optionally, when using the encoded semantic representation of each mask unit under the m-th data path in the current training data and the compressed data source under the m-th data path in the current training data to calculate the model loss value under the m-th data path in the current training data, for any mask unit in the m-th data path in the current training data, the encoded semantic representation of any mask unit under the m-th data path in the current training data and the compressed data source under the m-th data source in the current training data can be used to calculate the model loss value under any mask unit to obtain the model loss value under each mask unit under the m-th data path in the current training data, thereby performing a weighted summation (such as a mean operation or a summation operation, etc.) of the model loss values under each mask unit under the m-th data path in the current training data to obtain the model loss value under the m-th data path in the current training data. Specifically, the electronic device can use formula 1.1 to calculate the model loss value under any mask unit:

Among them, Ex can be the expectation of the encoded data source under the mth data in the current training data (that is, the expectation of each encoded semantic representation), or it can be the expectation of the compressed data source under the mth data in the current training data, etc., and the embodiment of the present invention is not limited to this; the corresponding f(x) can be the encoded semantic representation of any mask unit under the mth data in the current training data, f(x ⁺ ) can be the compressed semantic representation of any mask unit under the mth data in the current training data, W can be the number of compressed semantic representations under the mth data in the current training data, f(x _j ) can be the compressed semantic representation of the jth semantic unit under the mth data in the current training data except any mask unit, L can be the corresponding loss value, and so on.

Exemplarily, as shown in Figure 2, assume that the number of compressed semantic representations under the mth data path in the current training data is 4, and any mask unit under the mth data path in the current training data is the third semantic unit. In this case, f(x ⁺ ) can be the compressed semantic representation of the third semantic unit under the mth data path in the current training data; at this time, the value of W can be 4, the first semantic unit under the mth data path in the current training data can be the first semantic unit under the mth data path in the current training data except any mask unit, the second semantic unit can be the second semantic unit under the mth data path in the current training data except any mask unit, and the fourth semantic unit can be the third semantic unit under the mth data path in the current training data except any mask unit.

It can be seen that the embodiment of the present invention can compare and learn the encoded semantic representation of any mask unit under the m-th data channel in the current training data and the compressed data source under the m-th data channel in the current training data, thereby bringing the encoder semantic representation after the mask (i.e., the mask) (i.e., the encoded semantic representation) and the semantic representation before the underlying mask (i.e., the compressed semantic representation of any mask unit) closer, and can distance the encoded semantic representation of any mask unit from the compressed semantic representation of each semantic unit outside any mask unit.

In another implementation, for the m-th data in M data, the electronic device may calculate the model loss value under the m-th data based on the encoded data source under the m-th data in each training data and the compressed data source under the m-th data in each training data; specifically, for any encoded semantic representation under the m-th data in any training data set, the electronic device may calculate the model loss value under the semantic unit corresponding to any encoded semantic representation based on any encoded semantic representation under the m-th data in any training data and the compressed data source under the m-th data in any training data, so as to obtain the model loss value of each semantic unit under the m-th data in any training data, and perform weighted summation on the model loss value of each semantic unit under the m-th data in any training data, so as to obtain the model loss value under the m-th data in any training data. Based on this, the model loss value under the m-th data may be calculated based on the model loss value under the m-th data in each training data.

Among them, based on any encoded semantic representation under the m-th data in any training data and the compressed data source under the m-th data in any training data, the specific method of calculating the model loss value under the semantic unit corresponding to any encoded semantic representation is the same as the specific method of calculating the model loss value under any mask unit, and the embodiments of the present invention will not go into details about this.

Furthermore, when optimizing the model parameters in the initial language model corresponding to the corresponding data according to the model loss value under each data channel to obtain the intermediate language model corresponding to each data channel, for the m-th data among the M data channels, the electronic device can optimize the model parameters in the initial language model corresponding to the m-th data according to the model loss value under the m-th data channel to obtain the intermediate language model corresponding to the m-th data channel.

After obtaining the training data set, the embodiment of the present invention can compress the initial data sources under each data path in each training data included in the training data set to obtain the compressed data sources under each data path in each training data. One training data includes the initial data sources under each data path of an object in M data paths, and one data path corresponds to one language model, M is a positive integer; the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source. Then, the compressed data sources under each data path in each training data can be masked to obtain the masked data sources under each data path in each training data, and the initial language model corresponding to each data path can be called to encode the masked data sources under the corresponding data path in each training data to obtain the encoded data sources under each data path in each training data. Based on this, the model loss value under each data path can be calculated based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, and the model parameters in the initial language model corresponding to the corresponding data path can be optimized according to the model loss value under each data path to obtain the intermediate language model corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path. It can be seen that the embodiments of the present invention can conveniently perform model pre-training to improve the efficiency of model pre-training, that is, can effectively improve the speed of model training and reasoning.

Based on the above description, the embodiment of the present invention also proposes a more specific model pre-training method. Accordingly, the model pre-training method can be executed by the electronic device (terminal or server) mentioned above; or, the model pre-training method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the model pre-training method as an example; please refer to Figure 3, the model pre-training method may include the following steps S301-S308:

S301, obtaining a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, and M is a positive integer.

S302, respectively compressing the initial data sources of each data path in each training data included in the training data set to obtain compressed data sources of each data path in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source.

Specifically, for any training data in the training data set and any data in the M data, the electronic device can determine the text compression length; and according to the text compression length, the initial data source under any data in any training data is divided to obtain N semantic representation groups to be compressed, and the number of semantic representations in one semantic representation group to be compressed is less than or equal to the text compression length; wherein, the number of semantic representations (here, the initial semantic representations) in each semantic representation group to be compressed in the first N-1 semantic representation groups to be compressed can be equal to the text compression length, and the number of semantic representations in the Nth semantic representation group to be compressed in the N semantic representation groups to be compressed can be less than or equal to the text compression length. Further, each of the N semantic representation groups to be compressed can be compressed to obtain the compressed semantic representation of each semantic representation group to be compressed, so as to obtain the compressed data source under any data in any training data. Optionally, a semantic representation group to be compressed can be taken as a sentence, so as to achieve sentence-level compression, and then achieve the following sentence-level masking.

Optionally, the text compression length can be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this. Optionally, the value of N can be the number of sentences in the text corresponding to any data in any training data. In this case, the number of text compression lengths can be equal to N, and one text compression length is the number of word segments in a sentence, that is, one text compression length can correspond to one sentence, and so on; the embodiment of the present invention does not limit this.

Exemplarily, as shown in FIG2 , assuming that the text compression length is 3, and the text length corresponding to the initial data source under any data path in any training data is 12 tokens, and one token corresponds to an initial semantic representation, in this case, the initial semantic representation of every 3 tokens can be used as a semantic representation group to be compressed, so that every 3 tokens can be compressed into 1 representation unit (i.e., semantic unit), that is, 3 initial semantic representations can be compressed into 1 compressed semantic representation; then correspondingly, the number of compressed semantic representations under any data path in any training data can be 4 (i.e., the text representation length can be 4), where the text representation length can refer to the number of semantic representations used to represent a text. It can be seen that when the initial data source is a long text data source, the embodiment of the present invention can compress the long text data source into a corresponding short text data source (i.e., compressed data source).

Optionally, the compressed data source of each data channel in each training data is obtained by compressing the initial data source of each data channel in each training data through the initial compression model, that is, the electronic device can call the initial compression model to compress the initial data source of each data channel in each training data. Optionally, a compression model can be a CNN (Convolutional Neural Network) semantic compression model (also called a semantic compression module), or a mean pooling module, etc., which is not limited in the embodiments of the present invention.

In an embodiment of the present invention, when the compression model is a CNN semantic compression model, the electronic device may also optimize the model parameters in the initial compression model based on the model loss values under each data channel to obtain an intermediate compression model; and determine the target compression model based on the intermediate compression model; specifically, the model parameters in the intermediate compression model may continue to be optimized until the compression convergence condition is reached, such as when the number of iterations (i.e., the number of optimizations) reaches a first preset number of iterations, or the compression loss value is less than a first preset loss threshold, it may be determined that the compression convergence condition is reached, and so on. Optionally, the first preset number of iterations and the first preset loss threshold may be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this.

Among them, the compression loss value can be determined based on the model loss value under each data channel, that is, the electronic device can determine the compression loss value based on the model loss value under each data channel; based on this, the model parameters in the initial compression model can be optimized in the direction of reducing the compression loss value.

In one embodiment, when determining the compression loss value based on the model loss value under each data channel, the electronic device may perform weighted summation on the model loss value under each data channel to obtain a weighted summation loss value, and use the weighted summation loss value as the compression loss value; optionally, the weight value of the model loss value under each data channel in the weighted summation process may be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

In another implementation, the electronic device may select the maximum model loss value from the model loss values under each channel of data, and use the selected model loss value as the compression loss value, and so on.

S303, respectively masking the compressed data source under each data path in each training data to obtain the masked data source under each data path in each training data, and respectively calling the initial language model corresponding to each data path, encoding the masked data source under the corresponding data path in each training data to obtain the encoded data source under each data path in each training data.

Specifically, for any training data in the training data set and any data in the M-path data, the electronic device can determine the mask probability, and determine at least one mask unit from the compressed data source under any data in any training data according to the mask probability, and one mask unit corresponds to a compressed semantic representation; based on this, the compressed semantic representation of each determined mask unit can be masked respectively to obtain the mask data source under any data in any training data, and the mask data source under any data in any training data includes the mask semantic representations of each determined mask unit. Accordingly, the mask data under any data in any training data can also include the compressed semantic representation of each semantic unit under any data in any training data except for each determined mask unit.

Optionally, the mask probability can be set according to experience or according to actual needs, and the embodiments of the present invention are not limited to this; an exemplary mask probability can be 15%. In this case, the compressed semantic units can be masked with a probability of 15%, that is, 15% of the compressed semantic representations of the semantic units can be randomly masked, and so on.

It can be seen that the embodiment of the present invention can mask the compressed semantic representation, so that the method mentioned in the embodiment of the present invention can be a sentence-level mask model pre-training method, that is, a sentence-level mask pre-training method, which can improve the semantic representation ability of single-channel data. Based on this, the pre-training of the target language model can be achieved by means of a mask language model (MLM), that is, the unsupervised pre-training of the mask language model can be performed for the compressed semantic representation, thereby improving the pre-training efficiency and further improving the model performance of the target language model; optionally, the language model can correspond to two training tasks, one training task can be to use the MLM method to enable the encoder of the Transformer (a model that uses the attention mechanism to increase the model training speed) to realize the task of fusing bidirectional features (i.e., the pre-training task), and the other training task can be a fine-tuning task.

Optionally, a language model may include an encoder; optionally, during the pre-training process, a language model may also include a masked language model (an output layer); then correspondingly, in a downstream task, a language model may include an encoder and an NLP (natural language processing) layer (another output layer); wherein the encoder included in the language model in the downstream task may be an encoder obtained through the pre-training process.

S304, based on the encoded data source under each data channel in each training data and the compressed data source under each data channel in each training data, respectively calculate the model loss value under each data channel, and optimize the model parameters in the initial language model corresponding to the corresponding data channel according to the model loss value under each data channel, so as to obtain the intermediate language model corresponding to each data channel.

S305, based on the training data set, iteratively optimize the model parameters in the intermediate language model corresponding to each channel of data until a convergence condition is reached, and obtain the to-be-associated language model corresponding to each channel of data.

It should be understood that after the intermediate language model corresponding to each channel of data is obtained, the model parameters in the intermediate language model corresponding to each channel of data may be continuously optimized until a convergence condition is reached.

Optionally, when the number of iterations reaches the second preset number of iterations, it is determined that the convergence condition is reached. At this time, the number of iterations of the language model to be associated corresponding to each data path can be the same or different, and the embodiment of the present invention does not limit this; that is, the second preset number of iterations corresponding to any two data paths can be the same or different, and the embodiment of the present invention does not limit this. Alternatively, it can also be determined that the convergence condition is reached when the model loss value reaches the second preset loss threshold, and so on; it should be noted that the second preset loss threshold corresponding to any two data paths can be the same or different, and the embodiment of the present invention does not limit this. It can be seen that one data path can correspond to one convergence condition, and the convergence conditions corresponding to any two data paths can be the same or different, and the embodiment of the present invention does not limit this; it should be understood that for any data path in the M data paths, when the convergence condition corresponding to any data path is reached, the language model to be associated corresponding to any path can be obtained. Optionally, the second preset number of iterations and the second preset loss threshold corresponding to each data path can be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this.

It should be understood that when the convergence condition corresponding to any one path of data is reached, it means that: the number of iterations reaches the second preset number of iterations corresponding to any one path of data; when the second preset number of iterations corresponding to each path of data is the same, the number of training times of the initial language models corresponding to each path of data can be the same, that is, the language models to be associated corresponding to each path of data can be obtained at the same time.

Optionally, when determining the compression loss value based on the model loss value under each data channel, if any data channel has reached the convergence condition in the previous iteration, that is, the model loss value of any data channel in the current iteration is empty, then the compression loss value can be determined based on the model loss value that is not empty among the model loss values under each data channel; or, it can be determined that the compression convergence condition is reached when any data channel reaches the convergence condition, and so on.

In an embodiment of the present invention, when iteratively optimizing the model parameters in the intermediate language model corresponding to each data path based on the training data set, if the compression model is a CNN semantic compression model, the compression model can be updated during the iteration process, so the compressed data source under any data path in any training data can be updated along with the update of the compression model; in this case, the above-mentioned compression processing of the initial data source under each data path in each training data included in the training data set can be iteratively performed to obtain the compressed data source under each data path in each training data, thereby realizing iterative training of the intermediate language model corresponding to each data path. Alternatively, if the compression model is a mean pooling module, the compression model may not be updated during the iteration process, so the compressed data source under any data path in any training data may remain unchanged; in this case, the above-mentioned masking of the compressed data source under each data path in each training data can be iteratively performed to obtain the masked data source under each data path in each training data, thereby realizing iterative training of the intermediate language model corresponding to each data path, and so on.

S306, determining the current compressed data source of each data channel in each training data, and based on the current compressed data source of each data channel in each training data, determining the data source to be encoded of each data channel in the corresponding training data.

In one embodiment, the electronic device can call the compression model at the current system time, and compress the initial data source under each data path in each training data to obtain the current compressed data source under each data path in each training data. Optionally, the compression model at the current system time can be a target compression model (i.e., a compression model when the compression convergence condition is met), or a compression model when the compression convergence condition is not met (at this time, the model training of the compression model can continue to be performed based on the following associated loss value, that is, the following associated loss value can be used as the compression loss value at the current system time for model training), etc.; the embodiment of the present invention is not limited to this.

In another implementation manner, the electronic device may use the compressed data source under each data path in each training data as the current compressed data source under the corresponding data path in the corresponding training data to determine the current compressed data source under each data path in each training data, and so on. Exemplarily, when the compression model is a mean pooling module, the compressed data source under each data path in each training data is the same, then the compressed data source under each data path in each training data may be directly used as the current compressed data source under each data path in each training data.

Optionally, the electronic device may use the current compressed data source under any data path in any training data as the data source to be encoded under any data path in any training data; or, it may mask the current compressed data source under any data path in any training data according to the above-mentioned masking probability to obtain the current masked data source under any data path in any training data, and use the current masked data source under any data path in any training data as the data source to be encoded under any data path in any training data, and so on.

S307, calling the to-be-associated language model corresponding to each data path, respectively encoding the to-be-encoded data source under the corresponding data path in each training data, and obtaining the current encoded data source under each data path in each training data.

Optionally, the electronic device may call the encoder in the language model to be associated corresponding to each path of data, and encode the data source to be encoded under the corresponding path of data in each training data, such as when a data source to be encoded is a corresponding current compressed data source; or, it may also call the encoder and mask language model in the language model to be associated corresponding to each path of data (that is, call the entire language model to be associated), and encode the data source to be encoded under the corresponding path of data in each training data, such as when a data source to be encoded is a corresponding current mask data source, and so on; the embodiment of the present invention is not limited to this.

S308, based on the current encoded data source under each data channel in each training data, calculate the association loss value, and optimize the model parameters in the language model to be associated corresponding to each data channel in the direction of reducing the association loss value, to obtain the intermediate language model to be associated corresponding to each data channel, so as to determine the target language model corresponding to each data channel based on the intermediate language model to be associated corresponding to each data channel.

Specifically, when calculating the associated loss value based on the current coded data source under each data path in each training data, the electronic device can determine at least one comparative learning group based on the current coded data source under each data path in each training data, and one comparative learning group includes the current coded data sources under any two data paths in each training data; then, each comparative learning group in at least one comparative learning group can be traversed, and the currently traversed comparative learning group can be used as the current comparative learning group, and the two data paths corresponding to the current comparative learning group can be used as the first data path and the second data path. Based on this, the loss value under the current comparative learning group can be calculated based on the current coded data source under the first data path in each training data and the current coded data source under the second data path in each training data; after traversing each comparative learning group in at least one comparative learning group, the loss value under each comparative learning group is obtained, and the associated loss value is calculated based on the loss value under each comparative learning group. Optionally, the loss values under each comparative learning group may be weightedly summed (such as mean operation or sum operation, etc.) to obtain an associated loss value; optionally, the weights of the loss values under each comparative learning group may be set according to experience or according to actual needs, and the embodiments of the present invention are not limited to this.

In one embodiment, when calculating the loss value of the current comparative learning group based on the current encoded data source under the first data path in each training data and the current encoded data source under the second data path in each training data, for any current encoded semantic representation under the first data path in any training data, the loss value of any current encoded semantic representation can be calculated based on any current encoded semantic representation under the first data path in any training data and the current encoded data source under the second data path in each training data, thereby obtaining the loss value of each current encoded semantic representation under the first data path in each training data; accordingly, the loss value of each current encoded semantic representation under the first data path in each training data can be weighted summed (such as mean operation or summation operation, etc.) to obtain the loss value of the current comparative learning group.

Optionally, the electronic device may use formula 1.1 to calculate the loss value of any current coded semantic representation based on any current coded semantic representation under the first path of data in any training data and the current coded data source under the second path of data in each training data. Exemplarily, W-1 may be the sum of the number of current coded semantic representations under the second path of data in each training data set except any training data, Ex may be the expectation between each current coded semantic representation under the first path of data in each training data set or the expectation between each current coded semantic representation under the second path of data in each training data set, etc., f(x) may be any current coded semantic representation under the first path of data in any training data, f(x ⁺ ) may be each current coded semantic representation under the second path of data in any training data (in this case, the number of f(x ⁺ ) may be multiple, and the numerator may be the sum of multiple products), and f(x _j ) may be the current coded semantic representation of the jth semantic unit under the second path of data in each training data set except any training data.

In another embodiment, when calculating the loss value of the current comparative learning group based on the current coded data source under the first data path in each training data and the current coded data source under the second data path in each training data, the semantic representation integration of the current coded data source under the first data path in each training data can be performed respectively (that is, the semantic representation integration of each current coded semantic representation under the first data path in any training data) to obtain the semantic representation integration result under the first data path in each training data, and the semantic representation integration of the current coded data source under the second data path in each training data can be performed respectively to obtain the semantic representation integration result under the second data path in each training data. Based on this, for any training data in the training data set, the loss value of any training data under the current comparative learning group can be calculated based on the semantic representation integration result under the first data path in any training data and the semantic representation integration result under the second data path in each training data, so as to obtain the loss value of each training data under the current comparative learning group; accordingly, the loss value of each training data under the current comparative learning group can be weighted summed to obtain the loss value under the current comparative learning group, and so on. Optionally, the above-mentioned semantic representation integration may refer to semantic representation splicing, that is, splicing together the various currently encoded semantic representations in the same currently encoded data source; or, semantic representation integration may also refer to mean operation, that is, performing mean operation on the various currently encoded semantic representations in the same currently encoded data source, and so on; the embodiments of the present invention are not limited to this.

Optionally, the electronic device may use formula 1.1 to calculate the loss value of any training data under the current comparative learning group based on the semantic representation integration result under the first data path in any training data and the semantic representation integration result under the second data path in each training data. Exemplarily, W can be the number of training data in the training data set, Ex can be the expectation between the semantic representation integration results under the first data path in each training data or the expectation between the language representation integration results under the second data path in each training data, etc., f(x) can be the semantic representation integration result under the first data path in any training data, f(x ⁺ ) can be the semantic representation integration result under the second data path in any training data, and f(x _j ) can be the semantic representation integration result under the second data path in the jth training data other than any training data in the training data set.

Further, when determining the target language model corresponding to each path of data based on the intermediate language model to be associated corresponding to each path of data, the electronic device may continue to perform model training (i.e., model pre-training) on the intermediate language model to be associated corresponding to each path of data until the associated convergence condition is reached (such as the number of iterations reaches a third preset number of iterations or the associated loss value is less than a third preset loss threshold, etc.), and obtain the target language model corresponding to each path of data. Optionally, the third preset number of iterations and the third preset loss threshold may be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

It can be seen that the embodiment of the present invention can perform comparative learning on the encoded semantic representations of each data channel, which can effectively close the representation of the semantic space, as shown in Figure 4; based on this, the embodiment of the present invention can make the subsequent merging of semantic representations between multiple data sources more reasonable after the semantic representation modeling of single-channel data (that is, after obtaining the language model to be associated corresponding to each channel of data).

Optionally, the encoder in the target language model corresponding to each data channel can be used in downstream tasks to determine the task language model corresponding to each data channel based on the encoder in the target language model corresponding to each data channel, and the task language model corresponding to one data channel may include the encoder in the target language model corresponding to the corresponding data channel. Optionally, the electronic device can also fine-tune the task language model corresponding to each data channel to obtain the target task language model corresponding to each data channel; wherein, the target task language model corresponding to each data channel can be applied to the target task, and the target task can be any downstream task, which is not limited in the embodiment of the present invention.

Optionally, after the electronic device determines the coded data source under each channel of data in the target data through the target language model or the target task language model corresponding to each channel of data, the electronic device also performs data fusion on the coded data source under each channel of data in the target data (such as sampling or averaging the coded data source under each channel of data, etc.), thereby obtaining a fused data source of the target data (including at least one fused semantic representation), etc.; it should be noted that the embodiments of the present invention do not limit the specific implementation methods of data fusion. Optionally, the target data may include the initial data source of the target object under each channel of data, and the target object may be any object, which is not limited by the embodiments of the present invention.

After obtaining the training data set, the embodiment of the present invention can compress the initial data sources under each data path in each training data included in the training data set to obtain the compressed data sources under each data path in each training data. One training data includes the initial data sources under each data path of an object in M data paths, and one data path corresponds to one language model, M is a positive integer; the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source. Then, the compressed data sources under each data path in each training data can be masked to obtain the masked data sources under each data path in each training data, and the initial language model corresponding to each data path can be called respectively, and the masked data sources under the corresponding data path in each training data can be encoded to obtain the encoded data sources under each data path in each training data. Based on this, the model loss value under each data path can be calculated respectively based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, and the model parameters in the initial language model corresponding to the corresponding data path can be optimized according to the model loss value under each data path to obtain the intermediate language model corresponding to each data path. Furthermore, based on the training data set, the model parameters in the intermediate language model corresponding to each data channel can be iteratively optimized until the convergence condition is reached, and the language model to be associated corresponding to each data channel is obtained; thereby determining the current compressed data source under each data channel in each training data channel, and based on the current compressed data source under each data channel in each training data channel, determining the data source to be encoded under each data channel in the corresponding training data channel. Then, accordingly, the language model to be associated corresponding to each data channel can be called to encode the data source to be encoded under the corresponding data channel in each training data channel, respectively, to obtain the current encoded data source under each data channel in each training data channel; and based on the current encoded data source under each data channel in each training data channel, calculate the associated loss value, and optimize the model parameters in the language model to be associated corresponding to each data channel in the direction of reducing the associated loss value, to obtain the intermediate language model to be associated corresponding to each data channel, so as to determine the target language model corresponding to each data channel based on the intermediate language model to be associated corresponding to each data channel. It can be seen that the embodiment of the present invention can realize unsupervised pre-training within the compressed text, that is, unsupervised pre-training of the masked language model can be performed for the compressed semantic representation, so as to solve the problem of slow training and reasoning speed caused by long text and/or multi-channel data through compression processing, so as to conveniently perform model pre-training to improve the model pre-training efficiency, that is, to effectively improve the training and reasoning speed of the model, and to improve the semantic representation ability of a single-channel data; in addition, when the value of M is greater than 1, the method mentioned in the embodiment of the present invention can be a pre-training method that combines multiple data sources, which can align the semantic space representations of different data, that is, to bring the encoded semantic representations under each channel of data closer, etc., so that different data sources of the same object have a strong correlation, that is, to be complementary, so as to obtain a better final semantic representation (that is, a fused semantic representation), which can improve the accuracy of the semantic representation. Based on this, the semantic representations of multiple data can be aligned through comparative learning technology, which can effectively improve the generalization ability of downstream tasks.

Based on the description of the relevant embodiments of the above-mentioned model pre-training method, the embodiment of the present invention further proposes a model pre-training device, which can be a computer program (including program code) running in an electronic device; as shown in FIG5 , the model pre-training device can include an acquisition unit 501 and a processing unit 502. The model pre-training device can execute the model pre-training method shown in FIG1 or FIG3 , that is, the model pre-training device can run the above-mentioned units:

An acquisition unit 501 is used to acquire a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, where M is a positive integer;

The processing unit 502 is used to compress the initial data sources of each data path in each training data included in the training data set to obtain compressed data sources of each data path in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source;

The processing unit 502 is further configured to respectively mask the compressed data sources under the respective paths of data in the respective training data to obtain the masked data sources under the respective paths of data in the respective training data, and respectively call the initial language models corresponding to the respective paths of data to encode the masked data sources under the corresponding paths of data in the respective training data to obtain the encoded data sources under the respective paths of data in the respective training data;

The processing unit 502 is further used to calculate the model loss value of each data path based on the encoded data source of each data path in the each training data and the compressed data source of each data path in the each training data, and optimize the model parameters in the initial language model corresponding to the corresponding data path according to the model loss value of each data path, so as to obtain the intermediate language model corresponding to each data path, and determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path.

In one implementation, when the processing unit 502 compresses the initial data sources of each path of data in each training data included in the training data set to obtain the compressed data sources of each path of data in each training data, it can be specifically used to:

Determine a text compression length for any training data in the training data set and any data in the M data;

According to the text compression length, data division is performed on the initial data source under any one of the data paths in any one of the training data to obtain N semantic representation groups to be compressed, wherein the number of semantic representations in one semantic representation group to be compressed is less than or equal to the text compression length;

Compression processing is performed on each of the N semantic representation groups to be compressed to obtain compressed semantic representations of each of the semantic representation groups to be compressed, so as to obtain a compressed data source under any one of the data in any one of the training data.

In another implementation manner, when the processing unit 502 respectively masks the compressed data sources under the respective data paths in the respective training data to obtain the masked data sources under the respective data paths in the respective training data, it can be specifically used to:

Determine a mask probability, and determine at least one mask unit from the compressed data source under any one of the data paths in any one of the training data according to the mask probability, wherein one mask unit corresponds to one compressed semantic representation;

The compressed semantic representation of each determined mask unit is masked respectively to obtain a mask data source under any one path of data in any one of the training data, and the mask data source under any one path of data in any one of the training data includes the mask semantic representation of each determined mask unit.

In another implementation, an encoded data source includes an encoded semantic representation of each mask unit in H mask units, where H is a positive integer; when the processing unit 502 calculates the model loss value of each data path based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, the processing unit 502 can be specifically used to:

For the m-th data in the M-th data, determine the encoding semantic representation of each mask unit under the m-th data in the corresponding training data from the encoding data source under the m-th data in each training data, m∈[1,M];

Calculate the model loss value of the m-th data path based on the encoded semantic representation of each mask unit under the m-th data path in the respective training data and the compressed data source under the m-th data path in the corresponding training data;

When the processing unit 502 optimizes the model parameters in the initial language model corresponding to the corresponding path data according to the model loss values under the respective paths of data, and obtains the intermediate language model corresponding to the respective paths of data, it can be specifically used to:

According to the model loss value under the m-th data, the model parameters in the initial language model corresponding to the m-th data are optimized to obtain the intermediate language model corresponding to the m-th data.

In another implementation manner, when the processing unit 502 determines the target language model corresponding to each path of data based on the intermediate language model corresponding to each path of data, it can be specifically used to:

Based on the training data set, iteratively optimize the model parameters in the intermediate language model corresponding to each channel of data until a convergence condition is reached, thereby obtaining the language model to be associated corresponding to each channel of data;

Determine the current compressed data source of each data path in each training data, and determine the data source to be encoded of each data path in the corresponding training data based on the current compressed data source of each data path in each training data;

Calling the to-be-associated language model corresponding to each path of data, respectively encoding the to-be-encoded data source under the corresponding path of data in each training data, to obtain the current encoded data source under each path of data in each training data;

Based on the current encoded data source under each data path in the each training data, an associated loss value is calculated, and the model parameters in the language model to be associated corresponding to each data path are optimized in the direction of reducing the associated loss value to obtain the intermediate language model to be associated corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model to be associated corresponding to each data path.

In another implementation manner, when the processing unit 502 calculates the association loss value based on the current encoded data source under each data path in each training data, it can be specifically used to:

Based on the current encoding data sources under the respective data paths in the respective training data, determining at least one comparative learning group, where one comparative learning group includes the current encoding data sources under any two data paths in the respective training data;

Traversing each comparative learning group in the at least one comparative learning group, and taking the currently traversed comparative learning group as the current comparative learning group, and taking the two paths of data corresponding to the current comparative learning group as the first path of data and the second path of data;

Calculate the loss value of the current comparative learning group based on the current encoded data source under the first path of data in each of the training data and the current encoded data source under the second path of data in each of the training data;

After traversing each comparative learning group in the at least one comparative learning group, the loss value under each comparative learning group is obtained, and the associated loss value is calculated based on the loss value under each comparative learning group.

In another implementation manner, the compressed data sources of the various data paths in the various training data are obtained by compressing the initial data sources of the various data paths in the various training data using an initial compression model, and the processing unit 502 may also be used to:

Based on the model loss values under the data of each channel, the model parameters in the initial compression model are optimized to obtain an intermediate compression model;

Based on the intermediate compression model, a target compression model is determined.

According to one embodiment of the present invention, each step involved in the method shown in FIG. 1 or FIG. 3 can be performed by each unit in the model pre-training device shown in FIG. 5. For example, step S101 shown in FIG. 1 can be performed by the acquisition unit 501 shown in FIG. 5, and steps S102-S104 can be performed by the processing unit 502 shown in FIG. 5. For another example, step S301 shown in FIG. 3 can be performed by the acquisition unit 501 shown in FIG. 5, and steps S302-S308 can be performed by the processing unit 502 shown in FIG. 5, and so on.

According to another embodiment of the present invention, each unit in the model pre-training device shown in Figure 5 can be separately or completely combined into one or several other units to constitute, or one (some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, any model pre-training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

According to another embodiment of the present invention, a model pre-training device as shown in FIG5 can be constructed, and the model pre-training method of the embodiment of the present invention can be implemented by running a computer program (including program code) capable of executing each step involved in the corresponding method as shown in FIG1 or FIG3 on a general electronic device such as a computer including a central processing unit (CPU), a random access storage medium (RAM), a read-only storage medium (ROM) and other processing elements and storage elements. The computer program can be recorded on, for example, a computer storage medium, and loaded into the above-mentioned electronic device through the computer storage medium, and run therein.

After obtaining the training data set, the embodiment of the present invention can compress the initial data sources under each data path in each training data included in the training data set to obtain the compressed data sources under each data path in each training data. One training data includes the initial data sources under each data path of an object in M data paths, and one data path corresponds to one language model, M is a positive integer; the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source. Then, the compressed data sources under each data path in each training data can be masked to obtain the masked data sources under each data path in each training data, and the initial language model corresponding to each data path can be called to encode the masked data sources under the corresponding data path in each training data to obtain the encoded data sources under each data path in each training data. Based on this, the model loss value under each data path can be calculated based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, and the model parameters in the initial language model corresponding to the corresponding data path can be optimized according to the model loss value under each data path to obtain the intermediate language model corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path. It can be seen that the embodiments of the present invention can conveniently perform model pre-training to improve the efficiency of model pre-training, that is, can effectively improve the speed of model training and reasoning.

Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

With reference to Fig. 6, the structural block diagram of the electronic device 600 that can be used as the server or client of the present invention will now be described, which is an example of the hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only used as examples, and are not intended to limit the implementation of the present invention described herein and/or required.

As shown in Figure 6, electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of electronic device 600 can also be stored. Computing unit 601, ROM 602 and RAM 603 are connected to each other via bus 604. Input/output (I/O) interface 605 is also connected to bus 604.

A plurality of components in the electronic device 600 are connected to the I/O interface 605, including: an input unit 606, an output unit 607, a storage unit 608, and a communication unit 609. The input unit 606 may be any type of device capable of inputting information to the electronic device 600, and the input unit 606 may receive input digital or character information, and generate key signal inputs related to user settings and/or function control of the electronic device. The output unit 607 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 608 may include, but is not limited to, a disk, an optical disk. The communication unit 609 allows the electronic device 600 to exchange information/data with other devices via a computer network such as the Internet and/or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and/or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and/or the like.

The computing unit 601 may be a variety of general and/or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above. For example, in some embodiments, the model pre-training method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 600 via ROM 602 and/or communication unit 609. In some embodiments, the computing unit 601 may be configured to perform the model pre-training method in any other appropriate manner (e.g., by means of firmware).

The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions/operations specified in the flow chart and/or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and/or data to a programmable processor.

To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope covered by the present invention.

Claims (10)

1. A model pre-training method, characterized by comprising: Obtain a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, where M is a positive integer; Performing compression processing on the initial data sources of each data path in each training data included in the training data set to obtain compressed data sources of each data path in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source; Masking the compressed data sources under each path of data in each of the training data respectively to obtain the masked data sources under each path of data in each of the training data, and respectively calling the initial language models corresponding to each path of data to encode the masked data sources under the corresponding path of data in each of the training data to obtain the encoded data sources under each path of data in each of the training data; Based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data, the model loss value under each data path is calculated respectively, and according to the model loss value under each data path, the model parameters in the initial language model corresponding to the corresponding data path are optimized respectively to obtain the intermediate language model corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path. 2. The method according to claim 1, characterized in that the step of compressing the initial data sources of the various data paths in the various training data included in the training data set to obtain the compressed data sources of the various data paths in the various training data comprises: Determine a text compression length for any training data in the training data set and any data in the M data; According to the text compression length, data division is performed on the initial data source under any one of the data paths in any one of the training data to obtain N semantic representation groups to be compressed, wherein the number of semantic representations in one semantic representation group to be compressed is less than or equal to the text compression length; Compression processing is performed on each of the N semantic representation groups to be compressed to obtain compressed semantic representations of each of the semantic representation groups to be compressed, so as to obtain a compressed data source under any one of the data in any one of the training data. 3. The method according to claim 2, characterized in that the step of respectively masking the compressed data sources under the respective data paths in the respective training data to obtain the masked data sources under the respective data paths in the respective training data comprises: Determine a mask probability, and determine at least one mask unit from the compressed data source under any one of the data paths in any one of the training data according to the mask probability, wherein one mask unit corresponds to one compressed semantic representation; The compressed semantic representation of each determined mask unit is masked respectively to obtain a mask data source under any one path of data in any one of the training data, and the mask data source under any one path of data in any one of the training data includes the mask semantic representation of each determined mask unit. 4. The method according to any one of claims 1 to 3, characterized in that an encoded data source includes an encoded semantic representation of each mask unit in H mask units, where H is a positive integer; and the step of calculating the model loss value of each data path based on the encoded data source under each data path in each training data and the compressed data source under each data path in each training data comprises: For the m-th data in the M-th data, determine the encoding semantic representation of each mask unit under the m-th data in the corresponding training data from the encoding data source under the m-th data in each training data, m∈[1,M]; Calculate the model loss value of the m-th data path based on the encoded semantic representation of each mask unit under the m-th data path in the respective training data and the compressed data source under the m-th data path in the corresponding training data; The optimizing the model parameters in the initial language model corresponding to the corresponding path data according to the model loss value under each path data, and obtaining the intermediate language model corresponding to each path data, comprises: According to the model loss value under the m-th data, the model parameters in the initial language model corresponding to the m-th data are optimized to obtain the intermediate language model corresponding to the m-th data. 5. The method according to any one of claims 1 to 3, characterized in that the step of determining the target language model corresponding to each path of data based on the intermediate language model corresponding to each path of data comprises: Based on the training data set, iteratively optimize the model parameters in the intermediate language model corresponding to each channel of data until a convergence condition is reached, thereby obtaining the language model to be associated corresponding to each channel of data; Determine the current compressed data source of each data path in each training data, and determine the data source to be encoded of each data path in the corresponding training data based on the current compressed data source of each data path in each training data; Calling the to-be-associated language model corresponding to each path of data, respectively encoding the to-be-encoded data source under the corresponding path of data in each training data, to obtain the current encoded data source under each path of data in each training data; Based on the current encoded data source under each data path in the each training data, an associated loss value is calculated, and the model parameters in the language model to be associated corresponding to each data path are optimized in the direction of reducing the associated loss value to obtain the intermediate language model to be associated corresponding to each data path, so as to determine the target language model corresponding to each data path based on the intermediate language model to be associated corresponding to each data path. 6. The method according to claim 5, characterized in that the calculation of the association loss value based on the current encoding data source under the respective data in the respective training data comprises: Based on the current encoding data sources under the respective data paths in the respective training data, determining at least one comparative learning group, where one comparative learning group includes the current encoding data sources under any two data paths in the respective training data; Traversing each comparative learning group in the at least one comparative learning group, and taking the currently traversed comparative learning group as the current comparative learning group, and taking the two paths of data corresponding to the current comparative learning group as the first path of data and the second path of data; Calculate the loss value of the current comparative learning group based on the current encoded data source under the first path of data in each of the training data and the current encoded data source under the second path of data in each of the training data; After traversing each comparative learning group in the at least one comparative learning group, the loss value under each comparative learning group is obtained, and the associated loss value is calculated based on the loss value under each comparative learning group. 7. The method according to any one of claims 1 to 3, characterized in that the compressed data sources of the various data paths in the various training data are obtained by compressing the initial data sources of the various data paths in the various training data through an initial compression model, and the method further comprises: Based on the model loss values under the data of each channel, the model parameters in the initial compression model are optimized to obtain an intermediate compression model; Based on the intermediate compression model, a target compression model is determined. 8. A model pre-training device, characterized in that the device comprises: An acquisition unit is used to acquire a training data set, where one training data set includes an initial data source of an object under each data path in M data paths, and one data path corresponds to one language model, where M is a positive integer; A processing unit, configured to compress the initial data sources of each path of data in each training data included in the training data set to obtain compressed data sources of each path of data in each training data, wherein the number of compressed semantic representations in one compressed data source is less than the number of initial semantic representations in the corresponding initial data source; The processing unit is further configured to respectively mask the compressed data sources under the respective paths of data in the respective training data to obtain the masked data sources under the respective paths of data in the respective training data, and respectively call the initial language models corresponding to the respective paths of data to encode the masked data sources under the corresponding paths of data in the respective training data to obtain the encoded data sources under the respective paths of data in the respective training data; The processing unit is further used to calculate the model loss value of each data path based on the encoded data source of each data path in the each training data and the compressed data source of each data path in the each training data, and optimize the model parameters in the initial language model corresponding to the corresponding data path according to the model loss value of each data path, so as to obtain the intermediate language model corresponding to each data path, and determine the target language model corresponding to each data path based on the intermediate language model corresponding to each data path. 9. An electronic device, comprising: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7. 10. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute the method according to any one of claims 1 to 7.