JP6743942B2 - Vocabulary table selection method, device, and computer-readable storage medium - Google Patents

Description

The present invention relates to the field of language information processing, and more particularly to a method for selecting a vocabulary table, an apparatus, and a computer-readable storage medium.

In natural language processing, when solving a natural language processing problem with a neural network model, it is usually necessary to specify a vocabulary table and train the neural network model to obtain a model having a specific function. For example, a model for extracting a named entity is trained by a vocabulary table containing other vocabularies of a person entity and a non-person entity, and the model finally obtained is It can be used to find a person name entity in the input natural language.

When training a model with a vocabulary table of the prior art, a sample vocabulary is usually extracted from the original corpus to obtain a vocabulary table consisting of the sample vocabulary, and then the model is trained based on the vocabulary table. The vocabulary table obtained based on the original corpus usually contains a large amount of vocabulary, and some low-value vocabulary exists, so when training the model with the vocabulary table, the training efficiency is low and the time required is long. Also, the model obtained by training is also inaccurate.

The technical problem to be solved by the embodiments of the present invention is to provide a vocabulary table selection method, a device, and a computer-readable recording medium to select and generate a vocabulary table more suitable for model training, thereby improving the model training efficiency. It can improve and reduce training time, and increase the accuracy of trained models.

In order to solve the above technical problems, the vocabulary table selection method provided by the embodiment of the present invention is as follows.
The vocabulary weighting layer is introduced into the target neural network model to build a preliminary training model, and the vocabulary weighting layer weights the target vocabulary in the first vocabulary table according to the vocabulary weight, and the target vocabulary obtained by the weighting process is used. Inputting to the target neural network model,
Based on the first vocabulary table, the preliminary training model is trained to update the model parameters of the preliminary training model and the vocabulary weights of the vocabulary weighting layer, and after the training, the vocabulary of the target vocabulary in the first vocabulary table. Obtaining the weights,
Selecting the first vocabulary table according to the vocabulary weight to obtain a second vocabulary table.

Preferably, after obtaining the second vocabulary table, further,
The target neural network model is trained by the second vocabulary table.

Preferably, in the above method, said step of weighting the target vocabulary in the first vocabulary table by said vocabulary weight comprises:
Performing a weighted sum for the target word vector corresponding to the target vocabulary and the unknown word vector corresponding to the unknown vocabulary, of which the first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary, The second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the sum of the first weight and the second weight is a predetermined value, the unknown vocabulary exists in the first vocabulary table All vocabularies that do not exist in the first vocabulary table correspond to the same unknown word vector.

Preferably, in the above method, the first weight may be a first function of vocabulary weights of the target vocabulary, such that the first function maps the vocabulary weights of the target vocabulary from 0 to 1. Used, the second weight is a second function of the first weight, and has a negative correlation with the first weight.

Preferably, in the above method, the target word vector and the unknown word vector are initialized by a random initialization or a word vector pre-training algorithm.

Preferably, in the above method, the step of selecting the first vocabulary table according to the vocabulary weight selects a second number of vocabularies from the first vocabulary table according to the order of increasing vocabulary weights, and selects the second vocabulary table. You can get a vocabulary table. Alternatively, the second vocabulary table can be obtained by selecting a vocabulary having a vocabulary weight within a preset numerical range from the first vocabulary table. Therefore, the number of vocabularies in the second vocabulary table is smaller than that in the first vocabulary table.

Preferably, in the above method, before constructing the preliminary training model, the method further comprises:
By performing data cleaning on the original corpus data, the original corpus data after the data cleaning is divided into sentences, and the sentences are divided to obtain a plurality of vocabularies. Select a number of vocabularies to get the first vocabulary table.

Preferably, in the above method, both the target neural network model and the preliminary training model are models built for the same target task.

Embodiments of the invention further provide a vocabulary table selection device, which comprises:
A vocabulary weighting layer is introduced into the target neural network model to build a preliminary training model, the vocabulary weighting layer weights the target vocabulary in the first vocabulary table according to the vocabulary weight, and the target vocabulary obtained by the weighting process is used. A modeling unit of a preliminary training model input to the target neural network model,
Based on the first vocabulary table, the preliminary training model is trained to update the model parameters of the preliminary training model and the vocabulary weights of the vocabulary weighting layer, and after the training, the vocabulary of the target vocabulary in the first vocabulary table. A first training unit for obtaining weights,
A vocabulary selection unit that selects the first vocabulary table according to the vocabulary weight to obtain a second vocabulary table.

Preferably, the vocabulary table selection device further comprises:
A second training unit is provided for training the target neural net model according to the second vocabulary table.

Preferably, in the vocabulary table selection device, the preliminary training model modeling module performs weighted summation on a target word vector corresponding to the target vocabulary and an unknown word vector corresponding to an unknown vocabulary, of which the The first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary, the second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the first weight and the second The sum of the weights is a predetermined value, the unknown vocabulary that does not exist in the first vocabulary table, and the vocabulary that does not exist in the first vocabulary table all correspond to the same unknown word vector.

Preferably, in the vocabulary table selection device, the first weight may be a first function of the vocabulary weight of the target vocabulary, and the first function may set the vocabulary weight of the target vocabulary to 0 to 1. Used for mapping, the second weight is a second function of the first weight and has a negative correlation with the first weight.

Preferably, in the vocabulary table selection device, the target word vector and the unknown word vector are initialized by stochastic initialization or a word vector preliminary training algorithm.

Preferably, in the vocabulary table selection device, the vocabulary selection unit selects a second number of vocabularies from the first vocabulary table in descending order of vocabulary weight to obtain the second vocabulary table. it can. Alternatively, the second vocabulary table can be obtained by selecting a vocabulary having a vocabulary weight within a preset numerical range from the first vocabulary table. Therefore, the number of vocabularies in the second vocabulary table is smaller than that in the first vocabulary table.

Preferably, the vocabulary table selection device further comprises:
The vocabulary table generation unit performs data cleaning on the original corpus data to divide the original corpus data after data cleaning into sentences, and also divides the sentence to obtain multiple vocabulary, and to determine the frequency of occurrence of the vocabulary in the original corpus. A first number of vocabularies is selected in descending order to obtain the first vocabulary table.

Preferably, in the vocabulary table selection device, both the target neural network model and the preliminary training model are models constructed for the same target task.

Embodiments of the present invention further provide a vocabulary table selection device, which includes a memory, a processor, and a computer program stored in the memory and executable by the processor. When the computer program is executed by the processor, the above vocabulary table selecting method can be realized.

Furthermore, a computer-readable recording medium is provided according to an embodiment of the present invention, and a computer program is stored in the computer-readable recording medium, and when the computer program is executed by a processor, the vocabulary table selecting method described above. Can be realized.

Compared with the prior art, the method of selecting a vocabulary table, the apparatus and the computer-readable storage medium according to the embodiment of the present invention trains a preliminary training model with a first vocabulary table, and calculates model parameters and vocabulary weights in a training process. After the training, the vocabulary weight of the target vocabulary obtained is used to further select the vocabulary in the first vocabulary table to obtain the second vocabulary table for training the target neural network model. Since the second vocabulary table contains high-value vocabulary, when training the target neural network model with the second vocabulary table, the training efficiency of the model can be improved and the training time can be reduced, and the accuracy of the trained model can be improved. Improve sex.

In order to describe the technical solution of the embodiments of the present invention more clearly, the following briefly introduces the accompanying drawings required for describing the embodiments of the present invention. Obviously, the following drawings illustrate the present invention. These attached drawings can obtain other attached drawings on the assumption that they are only some embodiments and do not require advanced technology for those skilled in the art.
It is an application scene of the vocabulary table selection method according to the embodiment of the present invention. It is a flow chat of the selection method of the vocabulary table according to the embodiment of the present invention. It is another flow chat of the selection method of the vocabulary table according to the embodiment of the present invention. It is another flow chat of the method of selecting a vocabulary table according to the embodiment of the present invention. 6 is a structural diagram of a named entity extraction model applied to a vocabulary table selection method according to an embodiment of the present invention. FIG. FIG. 6 is a structural diagram of a preliminary training model according to an embodiment of the present invention. FIG. 3 is a structural diagram of a vocabulary table selection device according to an embodiment of the present invention. FIG. 7 is another structural diagram of a vocabulary table selection device according to an embodiment of the present invention. It is another structural diagram of the selection device of the vocabulary table of the embodiment of the present invention.

In order to further clarify the technical problem, the technical solution, and the superior point to be solved by the present invention, a detailed description will be given with reference to the accompanying drawings and specific examples. In the following, specific details such as detailed arrangements and units are provided only for understanding the embodiments of the present invention. Therefore, it is obvious to those skilled in the art that various modifications and corrections can be made to the described embodiments within the scope of the gist of the present invention. Also, for clarity and brevity, descriptions of known functions and structures have been omitted.

It goes without saying that any "one embodiment" or "one embodiment" described in the specification includes a particular feature, structure or characteristic related to the embodiment in at least one embodiment of the present invention. Means Therefore, the terms “in one embodiment” or “in one embodiment” described in the specification do not necessarily mean the same embodiment. In addition, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

In each embodiment of the present invention, the number of each process below does not mean the order of execution, but the execution order of each process should be determined by its function and unique logic. No limitations should be placed on the process.

As mentioned above, polysemous words have a plurality of different semantic terms, so it is important to distinguish the meanings of polysemous words in different contexts. Therefore, the vocabulary table selection method according to the embodiment of the present invention can generate word expressions corresponding to different meaning terms in polysemous words, has a relatively small calculation amount of the method, and takes a short time. Therefore, the generation efficiency of word expressions can be improved.

FIG. 1 shows an exemplary system architecture 100 of an embodiment to which the vocabulary table selection method of the present application can be applied. As shown in FIG. 1, the system architecture 100 includes terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types such as wired, wireless communication links, or fiber optic cables.

The user can transmit text data and the like by using the terminal devices 101, 102 and 103 and interacting with the server 105 via the network 104. Various communication client applications such as a file editing application, an information search application, and an information transmission application can be mounted on the terminal devices 101, 102, 103. The terminal devices 101, 102, 103 are various electronic devices that have a display and transmit information and files, and include, but are not limited to, smartphones, tablet PCs, laptop PCs, desktop PCs, and the like.

The server 105 is a server capable of selecting a vocabulary table. Specifically, the server can collect original corpus data from the Internet 104 and generate a vocabulary table, for example, collect corpus data from a site on the Internet. Of course, it is possible to transmit the corpus data collected in advance from the terminal devices 101, 102, 103 to the server 105, but the vocabulary table selection method according to the embodiment of the present application is generally executed by the server 105. Correspondingly, a vocabulary table selection device can be installed in the server 105.

It should be understood, of course, that the number of terminal devices, networks, and servers in Figure 1 is only schematic. It can have any number of terminal devices, networks, servers depending on the needs to be realized.

FIG. 2A shows a flow chat of a method for selecting a vocabulary table according to an embodiment of the present invention, which can be applied to various neural net models that require training by designating a vocabulary table. The method can reduce the vocabulary table, improve the model training efficiency, reduce the training time, and improve the accuracy of the model obtained by training. As shown in FIG. 2A, the method of selecting the vocabulary table may include the following.

In step 201, a vocabulary weighting layer is introduced into the target neural network model to build a pre-training model, the vocabulary weighting layer weighting the target vocabulary in the first vocabulary table based on the vocabulary weights, and weighting. The target vocabulary obtained by the processing is input to the target neural network model.

Here, the target neural network model is a neural network model for which it is necessary to specify a vocabulary table for training, and the first vocabulary table is an original vocabulary table originally used for training the target neural network model. .. The vocabulary table selection method according to the embodiment of the present invention can reduce the vocabulary in the first vocabulary table.

In step 201, according to the embodiment of the present invention, a new layer structure, that is, a vocabulary weighting layer is added to the target neural network model to construct a new model referred to as a preliminary training model in the present application. In the preliminary training model, the vocabulary weighting layer weights the target vocabulary in the first vocabulary table based on the vocabulary weight, and inputs the target vocabulary obtained by the weighting process to the target neural network model. Therefore, the vocabulary weighting layer is provided between the input interface of the preliminary training model and the target neural network model. The vocabulary input to the preliminary training model is subjected to the above weighting processing and then input to the target neural network model. Here, the initial weight of the target vocabulary of the vocabulary weighting layer may be a preset initial value.

Specifically, from the vocabulary weighting layer, it is possible to perform a weighted sum for the target word vector corresponding to the target vocabulary and the unknown word vector corresponding to the unknown vocabulary, of which the first weight of the target word vector is The positive correlation function of the vocabulary weight of the target vocabulary, the second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the sum of the first weight and the second weight is a predetermined value. .. For example, the predetermined value is 1. The unknown vocabulary is a vocabulary that does not exist in the first vocabulary table and the vocabulary that does not exist in the first vocabulary table all correspond to the same unknown word vector.

Here, the target word vector and the unknown word vector can be initialized by stochastic initialization or a preliminary training algorithm of a word vector, where the preliminary training algorithm is a word vector corresponding to a vocabulary with a close sense. The Euclidean distance of can be reduced. It should be noted that the above is only two methods of word vector initialization that can be adopted by the embodiment of the present invention, and the embodiment of the present invention can also adopt other methods in the prior art. The embodiments of the invention are not specifically limited to this.

As one implementation, the first weight may be a first function of the vocabulary weight of the target vocabulary, and the first function is used to map the vocabulary weight of the target vocabulary from 0 to 1. .. The second weight may be a second function of the first weight and has a negative correlation with the first weight. It should be noted that the functions described above are only examples of embodiments of the present invention, and the embodiments of the present invention are not specifically limited thereto.

In step 202, based on the first vocabulary table, the preliminary training model is trained to update the model parameters of the preliminary training model and the vocabulary weights of the vocabulary weighting layer, and after the training, in the first vocabulary table. Get the vocabulary weight of the target vocabulary.

Here, in step 202, the preliminary training model is trained by the first vocabulary table, and the model parameter and the vocabulary weight of the target vocabulary of the vocabulary weighting layer are updated in the training process. In this way, when the training reaches the preset end condition, for example, when the iteration round reaches the predetermined number of times or the target function satisfies the predetermined condition, the training is terminated and the preliminary training model by the training is completed. And get the vocabulary weight of the target vocabulary of the vocabulary weighting layer.

In step 203, the first vocabulary table is selected according to the vocabulary weight to obtain a second vocabulary table.

Here, after obtaining the vocabulary weight of the target vocabulary in step 202, the first vocabulary table can be selected by the vocabulary weight so as to reduce the number of vocabularies required for model training, and the second vocabulary table can be obtained. Specifically, according to the embodiment of the present invention, the second vocabulary table can be obtained by selecting the second number of vocabularies from the first vocabulary table in the descending order of vocabulary weight. Alternatively, the second vocabulary table can be obtained by selecting a vocabulary having a vocabulary weight within a preset numerical range from the first vocabulary table. Therefore, the number of vocabularies in the second vocabulary table is smaller than that in the first vocabulary table. For example, one possible filtering method is: The second vocabulary table is constructed by retaining only the vocabulary whose vocabulary weight is larger than Q or smaller than -Q. Here, the Q is a parameter designated in advance by the user and is a positive real number.

By the above steps, the embodiment of the present invention can generate the second vocabulary table having relatively few vocabularies by the first vocabulary table, and the second vocabulary table includes the target vocabulary having high value. , Improve the training efficiency of the target neural network model, reduce the training time, and improve the accuracy of the model by training. For example, in a model for discovering personal name entities in natural language, it is possible to improve the accuracy of discovering personal name entities.

Further, as shown in FIG. 2B, the embodiment of the present invention may further include the following after step 203 described above.

In step 204, the target neural network model is trained according to the second vocabulary table.

Here, both the target neural network model and the preliminary training model are models built for the same target task. For example, the two models are both used to solve the same task, and both target to optimize the same objective function during training.

Since the number of vocabularies in the second vocabulary table was reduced and the vocabularies with high weight were included, the embodiments of the present invention improved the training efficiency of the target neural network model, reduced the training time, and obtained the training. The accuracy of the model can be increased.

FIG. 3 shows another process of the method for selecting a vocabulary table according to the embodiment of the present invention, which is a method for selecting the vocabulary table according to the embodiment of the present invention as a specific example of the target expression neural network model. This will be described in more detail. As shown in FIG. 3, the process includes:

In step 301, the first vocabulary table is generated by the original corpus.

Here, the original corpus collected in advance can be acquired, and the original corpus data is subjected to data cleaning to divide the original corpus data after data cleaning into sentences, and the sentence is divided to obtain a plurality of vocabularies. Then, the first number of vocabularies is selected according to the order of appearance frequency of the vocabulary in the original corpus, and the first vocabulary table is obtained.

In this embodiment, the method of selecting the vocabulary table can be executed by an electronic device (for example, the server 105 shown in FIG. 1), and the electronic device uses a wired connection method or a wireless connection method to connect to a network (for example, a website on the Internet). ), or text data can be collected by the terminal device 101, 102, 103, etc. and transmitted to the electronic device. The above wireless connection methods are 3G/4G/5G connection, WiFi connection, Bluetooth (registered trademark) connection, WiMAX connection, ZigBee connection, UWB (Ultra Wide Band) connection, and other known or future developed wireless connection methods. Can be included, but is not limited to.

Generally, since multiple encoding methods may be used for the text data collected from the network, as one implementation method, the embodiment of the present invention further includes the following procedure before the above step 301. You can

In step 300, the text data is collected and the text data is pre-processed to generate the original corpus required in the subsequent step 301. Specifically, the pretreatment includes:

a) Code unification: The text data is converted into the same code format. For example, all full-width characters are converted to half-width characters, and text data is converted to the same code in, for example, utf-8 format.

b) Data cleaning: Data cleaning is performed on the text data. Data washing removes noises unnecessary for text analysis from text data and retains only the contents including actual meaning information. Noise here usually means special symbols, links, emails, pictograms, emoticons, HTML tags (eg <html>, <title> and <body>, <br>, <span>, etc.) and other Symbols such as &lt, &gt, @, #, $, %, ^, &, *, (), <>, {}, [], and the like.

c) Data division: The text data after data washing is divided into morphemes, and stop words are removed to obtain the corpus data. For example, after the sentence is divided, the words are further divided into words according to different system application scenes, and the stop words are removed to obtain corpus data. Here, the morpheme includes at least one of a word, a collocation, and a word string. Stopwords typically include vocabularies that have no actual meaning, such as particles, nouns, and adverbs, and some high-frequency and low-frequency words. Conjunctions typically include two or more words, and word strings can include two or more collocations. Specifically, the text can be divided into sentences using the Natural Language Tool Kit (NLTK) of the Python library, and the sentence can be further divided into words by a tool such as a word divider. Both the word and the vocabulary in the text have the same meaning.

d) Data ID conversion: For the words divided in step c, the first number (for example, 30,000) of different words are selected according to the frequency of occurrence of the words to form the first vocabulary table, and Assign each unique ID. For example, among the 30,000 words in the first vocabulary table, 1 is assigned to the first word as an ID, 2 is assigned to the second word as an ID, and so on. For vocabularies that are not included in the first vocabulary table, that is, unknown vocabularies, 0 can be used as the IDs of all unknown vocabularies, so that each word can be replaced with the associated ID.

In step 302, a vocabulary weighting layer is introduced into the named entity extraction model to build a preliminary training model.

Here, FIG. 4 illustrates a structural diagram of the named entity extraction model, and FIG. 5 illustrates a structural diagram of the preliminary training model obtained by introducing the vocabulary weighting layer shown by the broken line frame into the named entity extraction model. is there. In an embodiment of the invention, each target vocabulary both has a unique vocabulary weight parameter.

Hereinafter, description will be made with reference to FIG.

a) Operator "+" represents vector addition. Assuming that the result of addition of vector A and vector B to n-dimensional vectors A and B is vector C, vector C is also an n-dimensional vector, and the i-th element C _i in vector C is Next.

So Ai is the i-th element in vector A and B _i is the i-th element in vector B.

b) The operator "x" represents vector multiplication. Assuming that the multiplication of A and b becomes a vector D for an n-dimensional vector A and a real number b, the vector D is also an n-dimensional vector, and the i-th element D _i in the vector D becomes ..

c) The operator “σ” corresponds to the activation function g(x). The activation function g(x) is a first function of the vocabulary weight x of the target vocabulary by calculating the first weight of the target word vector, and is used for mapping the vocabulary weight between 0 and 1. Equation 4 shows an example of the activation function.

d) The operator "1-" corresponds to the mapping function f(z). The mapping function f(z) is used to calculate the second weight of the unknown word vector and is the second function of the first weight z of the target word vector. The range of the mapping function f(z) is [0,1] and is negatively correlated with the first weight z of the target word vector, and decreases with increasing z, but increases in the opposite case. Equation 3 shows an example of the mapping function.

Therefore, the total value of the first weight and the second weight in this example becomes 1.

e) The word vector can be trained by the tool "word2vec", and the dimension of the word vector can be set to 256 during the training process. Vocabularies not included in the trained word vector can be probabilistically initialized to a 256-dimensional vector.

f) All vocabularies that are not within the 30,000 word range in the first vocabulary table are defined as unknown vocabularies. The unknown vocabulary shares the same stochastically initialized 256-dimensional vector as a word vector, that is, the unknown word vector.

g) In the embodiment of the present invention, the word vector parameters are collectively updated in the training process of the preliminary training model.

h) Each word together corresponds to a vocabulary weight for assessing the importance of the word. The vocabulary weight can be initialized to a positive real number of 0.5 and kept updated according to the training process of the preliminary training model.

i) The output of the vocabulary weighting layer is a weighted word vector. Therefore, in the vocabulary weighting layer in Fig. 5, the result of g(x) is multiplied by the target word vector of the target vocabulary, and the result of g(x) is also input to f(x), and the output of f(x) is unknown. The unknown word vector of the vocabulary is multiplied, the above two multiplication results are added, and the addition result is input to the proper expression extraction model as a weighted word vector corresponding to the target vocabulary.

Therefore, the output of the vocabulary weighting layer in Figure 5 is g(x)·A+(1-g(x))·U, where A represents the target word vector of the target vocabulary and U is the unknown word vector. And x represents the vocabulary weight of the target vocabulary. The weighted word vector is input to the proper expression extraction model of FIG. 4 and used for training the proper expression extraction model.

In step 303, a preliminary training model is trained based on the first vocabulary table.

As shown in FIG. 5, the preliminary training model inputs the vocabulary in the first vocabulary table to the vocabulary weighting layer and inputs the output of the vocabulary weighting layer to the named entity extraction model shown in FIG. The named entity extraction model shown in Fig. 4 has a standard bidirectional long short-term memory (LSTM) network structure. FIG. 4 shows an example of a sentence in which the input is three words, and in the case of inputting a longer sentence including more words, the illustrated structure can be repeated multiple times.

The structure shown in FIG. 4 will be described below.

a) Modules “x0”, “x1”, and “x2” each take an ID corresponding to the vocabulary in the sentence. These IDs are transformed into weighted word vectors by the vocabulary weighting layer.

b) The module "LSTM Cell" is the basic unit of the LSTM model. The forward and backward LSTM Cell use different model parameter sets.

c) The module "CONCAT" is an operator that connects two vectors. The output of the concatenation operation holds all the elements of the input vector, and the dimension of the output vector is the sum of the two input vector dimensions.

d) The module "SOFTMAX" is a standard Softmax layer for producing output. The parameters of the Softmax layer are shared by the current model.

e) Module "OUTPUT" is a one-dimensional binary output (0 or 1 output) that identifies whether the current vocabulary is a person name entity.

f) The model is optimized by Stochastic Gradient Descent (SGD), and model parameters and vocabulary weights are updated in the optimization process.

Step 304 filters the first vocabulary table based on the vocabulary weights to obtain the second vocabulary table.

Here, it is possible to arrange the vocabulary in descending order of the vocabulary weights obtained in step 303, and to select the second vocabulary table of the second number in the order of precedence, for example, the weight of the top 20,000 is large. Select words to compose the second vocabulary table.

Step 305 trains the target neural network model based on the second vocabulary table.

Here, the second vocabulary table is designated, and the proper expression extraction model is trained so that the proper expression extraction model does not include the vocabulary weighting layer, and the final target model can be learned.

Specifically, it is possible to perform a grid search using the named entity extraction model and adjust the hyperparameters of the model. Hyperparameters include batch size, learning rate, dropout rate and number of training iterations. And, based on the above hyperparameters, by training and initializing several (eg 9) models with different random seeds and combining them with the outputs of different models by the voting method, the final target model To get It should be pointed out that the embodiments of the present invention can adopt various existing model training implementation methods of the prior art, and the embodiments of the present invention are not specifically limited.

Based on the above method, the embodiment of the present invention further provides an apparatus for performing the above method, and according to FIG. 6, the vocabulary table selection apparatus 600 provided by the embodiment of the present invention includes the following.

The vocabulary weighting layer is introduced into the target neural network model by the modeling unit 601 of the preliminary training model to build the preliminary training model, and the vocabulary weighting layer weights the target vocabulary in the first vocabulary table based on the vocabulary weights. Then, the target vocabulary obtained by the weighting process is input to the target neural network model.

A first training unit 602 trains the preliminary training model based on the first vocabulary table to update model parameters of the preliminary training model and vocabulary weights of a vocabulary weighting layer, and after the training, Gets the vocabulary weight of the target vocabulary in the 1-vocabulary table.

The vocabulary selection unit 603 selects the first vocabulary table according to the vocabulary weight to obtain a second vocabulary table.

With the above units, the vocabulary table selection device 600 according to the embodiment of the present invention can simplify the first vocabulary table and select and generate the second vocabulary table more suitable for model training.

According to FIG. 7, another vocabulary table selection device 700 provided by the embodiment of the present invention includes the similar unit shown in FIG. 6, and further includes:

A second training unit 604 trains the target neural network model based on the second vocabulary table.

Here, the second training unit 604 trains the target neural network model with the second vocabulary table, but since the second vocabulary table contains fewer and higher quality vocabulary, it is necessary to improve the training efficiency and training. The time can be reduced and the accuracy of the learned model can be improved.

As one implementation method, in the vocabulary table selection device 600 or the vocabulary table selection device 700 according to the above-described embodiment, the modeling module 601 of the preliminary training model specifically includes a target word vector and an unknown vocabulary corresponding to the target vocabulary. The weighting is performed on the unknown word vector corresponding to, among them, the first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary, and the second weight of the unknown word vector is the target vocabulary. It is a negative correlation function of vocabulary weight, and the sum of the first weight and the second weight is a predetermined value. The unknown vocabulary is a vocabulary that does not exist in the first vocabulary table and the vocabulary that does not exist in the first vocabulary table all correspond to the same unknown word vector.

As one implementation, the first weight is a first function of the vocabulary weight of the target vocabulary, and the first function is used to map the vocabulary weight of the target vocabulary from 0 to 1. The second weight is a second function of the first weight and has a negative correlation with the first weight.

Here, the target word vector and the unknown word vector may be initialized by a probabilistic initialization or a word vector preliminary training algorithm.

As one implementation method, the vocabulary selection unit 603 is specifically used below. The second vocabulary table can be obtained by selecting a second number of vocabularies from the first vocabulary table according to the order of increasing vocabulary weight. Alternatively, the second vocabulary table can be obtained by selecting a vocabulary having a vocabulary weight within a preset numerical range from the first vocabulary table. Therefore, the number of vocabularies in the second vocabulary table is smaller than that in the first vocabulary table.

Further, the vocabulary table selection device 600 or the vocabulary table selection device 700 may include the following. An original corpus of vocabulary is obtained by dividing the original corpus data after data cleaning into sentences by dividing the original corpus data into data by performing data cleaning on the original corpus data by a vocabulary table generation unit (not shown). The first number of vocabularies is selected in the descending order of appearance frequency in to obtain the first vocabulary table.

It should be noted that, in the embodiment of the present invention, both the target neural network model and the preliminary training model are models constructed for the same target task, and, for example, in the training process, Use the same objective function as the optimization target.

According to FIG. 8, the embodiment of the present invention further provides the hardware structure of the vocabulary table selection device, as shown in FIG. 8, the vocabulary table selection device 800 includes a processor 802 and a computer program command. A memory 804 that is stored.

Here, when the computer program command is executed by the processor, it causes the processor 802 to perform the following steps.

A vocabulary weighting layer is introduced into the target neural network model to build a pre-training model, said vocabulary weighting layer being adapted to weight the target vocabulary in the first vocabulary table based on the vocabulary weights and obtained by the weighting process. The target vocabulary is input to the target neural network model.

Based on the first vocabulary table, the preliminary training model is trained to update the model parameters of the preliminary training model and the vocabulary weights of the vocabulary weighting layer, and after the training, the vocabulary of the target vocabulary in the first vocabulary table. Get the weight.

The second vocabulary table is obtained by selecting the first vocabulary table according to the vocabulary weight.

Further, as shown in FIG. 8, the vocabulary table selection device 800 further includes a network interface 801, an input device 803, a hard disk 805, and a display device 806.

The interfaces and devices are connected to each other via a bus architecture. The bus architecture may include any number of component interconnected buses and bridges. Specifically, one or a plurality of central processors (CPU) represented by the processor 802 and various circuits of one or a plurality of memories represented by the memory 804 are connected. The bus architecture can connect various other circuits together, such as peripheral devices, voltage regulators and power management circuits. Needless to say, the bus architecture is used to realize the articulated communication between these units. It is well known in the art that the bus architecture includes a power bus, a control bus, and a status signal bus in addition to the data bus, and thus detailed description thereof will be omitted.

The network interface 801 is connected to a network (for example, the Internet, a local area network, etc.), receives information from the network, stores the received information in the hard disk 805, and stores, for example, text data for generating the received corpus data. Save to hard disk 805.

The input device 803 receives various commands input by a worker and sends them to the processor 802 for execution. The input device 803 includes a keyboard or a click device (for example, a mouse, a track ball, a touch sensing plate, or a touch screen).

The display device 806 can display a result obtained by the processor 802 executing the command, for example, a generated second vocabulary table.

The memory 804 is used to store programs and data essential when the system is operating, and data such as intermediate results in the calculation process of the processor 802.

As can be appreciated, the memory 804 in embodiments of the invention may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Therefore, the nonvolatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM). .. Volatile memory can be random access memory (RAM), which is used as an external cache. The memory 804 of the devices and methods described herein includes, but is not limited to, these and any other compatible types of memory.

In some embodiments, memory 804 stores the following elements, which are executable modules or data structures, or sub-sets or extensions thereof, namely operating system 8041 and application programs 8042.

Here, the operating system 8041 includes various system programs such as a framework layer, a core layer, and a drive layer so as to realize various basic tasks and tasks based on hardware. The application program 8042 includes various application programs such as a browser so as to realize various application tasks. A program that implements the method of the embodiment of the present invention can be included in the application program 8042.

The method according to the above embodiments of the present invention can be applied to or implemented by the processor 802. The processor 802 may be an integrated circuit chip capable of processing signals. In an implementation process, each step of the above method can be completed by a hardware integrated logic circuit or a software type command in the processor 802. The processor 802 is a general purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field configurable circuit array (FPGA), discrete gate or transistor logic device, discrete hardware It may be a unit and may implement or execute each method, step and logic block diagram disclosed in the embodiments of the present invention. A general purpose processor may be a microprocessor or any conventional processor. Directly in combination with the steps of the method disclosed by the embodiments of the present invention, can be accomplished by execution by a hardware decoder processor or by a combination of hardware and software modules within the decoder processor. That is. The software module can be located in a storage medium mature in the art, such as random memory, flash memory, read only memory, programmable read only memory or electrically erasable programmable memory, register. The recording medium resides in memory 804, and processor 802 reads the information in memory 804 and combines it with its hardware to complete the steps of the method.

It will be appreciated that these embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode or combinations thereof. For hardware implementation, the process unit is one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gates. It can be realized in an array (FPGA), a general-purpose processor, a controller, a microcontroller, a microprocessor, or another electronic module or combination for realizing the functions of the present application.

With respect to software implementation, the techniques herein may be implemented by modules (eg, processes, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by the processor. The memory can be implemented within the processor or external to the processor.

Specifically, the computer program can implement the following steps when executed by the processor 802.

The target neural network model is trained by the second vocabulary table.

Specifically, the computer program can implement the following steps when executed by the processor 802.

The weighting is performed on the target word vector corresponding to the target vocabulary and the unknown word vector corresponding to the unknown vocabulary, of which the first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary, The second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the sum of the first weight and the second weight is a predetermined value. For example, the predetermined value is 1. The unknown vocabulary is a vocabulary that does not exist in the first vocabulary table and the vocabulary that does not exist in the first vocabulary table all correspond to the same unknown word vector.

The first weight can be a first function of the vocabulary weight of the target vocabulary, the first function is used to map the vocabulary weight of the target vocabulary from 0 to 1, and the second weight is It is a second function of the first weight and has a negative correlation with the first weight. The target word vector and the unknown word vector may be initialized by a probabilistic initialization or a word vector preliminary training algorithm.

Specifically, the computer program can implement the following steps when executed by the processor 802.

The second vocabulary table can be obtained by selecting a second number of vocabularies from the first vocabulary table according to the order of increasing vocabulary weight. Alternatively, the second vocabulary table can be obtained by selecting a vocabulary having a vocabulary weight within a preset numerical range from the first vocabulary table. Therefore, the number of vocabularies in the second vocabulary table is smaller than that in the first vocabulary table.

Specifically, the computer program can implement the following steps when executed by the processor 802.

By performing data cleaning on the original corpus data, the original corpus data after the data cleaning is divided into sentences, and the sentences are divided to obtain a plurality of vocabularies. Select a number of vocabularies to get the first vocabulary table.

It is preferable that both the target neural network model and the preliminary training model are models constructed for the same target task.

Those skilled in the art can combine each illustrated unit and algorithm step of the embodiments disclosed by the present specification, and can be realized by combining electronic hardware or computer software with electronic hardware. Whether these functions are executed by hardware or software depends on the specific application of the technical plan and design constraints. A person skilled in the art can implement the function using different methods depending on each identified application, but this implementation should not be considered to be beyond the scope of the present invention.

For a person skilled in the art, in order to make the description convenient and concise, reference may be made to the corresponding steps in the above-mentioned method embodiments regarding specific working steps of the above-mentioned system, apparatus and unit, and no detailed explanation will be given here.

It goes without saying that in the embodiments provided herein, the described method and apparatus can be implemented in other ways. For example, the device embodiments described above are exemplary only, for example, the partition of the module is only the logic function partition, and in practice other partition methods are possible. For example, multiple modules or units may be combined or assembled into another system, or certain features may be neglected or not implemented. Another, the shown or discussed coupling or direct coupling or communication connection between each other may be an indirect coupling or communication connection via an interface, device or module, which may be of electrical, mechanical or other form. Good.

The unit described as the separation part may or may not be physically separated, and the part displayed as the unit may be, but need not be, a physical unit, that is, in one place. It may be located and may be distributed to multiple network units. Depending on the actual needs, some or all of the units can be selected to realize the object of the embodiment plan of the present invention.

Further, each functional unit in each embodiment of the present invention may be integrated into one process unit, or each unit may physically exist alone, or two or more units may be provided. It is also possible for the units to be assembled into one unit.

The functions may be implemented in the form of software functional units and stored on a computer readable storage medium for sale or use in a separate product. As described above, the essence of the technical solution of the present invention, a part that contributes to the conventional technology, or a part of the technical solution can be realized in the form of a software product, and the computer software product is stored in a recording medium. Then, a computer device (which may be a personal computer, a server, a network facility, or the like) is caused to execute some or all steps of the method described in each embodiment of the present application by including some commands. The recording medium includes a medium capable of storing various program codes such as USB, mobile hard disk, ROM, RAM, magnetic disk or optical disk.

The above is a specific implementation method of the present invention, the protection scope of the present invention is not limited to this, and those skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present invention. It is included in the scope of protection of the invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (11)

The vocabulary weighting layer is introduced into the target neural network model to build a preliminary training model, and the vocabulary weighting layer weights the target vocabulary in the first vocabulary table according to the vocabulary weight, and the target vocabulary obtained by the weighting process is used. Inputting to the target neural network model,
Based on the first vocabulary table, the preliminary training model is trained, model parameters of the preliminary training model and vocabulary weights of the vocabulary weighting layer are updated, and after the training, the vocabulary weights of the target vocabulary in the first vocabulary table are updated. To get
Selecting the first vocabulary table according to the vocabulary weight to obtain a second vocabulary table,
A method for selecting a vocabulary table characterized by including. After obtaining the second vocabulary table, the selection method further comprises training the target neural net model with the second vocabulary table,
The selection method according to claim 1, wherein: Weighting the target vocabulary in the first vocabulary table by the vocabulary weights,
A weighted sum is performed on the target word vector corresponding to the target vocabulary and the unknown word vector corresponding to the unknown vocabulary, of which the first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary. , The second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the sum of the first weight and the second weight is a predetermined value, the unknown vocabulary in the first vocabulary table 3. The selection method according to claim 1, further comprising: all vocabularies that do not exist and that do not exist in the first vocabulary table correspond to the same unknown word vector. The first weight is a first function of the vocabulary weight of the target vocabulary, the first function maps the vocabulary weight of the target vocabulary from 0 to 1,
The second weight is a second function of the first weight, and has a negative correlation with the first weight,
4. The selection method according to claim 3, wherein: The step of selecting the first vocabulary table according to the vocabulary weights,
The second vocabulary table is obtained by selecting a second number of vocabularies from the first vocabulary table in descending order of vocabulary weight, or the vocabulary weight from the first vocabulary table falls within a preset numerical range. Selecting a vocabulary to obtain the second vocabulary table,
The number of vocabularies in the second vocabulary table is less than the number of vocabularies in the first vocabulary table,
The selection method according to claim 1 or 2, characterized in that. A model built for the same target task together with the target neural network model and the preliminary training model,
The selection method according to claim 1 or 2, characterized in that. A vocabulary weighting layer is introduced into the target neural network model to build a preliminary training model, the vocabulary weighting layer weights the target vocabulary in the first vocabulary table according to the vocabulary weight, and the target vocabulary obtained by the weighting process is used. A modeling unit of a preliminary training model input to the target neural network model,
Based on the first vocabulary table, the preliminary training model is trained, model parameters of the preliminary training model and vocabulary weights of the vocabulary weighting layer are updated, and after training, the vocabulary weights of the target vocabulary in the first vocabulary table are updated. The first training unit to get
A vocabulary selection unit that selects the first vocabulary table according to the vocabulary weight and obtains a second vocabulary table;
A vocabulary table selection device comprising: The selection device according to claim 7, further comprising: a second training unit that trains the target neural network model according to the second vocabulary table. A weighted sum is performed on the target word vector corresponding to the target vocabulary and the unknown word vector corresponding to the unknown vocabulary, of which the first weight of the target word vector is a positive correlation function of the vocabulary weight of the target vocabulary. , The second weight of the unknown word vector is a negative correlation function of the vocabulary weight of the target vocabulary, and the sum of the first weight and the second weight is a predetermined value, the unknown vocabulary in the first vocabulary table Vocabularies that do not exist and that do not exist in the first vocabulary table all correspond to the same unknown word vector,
The selection device according to claim 7 or 8, characterized in that. A program for causing a computer to execute the method of selecting a vocabulary table according to any one of claims 1 to 6. A computer-readable storage medium storing the program according to claim 10.

Images