CN106569998A - Text named entity recognition method based on Bi-LSTM, CNN and CRF - Google Patents

Abstract

The invention discloses a text named entity recognition method based on Bi-LSTM, CNN and CRF. The method includes the following steps: (1) using a convolutional nerve network to encode and convert information on text word character level to a character vector; (2) combining the character vector and word vector into a combination which, as an input, is transmitted to a bidirectional LSTM neural network to build a model for contextual information of every word; and (3) in the output end of the LSTM neural network, utilizing continuous conditional random fields to carry out label decoding to a whole sentence, and mark the entities in the sentence. The invention is an end-to-end model without the need of data pre-processing in the un-marked corpus with the exception of the pre-trained word vector, therefore the invention can be widely applied for statement marking of different languages and fields.

Description

A kind of text based on Bi-LSTM, CNN and CRF names entity recognition method

Technical field

The present invention relates to natural language processing, more particularly to it is a kind of based on two-way LSTM neutral nets, convolutional neural networks Entity recognition method is named with the text of condition random field (CRF).

Background technology

Natural language processing (Nature Language Processing, abbreviation NLP) is a collection linguistics and calculating The cross discipline that machine science is integrated.Name Entity recognition (Named Entity Recognition, abbreviation NER) is nature language A basic task in speech process, it is intended to identify the proper noun and significant numeral classifier phrase in natural language text, And classified.With the rise of information extraction and big data concept, name Entity recognition task is increasingly subject to people's attention, Become the important component part of the natural language processings such as the analysis of public opinion, information retrieval, automatic question answering, machine translation.How from sea Automatically identify name entity in the internet text message of amount, accurately and rapidly, be increasingly becoming academia and industrial quarters is closed The hot issue of note.

High performance name entity recognition method is normally applied condition random field (CRF), SVM and perceptron to be come to manual place The feature of reason is modeled.Some researchers propose a kind of efficient neutral net mould for only needing to a small amount of Feature Engineering Type, the model need not from a large amount of unlabeled data learnings to primary word vector characteristics, but by unsupervised learning Method extracted vector feature from mass data.

But this model has many restrictions.First, the model make use of a simple feedforward neural network, the network The contextual information of each word is limited among the window of regular length, this method is abandoned between long range word Information；Secondly because only rely on term vector information, the method can not probe into word character level characteristics, such as prefix Information and suffix information etc., these information are often very useful, especially in the new word of the very poor term vector expression of training effect In.We seek a kind of significantly more efficient neural network model that can solve the problem that the problems referred to above.

Recurrent neural network (Recurrent neural network, abbreviation RNN) can process the input of variable-length Vector, and can have memory over a long time in processing procedure.In recent years, RNNs various NLP tasks (such as speech recognition, Machine translation and Language Modeling etc.) there is huge success in process.Shot and long term with forget gate memory artificial neuron The important dependence of long-distance dependence can easily be learned for network (Long-short term memory, abbreviation LSTM) Practise.

It is upper for the uncertain length of word for the sequence mark task of name Entity recognition and speech recognition etc. Hereafter problem, the restricted problem of context in other words, two-way LSTM (Bi-LSTM) neutral net is efficient：When LSTMs is in life In name Entity recognition task by past information to learn when, the restriction of computing capability and the quality of term vector can limit them Efficiency.

Convolutional neural networks (Convolutional neural network, abbreviation CNN) are equally used in NLP tasks The information of character aspect is modeled, and existing Successful utilization CNN extracts character feature and applies features to name in fact The example gone in body identification mission.

The content of the invention

The invention aims to the entity in natural text is detected and is labeled, to obtain effective text This information, proposes that a kind of text name based on two-way LSTM neutral nets, convolutional neural networks and condition random field (CRF) is real Body recognition methods.

Text based on Bi-LSTM, CNN and CRF names entity recognition method, comprises the steps：

(1) data prediction is carried out according to text data, subordinate sentence, participle is carried out to text, obtain character feature vector；

(2) character feature vector is carried out to each word using convolutional neural networks to extract；

(3) character feature vector is combined with term vector, LSTM neutral nets is passed to, using two-way LSTM neutral nets Read statement information characteristics are obtained to train；

(4) for semantic feature resulting in (3), entity mark is carried out to each word using condition random field, Mark the entity information in statement sequence.

Above steps can specifically using mode is implemented as described below：

Data prediction is carried out to text data, subordinate sentence, participle are carried out to text, the step for obtaining character feature vector can It is implemented as follows：

(1) subordinate sentence and participle are carried out to document using natural language processing instrument so that document is with sentence and one per Token is presented；

(2) sentence, word and the label to obtaining in previous step (1) is counted, and forms sentence table, vocabulary and label Table；

(3) character in the word list in previous step (1) is counted, forms character list；

(4) character feature vector is formed using the good term vector of existing pre-training and character list.

Described utilization convolutional neural networks are carried out to each word can be specifically real the step of character feature vector is extracted It is now as follows：

(1) C is made to be character list, d is the dimension of each character vector, and character vector matrix is：Q∈R^d×|C|；

(2) word k ∈ V are preset by continuous character [c₁,c₂,...,c_l] composition, wherein l for word k length, then k Character vector matrix is by C^k∈R^d×lBe given, wherein i-th is classified as character c_iVector；

(3) in C^kWith the kernel H ∈ R that a width is w^d×wBetween realize convolutional layer, it is inclined to convolutional layer addition after this Value bias is put, and whole convolution results are carried out nonlinear regression to realize Feature Mapping map f^k∈R^l-w+1, wherein, map letter Number f^kI-th element f^k[i] is given by (1) formula；

f^k[i]=tanh (<C^k[*,i:i+w-1],H>+b) (1)

Wherein * be all line numbers, C^k[*,i:I+w-1] it is C^kIn i-th row to the i-th+w-1 row,<,>For in Frobenius Product, b is bias vector；

(4) it is last, withExpress as the character pair of kernel H.

By character feature vector combine with term vector, be passed to LSTM neutral nets, using two-way LSTM neutral nets come The step of training obtains read statement information characteristics is as follows：

(1) x is defined_tFor the input character feature vector of t, h_tIt is the hidden layer that all useful informations are stored in t State vector, σ is that sigmoid returns layer, and * is inner product, U_i,U_f,U_c,U_oTo be directed to input x under different conditions_tWeight matrix, W_i,W_f,W_c,W_oTo hide layer state h_tWeight matrix, b_i,b_f,b_c,b_oFor bias vector；

(2) t forget gate calculating as shown in (2) formula：

f_t=σ (W_fh_t-1+U_fx_t+b_f) (2)

(3) h is updated in t_t-1All information of middle storage, computing formula is as shown in (3), (4) formula：

i_t=σ (W_ih_t-1+U_ix_t+b_i) (3)

WhereinFor the vector that t be introduced into cell state；

(4) information updating for storing the t-1 moment in t is the storage information of t, and computing formula is formula (5)：

WhereinFor the vector of t cell state；

(5) it is shown in the output such as formula (6) of t, and update h_t, computing formula such as formula (7)：

o_t=σ (W_oh_t-1+U_ox_t+b_o) (6)

h_t=o_t*tanh(C_t) (7)

Wherein o_tFor the output of t；h_tFor the vector of t hidden layer；

(6) h in above-mentioned steps_tAll information of storage last time, with same method a g is arranged again_tFor depositing Following information of storage, most latter two hidden layer information forms last output vector by cascading.

Entity mark is carried out to each word using condition random field, marking the entity information in statement sequence can have Body realizes that step is as follows：

(1) with z={ z₁,z₂,...,z_nRepresent list entries, wherein n for list entries length, z_iFor i-th word Input vector, y={ y₁,y₂,...,y_nFor z reality output sequence label, Y (z)={ y'₁,y'₂,...,y'_nFor z can The output label sequence of energy；

(2) bar of the condition random field probabilistic model definition for given list entries z on be possible to sequence label y' Part Probability p (y | z；W,b)：

Wherein,For potential gain function, W is weight vectors, and b is bias vector；And b_y',yRespectively with label to (y', y) corresponding weight vectors and bias vector；

(3) condition random field (CRF) training stage, use maximal condition Likelihood estimation select cause likelihood ratio L (W, B) maximum parameter, for training set { (z_i,y_i), the logarithm of likelihood ratio is：

(4) find the y of highest conditional probability in sequence label to carry out label for labelling to sequence：

Present invention beneficial effect compared with prior art：

A kind of combined with two-way LSTM neutral nets using convolutional neural networks method of the present invention come to character feature and Word feature is extracted, contrast and traditional method such as hidden Markov model, the present invention using vocabulary and context according to Bad relation, and carry out Probability statistic abstraction on this basis, so as to obtain text message in entity information；Using nerve net The mode that network is combined with condition random field, application scenarios extensively, only need to be trained one time for the corresponding different pieces of information of different problems, With regard to can solve the problem that multi-field name entity mark problem, possess advantage end to end.

Description of the drawings

Fig. 1 is the committed step workflow diagram that the text based on Bi-LSTM, CNN and CRF names Entity recognition；

Fig. 2 is convolutional neural networks structural representation；

Fig. 3 is two-way LSTM neutral nets schematic diagram.

Fig. 4 is based on the Named Entity Extraction Model training process algorithm of two-way LSTM and CRF.

Specific embodiment

The present invention is further elaborated with reference to the accompanying drawings and detailed description.

Present invention is generally directed to name Entity recognition task by above neural network ensemble, present a kind of based on two-way The mixed model of LSTM, CNN, the model can be while learning character characteristic vector and term vector, on this basis arrive study Characteristic vector label for labelling is carried out by condition random field.Thus obtaining the name Entity recognition result of efficiently and accurately. As shown in figure 1, be the committed step workflow diagram of the text name Entity recognition based on Bi-LSTM, CNN and CRF, wherein often The character list of individual word reaches and is completed by the convolutional neural networks structure of Fig. 2；Character feature vector is with term vector after cascade Result be input in two-way LSTM neutral nets, wherein LSTM neutral nets as shown in Figure 3, in Fig. 1 dotted line represent input There is one layer of dropout layer (preventing neutral net from producing over-fitting) in vector sum output vector.

The text based on Bi-LSTM, CNN and CRF of the present invention names entity recognition method, its concrete steps (1)～(4) It is as follows：

(1) data prediction is carried out according to text data, subordinate sentence, participle is carried out to text, obtain character feature vector.Tool Body step is as follows：

(1.1) subordinate sentence and participle are carried out to document using natural language processing instrument so that document is with sentence and per one Individual word (token) is presented；

(1.2) sentence, word and label are counted, forms sentence table, vocabulary and label list；

(1.3) character in word list is counted, forms character list；

(1.4) character feature vector is formed using the good term vector of existing pre-training and character list.

(2) character feature vector is carried out to each word using convolutional neural networks to extract.Comprise the following steps that：

(2.1) C is made to be character list, d is the dimension of each character vector, and character vector matrix is：Q∈R^d×C；

(2.2) word k ∈ V are preset by continuous character [c₁,c₂,...,c_l] composition, wherein l is the length of word k, then k Character vector matrix by C^k∈R^d×lBe given, wherein i-th is classified as character c_iVector；

(2.3) in C^kWith the kernel H ∈ R that a width is w^d×wBetween realize convolutional layer, after this give convolutional layer addition Bias bias, and whole convolution results are carried out nonlinear regression to realize Feature Mapping map f^k∈R^l-w+1, wherein, mapping Function f^kI-th element f^k[i] is given by (1) formula；

f^k[i]=tanh (<C^k[*,i:i+w-1],H>+b) (1)

Wherein * be all line numbers, C^k[*,i:I+w-1] it is C^kIn i-th row to the i-th+w-1 row,<,>For in Frobenius Product, b is bias vector；

(2.4) it is last, withExpress as the character pair of kernel H.

(3) character feature vector is combined with term vector, is passed to LSTM neutral nets.As shown in Figure 4, using two-way LSTM neutral nets obtain read statement information characteristics to train.Comprise the following steps that：

(3.1) x is defined_tFor the input character feature vector of t, h_tIt is to store hiding for all useful informations in t Layer state vector, σ is that sigmoid returns layer, and * is inner product, U_i,U_f,U_c,U_oTo be directed to input x under different conditions_tWeight square Battle array, W_i,W_f,W_c,W_oTo hide layer state h_tWeight matrix, b_i,b_f,b_c,b_oFor bias vector；

(3.2) t forget gate calculating as shown in (2) formula：

f_t=σ (W_fh_t-1+U_fx_t+b_f) (2)

(3.3) h is updated in t_t-1All information of middle storage, computing formula is as shown in (3), (4) formula：

i_t=σ (W_ih_t-1+U_ix_t+b_i) (3)

WhereinFor the vector that t be introduced into cell state；

(3.4) information updating for storing the t-1 moment in t is the storage information of t, and computing formula is formula (5)：

WhereinFor the vector of t cell state；

(3.5) it is shown in the output such as formula (6) of t, and update h_t, computing formula such as formula (7)：

o_t=σ (W_oh_t-1+U_ox_t+b_o) (6)

h_t=o_t*tanh(C_t) (7)

Wherein o_tFor the output of t；h_tFor the vector of t hidden layer；

(3.6) h in above-mentioned steps_tAll information of storage last time, with same method a g is arranged again_tFor Following information of storage, most latter two hidden layer information forms last output vector by cascading.

(4) for semantic feature resulting in (3), entity mark is carried out to each word using condition random field, Mark the entity information in statement sequence.Comprise the following steps that：

(4.1) with z={ z₁,z₂,...,z_nRepresent list entries, wherein n for list entries length, z_iFor i-th word Input vector, y={ y₁,y₂,...,y_nFor z reality output sequence label, Y (z)={ y'₁,y'₂,...,y'_nFor z's Possible output label sequence；

(4.2) condition random field probabilistic model is defined for given list entries z is on be possible to sequence label y' Conditional probability p (y | z；W,b)：

Wherein,For potential gain function, W is weight vectors, and b is bias vector；And b_y',yRespectively with label to (y', y) corresponding weight vectors and bias vector；

(4.3) condition random field (CRF) training stage, maximal condition Likelihood estimation is used to select to cause likelihood ratio L (W, b) maximum parameter, for training set { (z_i,y_i), the logarithm of likelihood ratio is：

(4.4) find the y of highest conditional probability in sequence label to carry out label for labelling to sequence：

Embodiment

By taking the document of New York Times English news as an example, text name entity is carried out during said method is applied in text and is known Not, design parameter and way are as follows in each step：

1. subordinate sentence and participle are carried out to document using natural language processing instrument so that document each word is a line, sentence It is spaced apart with space between sentence；

2. the sentence in 1, word and label counted respectively, sentence table, vocabulary and label list are formed, in Training document Label has " PER (name) " " LOC (place name) " " ORG (tissue) " " FAC (mechanism) " and " GPE (geopolitical name) " five classes, surveys Label is " * " in examination document, Jing statistics, the total sentence 17 of document, word 466 in 1；

3. word list enters line character statistics in pair 1, forms character list C；

4. using the dimensional vectors of GloVe 100 composition 100 in combination with vocabulary in 2 disclosed in 600,000,000 Stamfords for having trained Dimension term vector；

Character feature vector is carried out to each word with convolutional neural networks to extract：

5. make C be character list, d is the dimension of each character vector, and character vector matrix is：Q∈R^d×|C|；

6. assume word k ∈ V by continuous character [c₁,c₂,...,c_l] composition, wherein l for word k length, then k Character vector matrix is by C^k∈R^d×lBe given, wherein i-th is classified as character c_iVector；

7. in C^kWith the kernel H ∈ R that a width is w^d×wBetween realize convolutional layer, it is inclined to convolutional layer addition after this Value bias is put, and whole convolution results are carried out nonlinear regression to realize Feature Mapping map f^k∈R^l-w+1, especially, f^k's I-th element is given by (1) formula：

f^k[i]=tanh (<C^k[*,i:i+w-1],H>+b) (1)

Wherein * be all line numbers, C^k[*,i:I+w-1] it is C^kIn i-th row to the i-th+w-1 row,<A,B>=Tr (AB^T) be Frobenius inner products.

8. withExpress as the character pair of kernel H；

Using term vector resulting in character vector feature and 4 resulting in 8 as being input to two-way LSTM nerve nets In network：

9. the mode that two-way LSTM neural network parameters update is that, with 30 as batchsize, 0.9 is the boarding steps of momentum term Degree descent algorithm updates, and initialization learning rate is 0.015, and after each iteration, learning rate more new formula is

η_t=η₀(1+ρt) (1)

Wherein retardation rate ρ=0.05, t is the number of times for completing iteration；

10. x is defined_tFor the input character feature vector of t, h_tIt is the hidden layer that all useful informations are stored in t State vector, σ is that sigmoid returns layer, and * is inner product, U_i,U_f,U_c,U_oTo be directed to input x under different conditions_tWeight matrix, W_i,W_f,W_c,W_oTo hide layer state h_tWeight matrix, b_i,b_f,b_c,b_oFor bias vector；

11. t forget gate calculating as shown in (2) formula：

f_t=σ (W_fh_t-1+U_fx_t+b_f) (2)

12. update h in t_t-1All information of middle storage, computing formula is as shown in (3), (4) formula：

i_t=σ (W_ih_t-1+U_ix_t+b_i) (3)

13. information updatings for storing the t-1 moment in t are the storage information of t, and computing formula is formula (5)：

14. is shown in the output such as formula (6) of t, and updates h_t, computing formula such as formula (7)：

o_t=σ (W_oh_t-1+U_ox_t+b_o) (6)

h_t=o_t*tanh(C_t) (7)

H in 15. above-mentioned steps_tAll information of storage last time, with same method a g is arranged again_tFor depositing Following information of storage, most latter two hidden layer information forms last output vector by cascading.

Entity mark is carried out to each word using condition random field (CRF)：

16. use z={ z₁,z₂,...,z_nRepresent list entries, wherein z_iFor the input vector of i-th word, y={ y₁, y₂,...,y_nFor z reality output sequence label, Y (z)={ y'₁,y'₂,...,y'_nFor z possible output label sequence；

17. condition random fields (CRF) probabilistic model is defined for given list entries z is on be possible to sequence label y Conditional probability p (y | z；W,b)：

Wherein,For potential gain function, W is weight vectors, and b is bias vector；And b_y',yRespectively with label to (y', y) corresponding weight vectors and bias vector；

18. condition random fields (CRF) training stage, we use maximal condition possibility predication, for training set { (z_i, y_i), the logarithm of likelihood ratio is：

Maximum likelihood training process selects to cause likelihood ratio L (W, b) maximum parameter.

19. find the y of highest conditional probability to carry out label for labelling to sequence in sequence label：

20. station location markers by the word for having marked in original go out, and annotation results are put in order feed back to user.

Following table is the final annotation results in part of selected news documents.

Claims (5)

1. a kind of text based on Bi-LSTM, CNN and CRF names entity recognition method, it is characterised in that comprise the steps：

(1) data prediction is carried out according to text data, subordinate sentence, participle is carried out to text, obtain character feature vector；

(2) character feature vector is carried out to each word using convolutional neural networks to extract；

(3) character feature vector is combined with term vector, is passed to LSTM neutral nets, instructed using two-way LSTM neutral nets Get read statement information characteristics；

(4) for semantic feature resulting in (3), entity mark is carried out to each word using condition random field, is marked The entity information gone out in statement sequence.

2. a kind of text based on Bi-LSTM, CNN and CRF according to claim 1 names entity recognition method, and it is special It is to carry out data prediction to text data to levy, and to text subordinate sentence, participle are carried out, and obtains the step of character feature vector such as Under：

(1) subordinate sentence and participle are carried out to document using natural language processing instrument so that document is with sentence and per token Present；

(2) sentence, word and label are counted, forms sentence table, vocabulary and label list；

(3) character in word list is counted, forms character list；

(4) character feature vector is formed using the good term vector of existing pre-training and character list.

3. a kind of text based on Bi-LSTM, CNN and CRF according to claim 1 names entity recognition method, and it is special Levy is that the step of described utilization convolutional neural networks carry out character feature vector extraction to each word is as follows：

(1) C is made to be character list, d is the dimension of each character vector, and character vector matrix is：Q∈R^d×|C|；

(2) word k ∈ V are preset by continuous character [c₁,c₂,...,c_l] composition, wherein l is the length of word k, then the character of k Vector matrix is by C^k∈R^d×lBe given, wherein i-th is classified as character c_iVector；

(3) in C^kWith the kernel H ∈ R that a width is w^d×wBetween realize convolutional layer, after this give convolutional layer addition bias Bias, and whole convolution results are carried out nonlinear regression to realize Feature Mapping map f^k∈R^l-w+1, wherein, mapping function f^k I-th element f^k[i] is given by (1) formula；

f^k[i]=tanh (<C^k[*,i:i+w-1],H>+b) (1)

Wherein * be all line numbers, C^k[*,i:I+w-1] it is C^kIn i-th row to the i-th+w-1 row,<,>For Frobenius inner products, b is Bias vector；

(4) it is last, withExpress as the character pair of kernel H.

4. a kind of text based on Bi-LSTM, CNN and CRF according to claim 1 names entity recognition method, and it is special It is to combine character feature vector with term vector to levy, and is passed to LSTM neutral nets, is instructed using two-way LSTM neutral nets The step of getting read statement information characteristics is as follows：

(1) x is defined_tFor the input character feature vector of t, h_tIt is the hiding layer state that all useful informations are stored in t Vector, σ is that sigmoid returns layer, and * is inner product, U_i,U_f,U_c,U_oTo be directed to input x under different conditions_tWeight matrix, W_i, W_f,W_c,W_oTo hide layer state h_tWeight matrix, b_i,b_f,b_c,b_oFor bias vector；

(2) t forget gate calculating as shown in (2) formula：

f_t=σ (W_fh_t-1+U_fx_t+b_f) (2)

(3) h is updated in t_t-1All information of middle storage, computing formula is as shown in (3), (4) formula：

i_t=σ (W_ih_t-1+U_ix_t+b_i) (3)

${\tilde{C}}_t=\tanh (W_c{h}_{t-1}+U_c{x}_t+b_c)---\left(4\right)$

WhereinFor the vector that t be introduced into cell state；

(4) information updating for storing the t-1 moment in t is the storage information of t, and computing formula is formula (5)：

$C_t=f_t*C_{t-1}+i_t*{\tilde{C}}_t---\left(5\right)$

WhereinFor the vector of t cell state；

(5) it is shown in the output such as formula (6) of t, and update h_t, computing formula such as formula (7)：

o_t=σ (W_oh_t-1+U_ox_t+b_o) (6)

h_t=o_t*tanh(C_t) (7)

Wherein o_tFor the output of t；h_tFor the vector of t hidden layer；

(6) h in above-mentioned steps_tAll information of storage last time, with same method a g is arranged again_tFor storage not The information come, most latter two hidden layer information forms last output vector by cascading.

5. a kind of text based on Bi-LSTM, CNN and CRF according to claim 1 names entity recognition method, and it is special It is to carry out entity mark to each word using condition random field to levy, the step of mark the entity information in statement sequence It is as follows：

(1) with z={ z₁,z₂,...,z_nRepresent list entries, wherein n for list entries length, z_iFor the input of i-th word Vector, y={ y₁,y₂,...,y_nFor z reality output sequence label, Y (z)={ y'₁,y'₂,...,y'_nIt is the possible of z Output label sequence；

(2) condition of the condition random field probabilistic model definition for given list entries z on be possible to sequence label y' is general Rate p (y | z；W,b)：

$p\left(y|z;W,b\right)=\frac{\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}{\mathrm{\&psi;}}_i\left(y_{i-1},y_i,z\right)}{\underset{y^{\&prime;}\&Element;Y\left(z\right)}{\mathrm{\&Sigma;}}\underset{i=1}{\overset{n}{\mathrm{\&Pi;}}}{\mathrm{\&psi;}}_i\left({y^{\&prime;}}_{i-1},{y^{\&prime;}}_i,z\right)}---\left(8\right)$

Wherein,For potential gain function, W is weight vectors, and b is bias vector； And b_y',yRespectively with label to (y', y) corresponding weight vectors and bias vector；

(3) condition random field (CRF) training stage, use maximal condition Likelihood estimation select cause likelihood ratio L (W, b) most Big parameter, for training set { (z_i,y_i), the logarithm of likelihood ratio is：

$L(W,b)=\underset{i}{\&Sigma;}\log p\left(y\right|z;W,b)---\left(9\right)$

(4) find the y of highest conditional probability in sequence label to carry out label for labelling to sequence：

$y=\underset{y\&Element;Y\left(z\right)}{\mathrm{argmax}}p\left(y\right|z;W,b)---\left(10\right).$