CN111125316B

CN111125316B - Knowledge base question-answering method integrating multiple loss functions and attention mechanism

Info

Publication number: CN111125316B
Application number: CN201911369897.4A
Authority: CN
Inventors: 杨新武; 张煜
Original assignee: Beijing University of Technology
Current assignee: Beijing University of Technology
Priority date: 2019-12-26
Filing date: 2019-12-26
Publication date: 2022-04-22
Anticipated expiration: 2039-12-26
Also published as: CN111125316A

Abstract

The invention discloses a knowledge base question-answering method integrating multiple loss functions and an attention mechanism, which takes a question and a candidate answer as input, adopts Bi-LSTM and Bi-GRU as main feature extractors, integrates the attention mechanism at the same time, optimizes a model by using two loss functions, updates model parameters by adopting calculation loss value back propagation, and trains a network model until the model converges. And finally mapping the question and the candidate answers to a feature space with the same dimension through network training, calculating the semantic similarity of the question and the candidate answers by using the inner product of feature vectors of the question and the candidate answers, and simultaneously expanding the difference between different answers by using the cosine similarity between the candidate answers. The method has the advantages that the SimpleQuestions data set is tested, the model has strong characteristic mapping capability and high accuracy rate, and accordingly the superiority of the method is proved.

Description

Knowledge base question-answering method integrating multiple loss functions and attention mechanism

Technical Field

The invention relates to the field of deep learning, natural language processing and knowledge base question answering, in particular to a knowledge base question answering method integrating a multi-loss function and an attention mechanism.

Background

The knowledge base question-answering system can answer questions asked by users quickly and accurately, and compared with a search engine, the retrieval is more efficient, so that the knowledge base question-answering system becomes a new hotspot of research in the field of natural processing recently. The knowledge base question-answering process is divided into a plurality of modules, wherein the text matching of the answers is a key step of the knowledge base question-answering.

In the prior knowledge base question-answering method, a semantic parser is constructed by utilizing professional knowledge of semantics, a natural language question is converted into a logic form, and the logic form is converted into a corresponding database query expression to obtain a final answer. Although the method has a good effect, a large number of manual rules need to be set for the data set, and meanwhile, a semantic analyzer needs to be built by a semantic expert, so that the generalization capability is poor.

In recent years, deep learning methods have achieved very good results in the fields of text classification, emotion classification, named entity recognition and the like of natural language processing. The good result can be obtained mainly because the feature extractor adopted by the deep learning can learn a strong feature mapping capability, and this provides possibility for designing complex high-precision text matching models.

Two main methods exist at present for the matching process of answers in the knowledge base question answering. Firstly, a semantic analysis based method is used, the method is used for inquiring answers by mapping phrases in a question with a pre-constructed vocabulary and constructing a syntax tree. However, this method requires manual rule definition and is difficult to generalize. Another is a method based on deep learning, which selects answers by mapping questions and answers to corresponding vectors such that the questions are closer to the correct answers and the wrong answers are further apart. The method provided by the method simultaneously measures the distance between the answers, so that the distance between similar answers is longer, the difference between answer representations is enlarged, and the model performance is improved; in addition, an Attention mechanism is also merged into the method, the vector representation of the question is dynamically updated according to different aspects of answer vector representation, and the model performance is further improved. By performing experiments on SimpleQuestions data sets, the end-to-end accuracy rate of the method is over 77.2%, and compared with other methods, the method has strong competitiveness.

Disclosure of Invention

The problem that the precision of the existing knowledge base question-answering technology is not high, a large number of manual rules are needed, and the generalization performance is poor is solved.

The technical scheme adopted by the invention is a text matching method of a neural network and a fusion multi-loss function, the method maps the question description and the candidate answer of a user into a characteristic vector represented by a one-dimensional array on a specific space S by using the network, and the answer with the maximum score is selected as the candidate answer by calculating the matrix inner product score of the two vectors.

The method comprises three processes of gait energy image extraction, model training and identification, and specifically comprises the following steps:

a text matching method fusing a plurality of loss functions is characterized in that: the method comprises three processes of data preparation, model training and text matching, and specifically comprises the following steps;

step S1, data preparation process:

step S1.1, entity identification:

and identifying the entity in the problem to obtain an entity sequence.

Step S1.2, obtaining candidate answers:

and querying a knowledge base according to the result of the entity identification to obtain a candidate relation corresponding to the entity. Taking the corresponding relation of the correct answer as a positive example; the other relationships are negative examples.

Step S1.3, training set preparation:

for each problem sample in the data set, the above processing is carried out, and for the negative examples which are less than 50, other relations are randomly up-sampled from the relation pool to form the negative examples, so that the proportion of the positive samples to the negative samples is 1: 50.

Step S2, model training process:

and step S2.1, inputting the user question and the correct relation and the candidate relation obtained in the step S1 into the neural network proposed by the method, and respectively obtaining the feature representation q of the question, the feature representation r of the correct relation and the feature representation r' of the error relation through network mapping.

And step S2.2, forming a triad pair of (q, r, r') by the feature vectors obtained by the processing of the step S2.1, and calculating an inner product score1 of the question and the correct answer and an inner product score2 of the question and the wrong answer.

S2.3, calculating the loss value of the triad pair obtained in the step S2.2 according to a loss calculation formula provided by the method, and optimizing the neural network model by taking a loss function value as a target;

the formula is as follows

loss1＝max(0,-y*(S(r；q)-S(r’；q))+margin)

loss2＝y*|cos(r,r’)|*max(0,cos(r,r’)-γ)

Wherein y in loss1 indicates whether the candidate answer is correct, and takes a value of +1 or-1, because r is specified as correct, y is always 1, S is a function for calculating the product of the two, and margin is a set hyper-parameter and defaults to 0. Loss2 is a weighted cosine similarity Loss function, where cos represents the cosine distance between the two, y represents whether the two involved in the calculation are of the same class, where y is constant equal to 1 because r is not the same as r', and γ is a hyperparameter with a default of 0.

Step S2.4, repeating the step S2.1 to the step S2.3 until the neural network model converges;

step S3, an identification process;

s3.1, obtaining a candidate relation set of the user questions through the user questions;

and S3.2, taking the candidate relation set and the problem obtained in the step S3.1 as the input of the neural network, and obtaining the score of each candidate relation through network calculation.

And S3.3, sequencing the scores of the candidate relations, and selecting the answer with the maximum score as the final correct relation. The input text of the network is guaranteed to be noiseless and aligned.

Each training sample is subjected to network mapping to obtain a feature vector represented by a one-dimensional array, and the dimension of the feature is limited to 128-dimension to 512-dimension.

Text matching score calculation formula: s ═ Σ_if(q_i,r_i). Wherein q is_iFor feature coding of the problem, r_iFeatures for different aspects of the answer are encoded. The attention mechanism is utilized to influence the question representation according to different aspects of the answer, and the specific formula is as follows:

q_j＝∑_ia_ij×h_i (3)

wherein h is_jIndicating the hidden state of the jth word in the question, r_iShowing the hidden state of the ith part of the relationship, w_ijThe similarity of the current word to the current relational representation is calculated. Calculating the similarity between each word in the question and the relation and then normalizing to obtain a_ijI.e., to indicate which word in the question is more attentive to the current candidate answer. And finally, carrying out weighted summation on the obtained weights and the original question representation to obtain the final representation of the question for the current answer. In addition, a function f for measuring the similarity of the two_attThe calculation method is very diverse Hu. The traditional method is to calculate the inner product between vectors, and can also introduce weight matrix calculation or use a fully connected neural network after splicing the vectors, and the like.

Triple loss calculation formula:

loss1＝max(0,-y*(S(r_pos；q)-S(r_neg；q))+margin) (4)

loss2＝y*|cos(r_pos,r_neg)|*max(0,cos(r_pos,r_neg)-γ) (5)

the formula of the text matching score performs the final selection by calculating the degree of matching of the question and the answer. In this process, a loss function such as equation (4) is used, and the goal is to increase the degree of matching between the question and the correct answer and decrease the degree of matching between the question and the wrong answer. This loss function is fixed for a given hyper-parameter margin and takes a value of 0-1. For a similar answer, the score should be similar, and the interval between the correct answer and the incorrect answer is close to the value of margin, so that the direction distance of the similar answer is too close by the model, which is not favorable for the model to distinguish the correct answer. Therefore, the method adds the loss function of formula (5) as a target on the basis of the loss function, so that the longer the direction distance between different answer representations is, the better.

The method of the invention performs end-to-end experiments on SimpleQuestions data sets, and the experimental result is superior to other recent methods for performing experiments on the data sets.

Drawings

Fig. 1 is an overall flow chart according to the present invention.

Fig. 2 is a model structure diagram according to the present invention.

Fig. 3 is a flowchart of step S2 according to the present invention.

Fig. 4 is a flowchart of step S3 according to the present invention.

FIG. 5 is a flow chart of the question answering method of the present invention.

Detailed Description

For the purpose of promoting a better understanding of the objects, features and advantages of the invention, reference is made to the following detailed description taken in conjunction with the accompanying drawings.

Step S1, the entity in the question is obtained.

Step S1.1, firstly, processing a training data set, marking data in a question by using a knowledge base, and converting an entity recognition problem into a sequence marking task for processing.

And S1.2, training the processed data set by using the existing sequence labeling network model, such as Bi-LSTM + CRF.

And S1.3, marking the trained model on the test set, and marking out the entity of the test set.

And S1.4, regarding some entities, the entity marked in the question is not completely the same as the entity in the knowledge base, so that the used entities are processed by using name lists, and the entity in the question is linked with the entity in the knowledge base, so that the sequence number of the entity in the question in the knowledge base is obtained.

Step S2, training a text matching model fused with the multi-loss function:

and S2.1, obtaining entity recognition results of the training set and the test set in the step 1, and inquiring a knowledge base by using the entity recognition results to obtain all answers corresponding to the entity in the knowledge base, wherein the answers are called as candidate answers. If the candidate answers corresponding to an entity in the training set are less, the wrong candidate answers can be added in an up-sampling mode for training.

And S2.2, acquiring a certain number of triad pairs (q, r, r') from the training set in each training, namely the question, the correct answer corresponding to the question and the wrong answer corresponding to the question. The question q is characterized by Bi-LSTM and then attention mechanism is carried out with the result of answer representation, and three different aspects of question representation, namely q1, q2 and q3, are obtained. Similarly, the answer is divided and represented by r1, r2 and r3, and the matching score S, S ═ Σ of the answer and the question is finally calculated_if(q_i,r_i)。

And step S2.3, obtaining scores of the questions and different answers after the forward propagation is finished. And calculating loss values loss1 and loss2 of the training by using the obtained scores, and reversely propagating and optimizing model parameters by using an Adam optimizer with the loss1 and the loss2 as targets.

And step S2.4, repeatedly executing the step S2.2 to the step S2.4 until the model converges.

Step S3: and (3) prediction process:

and S3.1, utilizing the pre-trained entity recognition model to perform entity recognition on the question of the user.

S3.2, inquiring the entity identification result into a knowledge base to obtain the correspondence of the problem entity in the knowledge base; entity-related triples are then obtained as candidate answers.

And step S3.3, calculating the candidate answers and the question matching scores acquired in the step S3.2 in sequence.

Step S3.4, the most correct answer with the highest score is selected and pushed to the user.

The method adopts SimpleQuestions data set. This data set is all a first order problem, with 75910 samples in the training set, 10845 in the validation set, and 21687 in the test set. The hyper-parameters used during training when the model recognition efficiency is highest are shown in the following table X:

parameter(s)	Means of	Numerical value
			max_nrof_epochs	Algebra of training	32
epoch_size	Number of times of training in each generation	64
			neg_ans	Number of candidate answers to question upsampling	50
Optimizer	Selected optimizer	Adam
			learning_rate	Learning rate	0.001

The end-to-end experimental results are as follows, wherein the base model is the experimental result without the multiple loss function and attention mechanism proposed by the method:

model (model)	Results of the experiment
		basemodel	76.31
multi-loss+attentionmodel	77.23

The method constructs a text matching model fusing multiple loss functions, and further improves the model accuracy by fusing an attention mechanism and the loss functions of multiple optimization targets. The method can be applied to the fields of search engines, intelligent customer service and the like.

The above description is only an embodiment of the present invention, but the scope of the present invention is not limited thereto, and any person skilled in the art can understand that the modifications or substitutions are included in the scope of the present invention, and therefore, the scope of the present invention should be determined by the protection scope of the claims.

Claims

1. A knowledge base question-answering method fusing a multi-loss function and an attention mechanism is characterized by comprising the following steps: the method comprises three processes of data preparation, neural network model training and text matching, and specifically comprises the following steps;

in step S1, the data preparation process is as follows:

step S1.1, entity identification:

identifying an entity in the problem to obtain an entity sequence;

step S1.2, obtaining candidate answers:

inquiring a knowledge base according to the result of entity identification to obtain a candidate relation corresponding to the entity; taking the corresponding relation of the correct answer as a positive example; other relationships are negative examples;

step S1.3, training set preparation:

performing the above processing on each problem sample in the data set, and randomly upsampling other relations from the relation pool to form a negative example for the sample with less negative examples;

step S2, neural network model training process:

s2.1, processing the user question in step S1 to obtain an entity sequence of the question and a corresponding candidate relation sequence, inputting the sequences into Word Embedding to obtain vectors of each sequence, and then extracting sentence characteristics from two directions of positive sequence and negative sequence by using the vectors through Bi-LSTM; obtaining a hidden vector of each word in the question in a positive sequence network; obtaining a hidden vector of each word in a reverse order network, and then splicing the hidden vectors of each word obtained in the forward order network and the reverse order network; respectively obtaining a problem feature vector representation q, a correct feature vector representation r and an error feature vector representation r';

step S2.2, forming a ternary group pair of (q, r, r') by the feature vectors obtained by the processing in the step S2.1, inputting the ternary group pair into an Attention Model for calculating and matching Score, and obtaining an inner product Score S1 expressed by a calculated question and a correct answer and an inner product Score S2 expressed by a question and a wrong answer;

s2.3, according to the provided loss calculation formula, two loss functions are adopted for calculation, and the difference of expression between the relations is enlarged on the basis of enabling the problem and correct relation score to be higher and the error relation score to be lower;

calculating the loss value of the triad pair obtained in the step S2.2, and optimizing the neural network model by taking the loss function value as a target; the formula is as follows:

loss＝max(0,-y*(S(r；q)-S(r’；q))+margin)

loss2＝y*|cos(r,r’)|*max(0,cos(r,r’)-γ)

step S3, an identification process;

s3.1, obtaining a candidate relation set of the user questions through the user questions; the user problem is subjected to entity identification to obtain an identification result, the identification result is sent to a constructed knowledge base to be inquired, all relations corresponding to candidate entities are obtained, and the relations are used as a set of candidate relations; in the candidate relation set, the relation matched with the user problem is used as a positive example, and other relations are used as negative examples;

s3.2, taking the candidate relation set and the problem obtained in the step S3.1 as the input of a neural network, and obtaining the score of each candidate relation through network calculation;

s3.3, sorting the scores of the candidate relations, and selecting the relation with the largest score as the final correct relation; ensuring that the input text of the network is noiseless and aligned;

each training sample is subjected to network mapping to obtain a feature vector represented by a one-dimensional array, and the dimension of the feature is limited between 128 dimensions and 512 dimensions;

text matching score calculation formula: s ═ Σ_if(q_i,r_i) (ii) a Wherein q is_iFor feature coding of the problem, r_iCoding features for different aspects of the answer; the attention mechanism is utilized to influence the question representation according to different aspects of the answer, and the specific formula is as follows:

w_ij＝f_att(h_j,r_i) (1)

q_j＝∑_ia_ij×h_i (3)

wherein h is_jIndicating the hidden state of the jth word in the question, r_iShowing the hidden state of the ith part of the relationship, w_ijCalculating the similarity of the current word and the current relation representation; calculating the similarity between each word in the question and the relation and then normalizing to obtain a_ijI.e. indicating which word in the question is more attentive for the current candidate answer; and finally, carrying out weighted summation on the obtained weights and the original question representation to obtain the final representation of the question for the current answer.

2. The method of claim 1, wherein the method comprises the steps of: the output dimension of each training sample at the network mapping layer is the same, so that the inner product calculation of the vector can be carried out.

3. The method of claim 1, wherein the method comprises the steps of: the data set is processed to sample negative examples based on the knowledge base used to augment the training samples.