KR102352251B1 - Method of High-Performance Machine Reading Comprehension through Feature Selection - Google Patents

Abstract

The present invention relates to machine reading comprehension through quality selection, and to a question-and-answer model in which a machine understands a given context and answers related questions. The present invention is a machine reading comprehension model GF-Net (Gated Feature Networks) that improves sentence comprehension through pre-trained context information for a high-performance machine reading model and applies a feature gate method that selects useful qualities by the model itself. suggest

Description

Method of High-Performance Machine Reading Comprehension through Feature Selection

The present invention relates to a high-performance machine reading method through quality screening.

Machine Reading Comprehension is a question-and-answer model in which a machine understands a given context and answers related questions. As an example of a conventional question-and-answer model, Korean Patent Publication No. 10-2002-0030545 (4.25 2002.04.25), an automatic answer and search provision method for a natural language sentence-type question based on artificial intelligence and natural language processing technology has been proposed. Since a knowledge database is used, there is a problem that a lot of resources are required to build the database.

Machine reading comprehension must be studied essential for self-development AI that acquires and utilizes information through documents. As attention mechanism and end-to-end model became possible through a deep neural network, it is being actively studied, and three stages of encoding, mutual concentration, and response extraction have been established. In the encoding stage, besides word embedding, it was studied to increase the understanding of sentences by using qualities such as syllable embedding, part-of-speech, and entity name. Recently, language models are pre-trained through many corpora, such as CoVe and ELMo, and the results are used together to significantly improve overall model performance. Mutual attention is the attention-focusing method to find out which word in the query is related to the word appearing in the context suggested by R-Net, self-attention to review the context again, the query proposed by Bi-directional Attention Flow, and inter-context Two-way reciprocal focus has been studied and many studies are using the method. The response extraction step was studied by finding the starting position and ending position of the word corresponding to the correct answer, such as a pointer network and stochastic prediction dropout.

The present invention proposes a method for improving performance by applying a feature gate that selects useful features by the model itself in the encoding step and the response extraction step among the three steps of machine reading.

The present invention is a machine reading comprehension model GF-Net (Gated Feature Networks) that improves sentence comprehension through pre-trained context information for a high-performance machine reading model and applies a feature gate method that selects useful qualities by the model itself. is characterized by

(a) an encoding step of generating a context encoding vector and a query encoding vector through feature selection and pre-learning information, (b) a bidirectional mutual concentration step of generating a mutually focused result vector between the context and the query of the encoding step, and (c) and a response extraction step in which the feature selection is reflected by selecting the feature through the mutual concentration result vector and transmitting it to the pointer network.

Preferably, in the encoding step, the abstract information for each layer is mixed in the pre-trained multi-layer recurrent neural network language model by applying the normalization vector and the scalar parameter to the vector connecting the embedding vector and the hidden vector of the word used in the language model.

In addition, the encoding step inputs the word embedding, syllable embedding, and feature vector weighted for each feature according to the contextual situation of the word appearing in the sentence into the recurrent neural network, and connects the pre-learned ELMo vector to the output result to query the context encoding vector Create an encoding vector.

In addition, the bidirectional reciprocal concentration step between the context and the query is characterized in that the relation between the context and the query is found in the context encoding vector and the query encoding vector of the encoding step.

In addition, the selected features in the response extraction step in which the feature selection is reflected are connected to the existing mutually focused vector and transmitted to the pointer network.

The present invention has the effect of improving sentence comprehension through pre-trained context information for a high-performance machine reading model and selecting useful qualities by the model itself.

1 is a structural diagram of a model according to an embodiment of the present invention.
2 is a view for explaining a feature weight change using feature gates according to an embodiment of the present invention.
3 is an exemplary diagram for explaining an encoding layer, an interaction layer, and a pointing layer of a high-performance machine reading method through feature selection according to an embodiment of the present invention.
4 is an exemplary diagram illustrating a process in which the linguistic quality of a word of a high-performance machine reading method through feature selection is changed in the encoding layer according to an embodiment of the present invention.

In the high-performance machine reading method through feature selection according to an embodiment of the present invention, the feature gate improves sentence comprehension through pre-learned context information for a high-performance machine reading comprehension model and selects useful qualities by the model itself. ) method applied to the machine reading comprehension model GF-Net (Gated Feature Networks) is proposed. An embodiment of the present invention proposes a method for improving performance by applying a feature gate that selects useful features by the model itself in the encoding step and the response extraction step among the three steps of machine reading.

Machine Reading Comprehension (MRC) is an artificial intelligence technology in which a computer understands a document and automatically finds the answer corresponding to the user's question from a given document. Existing machine reading models combine the word embedding feature that maps words into a semantic space and linguistic features such as part-of-speech, entity name, and syntactic relationship. Although the method of inferring the correct answer by calculating the association was mainly used, the existing machine reading models used a method of simply combining word embedding and linguistic qualities. It was unavoidable due to a feature interference problem. In other words, although the linguistic qualities have different roles in the process of inferring the correct answer, they were simply combined and used, resulting in not contributing significantly to the improvement of machine reading performance. For example, part-of-speech can provide an important clue to limiting correct candidates to numbers (words with parts-of-speech corresponding to rhetoric) in questions related to date, time, exchange rate, etc. It can provide important clues in limiting the correct answer candidates to the corresponding semantic category. The syntactic relationship can provide an important clue in selecting a correct answer candidate based on information such as whether the subject of a question is a subject or an object. Therefore, it is necessary to selectively use the linguistic qualities that play different roles in the answer reasoning process so that they work differently depending on the role, rather than simply combining them.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 shows a structural diagram of the proposed model. The proposed model consists of three steps: encoding, mutual concentration, and response extraction. In the encoding step, word embeddings, syllable embeddings for out-of-vocabulary processing, pre-trained ELMo vectors, and feature vectors through feature gates proposed in the present invention are input. Mutual focus uses a method in which self-focus is used together with two-way mutual focus between queries and contexts. The final response extraction returns the starting and ending positions of the phrases to be used as responses in the context using a pointer network to which a screening function is added.

1, a system for implementing a high-performance machine reading method through feature selection according to an embodiment of the present invention includes an encoding layer, an interaction layer, and a pointing layer. is composed of layers). The encoding layer generates word embedding vectors and linguistic feature vectors in documents and queries. And through the gate mechanism that combines the linguistic feature vector with the word embedding vector, the linguistic features are selectively induced to participate in the machine reading process. In the interaction layer, the relationship between each word in the document and each word in the question is calculated through the bi-directional co-attention mechanism and the self-attention mechanism. In the pointing layer, after recombining the linguistic qualities with the vectors generated in the interaction step through the gate mechanism, the correct position in the document is determined using a pointer network. Here, the pointer network is an extended deep learning model of RNN that outputs a list of positions corresponding to an input sequence using an attention mechanism based on a recurrent neural network (RNN). Attention mechanism is a technique to increase the performance of the model by making the deep learning model pay attention to a specific vector. When the front part that makes a vector with input is the encoder and the rear part that outputs the target language (W, X, Y, Z) with the vector output by the encoder as input is called the decoder, the length of the source language and the target language will become longer. As the number increases, the performance of the model deteriorates. When predicting W, if you try to focus on all A, B, and C, the accuracy may be lowered, but making the model pay attention to only the important part is the attention mechanism. For example, assuming that we create an S2S model that translates German "Ich mochte ein bier" into English "I'd like a beer", we want to make the model pay attention to 'bier' when predicting the fourth word 'beer'. The assumption of the attention mechanism is that the result that the encoder receives 'bier' and turns it into a vector (encoder output) will be similar to the vector (decoder input) that the decoder uses when predicting 'beer'.

The core of the high-performance machine reading method through feature selection according to an embodiment of the present invention is a gate mechanism that selectively combines word information and linguistic feature information and the overall structure of an artificial neural network optimized for machine reading. By improving machine reading performance, it is possible to increase user satisfaction of various question and answer services (intelligent search, artificial intelligence assistant service, etc.)

A high-performance machine reading method through feature selection according to an embodiment of the present invention includes an encoding step through feature selection and pre-learning information, a bidirectional mutual concentration step between context and query, and a response extraction step reflecting feature selection.

First, for a high-performance machine reading comprehension model, the character selection process, which improves sentence comprehension through pre-trained context information, and selects useful qualities by the model itself, and the encoding step through pre-learning information will be described as follows.

In the present invention, the encoding step through feature selection and pre-learning information uses word embeddings and syllable embeddings, pre-trained ELMo vectors, and feature vectors. Syllable embedding is used to solve the problem of non-registered words where a specific word does not appear in the word embedding matrix. Syllable embedding expresses the syllables of a word as a vector through a convolutional neural network. The ELMo vector is generated as in Equation (1) after pre-training the Bidirectional Language Model.

[Equation 1]

는 양방향 LSTM의 입력(

)과 출력(

)을 연결한 벡터이다. LM은 언어모델(Language model)이고, L은 양방향 LSTM 계층 수를 의미한다. LSTM은 장단기 기억 네트워크(Long Short Term Memory networks)로 보통 LSTM으로 불리며, 장기 의존성을 학습할 수 있는 특별한 종류의 순환 신경망으로 Hochreiter와 Schmidhuber(1997)에 의해 소개된 것으로 이에 대한 자세한 설명은 생략하기로 한다.

is the input of the bidirectional LSTM (

) and output (

) is a concatenated vector. LM is a language model, and L is the number of bidirectional LSTM layers. LSTM is a long-short-term memory network, commonly called LSTM, and is a special kind of recurrent neural network that can learn long-term dependencies. It was introduced by Hochreiter and Schmidhuber (1997). do.

은 언어 모델에 사용되는 k번째 단어의 임베딩 벡터이고,

는 여러 층으로 이루어진 양방향 순환 신경망의 j번째 층의 은닉벡터이며,

는

와

를 연결(concatenation)한 벡터를 의미한다.

는 여러 층의 순환 신경망 벡터를 층 단위로 정규화하기 위해 사용하는 벡터로 각 층의 가중치는

로 구성된다.

는 효과적인 학습을 위해 사용한 스칼라 파라미터이다.

은

에 정규화 벡터(

), 스칼라 파라미터(

)을 적용하여 사전 학습된 다층 순환 신경망 언어모델에서 계층별 추상화 정보를 혼합한 결과를 의미한다.in formula (1)

is the embedding vector of the kth word used in the language model,

is the hidden vector of the j-th layer of a multi-layer bidirectional recurrent neural network,

Is

Wow

It means a vector concatenated with .

is a vector used to normalize multi-layer recurrent neural network vectors layer by layer, and the weight of each layer is

is composed of

is a scalar parameter used for effective learning.

silver

to the normalization vector(

), a scalar parameter (

) means the result of mixing abstract information for each layer in the pre-trained multi-layer recurrent neural network language model.

That is, Equation (1) means that the abstract information for each layer is properly mixed in the pre-trained multi-layered recurrent neural network language model. Finally, the feature vector to which the feature selection proposed in the present invention is applied is used. Feature selection uses the gate theory used in recurrent neural networks to have different weights for each feature. Figure 2 shows the change of feature vectors using feature gates.

The example of FIG. 2 is an example using the entity name feature and the syntax feature. In the example "Newton" and "DeMarcus Ware", object name information is more important than syntax information. On the other hand, since "was sacked" is not an entity name, it is necessary to lower the entity name weight. That is, as shown in FIG. 2, the feature gate for giving different ratios for each feature follows Equation (2).

[Equation 2]

는 문장에 나타나는 단어의 단어 임베딩, 음절 임베딩, ELMo벡터를 순환 신경망에 입력하여 나온 결과의 i번째 벡터를 나타낸다.

는 양방향 순환 신경망을 의미하고,

는 단어 임베딩,

는 음절 임베딩,

는 ELMo벡터를 나타낸다.in formula (2)

represents the i-th vector of the result obtained by inputting word embeddings, syllable embeddings, and ELMo vectors of words appearing in the sentence into the recurrent neural network.

means a bidirectional recurrent neural network,

is the word embedding,

is a syllable embedding,

denotes an ELMo vector.

는 i번째 단어에서 나타나는 j번째 자질의 게이트의 출력(자질의 가중치)로

의 활성화 함수(

)를 사용하여 0~1 사이의 값을 가진다.

는 j번째 자질 게이트에서 사용하는 가중치이고,

는 문맥이 반영된

와 자질 벡터

를 연결(concatenation)한 벡터를 의미한다. 게이트를 계산한 후

를 통해 각 자질 벡터에 게이트를 곱해 문맥 상황에 맞게 자질의 비중을 조절하게 된다.

는 게이트를 통해 얻은 자질의 가중치와 해당 자질 벡터(

)를 곱한 값이다.

is the output (feature weight) of the j-th feature gate appearing in the i-th word.

of the activation function (

) to have a value between 0 and 1.

is the weight used in the j-th feature gate,

is the context

With Qualities Vector

It means a vector concatenated with . After counting the gate

Each feature vector is multiplied by a gate to adjust the feature weight according to the context.

is the weight of the feature obtained through the gate and the corresponding feature vector (

) multiplied by

, 음절 임베딩

, ELMo벡터

, 자질 벡터

를 순환 신경망에 입력하여 출력된 결과에 ELMo벡터를 연결한

를 인코딩 벡터로 사용한다.

는 순환 신경망을 이용한 인코딩 벡터이다. 위 작업을 문맥과 질의에 각각 적용하여 문맥 인코딩 벡터

와 질의 인코딩 벡터

를 생성한다.Finally word embedding

, syllable embeddings

, ELMo vector

, the quality vector

is input to the recurrent neural network and the ELMo vector is connected to the output result.

is used as the encoding vector.

is an encoding vector using a recurrent neural network. By applying the above operation to the context and query respectively, the context encoding vector

and query encoding vector

to create

Next, the two-way mutual concentration between context and query will be described as follows.

The present invention uses self-focus together with two-way mutual focus to find the relationship between context and query. The two-way mutual concentration is calculated as Equation (3).

[Equation 3]

는

의

간 주의집중 점수 (attention score)이고,

는

를 구하는 함수이다.

는 문맥 i번째 단어의 인코딩 벡터,

는 질의 j번째 단어의 인코딩 벡터,

는 attention score 계산 시 사용하는 가중치이고,

는

와

의 요소별 곱셈(elementwise multiplication)이다.

는 문맥에서 질의 방향 주의집중 가중치이다.

는 attention score(

)를 이용해 문맥에서 질의 방향 주의 집중 가중치를 계산하는 softmax 함수이다.

는 질의 벡터가 문맥에 얼마만큼 중요한지 계산한 주의집중 벡터이고,

는 문맥 벡터가 질의에 얼마만큼 중요한지 계산한 주의집중 벡터이다. 식 (3)에서는 문맥 벡터와 질의 벡터간의 상호 집중 가중치를 구하고 문맥 벡터와 결합하여 생성된

는 질의의 단어 벡터가 문맥의 어떤 단어에 중요하게 작용하는지를 찾아낸다.

에서

는 함수를 의미하고,

는 문맥 단어들의 인코딩 벡터를 의미한다. 즉,

는 문맥 벡터와 질의 벡터간의 상호 집중 가중치를 구하고 문맥 벡터와 결합하여 생성된 결과이다. 자기 집중은 식 (4)와 같이 계산된다.in equation (3)

Is

of

is the attention score of the liver,

Is

is a function to find

is the encoding vector of the context i-th word,

is the encoding vector of the j-th word of the query,

is the weight used to calculate the attention score,

Is

Wow

is elementwise multiplication of

is the query direction attention weight in context.

is the attention score(

) is a softmax function that calculates the query direction attention weight in context.

is the attention vector calculating how important the query vector is to the context,

is the attention vector that calculates how important the context vector is to the query. In Equation (3), the mutually concentrated weight between the context vector and the query vector is obtained, and the generated weight is combined with the context vector.

finds which words in the context are important for the word vector of the query.

at

means a function,

denotes an encoding vector of context words. in other words,

is the result generated by finding the mutually concentrated weight between the context vector and the query vector and combining it with the context vector. The magnetic concentration is calculated as in equation (4).

[Equation 4]

는 문맥 질의 간 상호 주의 집중 벡터(F)와 자가 주의 집중 벡터(c) 입력을 입력 받는 양방향 순환 신경망 출력 벡터이다.

는 상호 주의집중 계층의 출력이고,

는 자기 주의집중 계층의 출력이다.

는 t번째 단어와 j번째 단어 간 자가 주의 집중 가중치 점수이다.

는 상호 집중이 반영된 벡터,

는 t번째 단어가 i번째 단어와의 유사도를 나타내는 주의집중 가중치이다. 즉, 현재 단어가 다른 단어들과의 유사도 계산을 통해 관련된 중요 단어를 파악하고, 주의집중 풀링(attention pooling)을 통해 문장 전체의 단어를 반영해주는

를 생성한다. 생성된 자기 주의집중 벡터가 도움이 되지 않을 수도 있기 때문에 Residual layer를 적용하여 상황에 맞게 적용되도록 한다. Residual layer는

로 계산된다. 여기서,

을 수행하는 계층이 Residual layer, Residual layer 출력이

이다.in equation (4)

is the bidirectional recurrent neural network output vector that receives the mutual attention vector (F) and self attention vector (c) inputs between context queries.

is the output of the reciprocal attention layer,

is the output of the self-attention layer.

is the self-attention weighted score between the t-th and j-th words.

is the vector reflecting the mutual concentration,

is the attention weight indicating the similarity of the t-th word to the i-th word. In other words, it identifies important words related to the current word through similarity calculation with other words, and reflects the words of the entire sentence through attention pooling.

to create Since the generated self-attention vector may not be helpful, apply a residual layer to suit the situation. The residual layer is

is calculated as here,

The layer that performs

to be.

Response extraction reflecting quality selection is explained as follows.

In the present invention, a pointer network to which a feature selection function is added is used for response extraction. Qualification selection in response extraction is expressed as Equation (5).

[Equation 5]

는 Residual layer의 출력

를 입력 받는 양방향 순환 신경망의 출력이고,

은 양방향 순환 신경망을 의미하며,

는 Residual layer 출력이다.

는 i번째 단어에서 나타나는 j번째 자질의 게이트,

는 j번째 자질 게이트에서 사용하는 가중치,

는 문맥이 반영된

와 자질 벡터

를 연결(concatenation)한 벡터,

는 게이트를 통해 얻은 자질의 가중치와 해당 자질 벡터(

)를 곱한 값,

는 Residual layer 출력(

)과 자질 벡터(

)를 연결한 값,

는 자질 벡터를 의미한다.in Equation 5

is the output of the residual layer

is the output of a bidirectional recurrent neural network that receives

stands for a bidirectional recurrent neural network,

is the residual layer output.

is the gate of the j-th feature appearing in the i-th word,

is the weight used in the j-th feature gate,

is the context

with qualities vector

A vector concatenated by

is the weight of the feature obtained through the gate and the corresponding feature vector (

) multiplied by

is the residual layer output (

) and the feature vector (

) concatenated,

is the feature vector.

Unlike the encoding step, in response extraction, features are selected through a mutually focused result vector. The selected features are connected to the existing mutually focused vector and transferred to the pointer network. The pointer network is as Equation (6).

[Equation 6]

는 포인터 네트워크에서의 attention score,

는 포인터 네트워크의 attention score 계산을 위한 가중치,

는 자질이 적용된 인코딩 벡터와 가중치 곱,

는 현 디코딩 상태 벡터와 가중치 곱,

은 포인터를 위한 디코더 상태 벡터를 나타내며 0번째 벡터는 질의의 인코딩 벡터를 주의집중 풀링하여 생성한다.

는 포인터 네트워크에서 계산된 인코더 위치 분포를 나타내며,

은 포인터 네트워크의 출력으로

은 시작 위치,

은 끝 위치를 의미한다.

은 다음 디코더 스텝 입력,

는 자질이 적용된 인코딩 벡터를 의미한다. In formula (6),

is the attention score in the pointer network,

is the weight for calculating the attention score of the pointer network,

is the weighted product of the encoding vector to which the feature is applied,

is the current decoding state vector and weighted product,

denotes the decoder state vector for the pointer, and the 0th vector is created by attention pooling the query encoding vector.

denotes the encoder position distribution computed in the pointer network,

is the output of the pointer network

is the starting position,

indicates the end position.

is the next decoder step input,

is the encoding vector to which the feature is applied.

In the present invention, SQuAD (Stanford Question Answering Dataset) is used for the experiment. SQuAD has more than 100,000 question-answer pairs, of which 87,599 are learning data and 10,570 are development evaluation data, and official evaluation data has not been released. The experiment learned through the public learning data, tuned the parameters with the development evaluation data, and submitted it to SQuAD to measure the performance.

In the present invention, Exact Match and F1-score are used as performance measures of the model. Exact Match is a method that gives 1 or 0 if the correct answer exactly matches the response predicted by the model, and F1-score is the F1-score measured in words. Table 1 shows the results of comparative analysis with the disclosed invention among the models submitted to SQuAD and the performance mentioned in the invention.

[Table 1]

In Table 1, single is the performance of a single model, and ensemble is the performance measured using multiple models of the same structure. It can be seen that the GF-Net proposed in the present invention shows higher performance than other comparative models.

In the present invention, a machine reading comprehension model is proposed to improve performance by adjusting the importance of input features through feature selection using feature gates. As a result of the experiment using SQuAD, it showed improved performance compared to the existing high-performance comparative model. The feature selection method proposed in the present invention is considered to be effective in adding features because the model adjusts the weight through the gate even if the validity of the new feature is not verified. As a future study, the quality screening method proposed in the present invention will be improved through other methods.

3 is an exemplary diagram for explaining an encoding layer, an interaction layer, and a pointing layer of a high-performance machine reading method through feature selection according to an embodiment of the present invention. 3, in the encoding layer, word embedding vectors (GloVe, Character Embedding, ELMo) and a linguistic feature vector (Feature) are combined through a feature gate to form a Bidirectional Recurrent Neural Network (BiRNN). By being used as input, the document (Context) and the question (Question) are encoded. In the interaction layer, the association between the words in the document and the words in the question is calculated based on the bi-directional co-attention and self-attention mechanisms. . In the pointing layer, after combining the linguistic features with the feature gates to the vectors generated in the interaction step, the location of the correct answer is determined through the pointer network.

4 is an exemplary diagram illustrating a process in which the linguistic quality of a word of a high-performance machine reading method through feature selection is changed in the encoding layer according to an embodiment of the present invention. As shown in Figure 4, "Who was sacked as the first half clock expired?" It shows the process in which the linguistic quality of the word 'Newton' in the document containing the correct answer is changed in the encoding layer when a question is input. As shown in Figure 4, the size of the linguistic features of 'Newton', the part-of-speech feature embedding vector, dependency feature embedding vector, and entity name feature embedding vector, is changed according to the degree to which it contributes to correct answer selection while passing through the feature gate. 4 shows a state in which the contribution is automatically assigned through the learning process in the order of the entity name feature gate (1.0), dependency relationship feature gate (0.8), and part-of-speech feature gate (0.5).

Claims (5)

(a) an encoding step of generating a context encoding vector and a query encoding vector through feature selection through a feature gate that adjusts the reflection degree of features by calculating weights between features;
(b) a bidirectional mutual concentration step of generating a mutually focused result vector between the context and the query of the encoding step;
(c) a response extraction step of transferring the mutual concentration result vector to a pointer network whose weight is set according to the feature gate;
High-performance machine reading method through quality screening, characterized in that it comprises. According to claim 1,
In the encoding step, a normalization vector and a scalar parameter are applied to the vector connecting the embedding vector and the hidden vector of the word used in the language model, and the layer-by-layer abstraction information is mixed in the pre-trained multilayer recurrent neural network language model. A high-performance machine reading method through screening. delete According to claim 1,
The high-performance machine reading method through feature selection, characterized in that the bidirectional mutual concentration step between the context and the query finds a relationship between the context and the query in the context encoding vector and the query encoding vector of the encoding step. According to claim 1,
A high-performance machine reading method through feature selection, characterized in that the features selected in the response extraction step in which the feature selection is reflected are connected to the existing mutually focused vector and transferred to a pointer network.

Images