KR102116518B1 - Apparatus for answering a question based on maching reading comprehension and method for answering a question using thereof - Google Patents

Abstract

The present disclosure relates to a device for answering questions based on machine reading and a method for answering questions using the same. To this end, the query response method includes: generating a feature vector for a question and a paragraph, encoding a hidden condition of a question based on the feature vector of the question, and generating a question vector based on the hidden condition of the question The method comprising: encoding a hidden state of a paragraph based on a feature vector of the paragraph, generating a modeling vector based on the hidden state of the paragraph, and based on the question vector and the modeling vector, And predicting a location.

Description

A question-and-answer device based on machine reading and a question-and-answer method using the same {APPARATUS FOR ANSWERING A QUESTION BASED MACHING READING COMPREHENSION AND METHOD FOR ANSWERING A QUESTION USING THEREOF}

The present disclosure relates to a device for answering questions based on machine reading and a method for answering questions using the same.

Machine Reading Comprehension (MRC) is a machine learning method that enables a machine to read and understand documents. When using a question-and-answer model based on MRC, machine reading is used to learn the location of the correct answer to the question and the question to find the answer in the document. You can infer the location of the correct answer to.

Recently, a deep learning model based on MRC using a recurrent neural network (RNN) has been proposed. The cyclic neural network is effective in processing time-series data such as MRC because it can determine the output of the current state by reflecting information of the previous state. However, in the RNN, the farther away from the current state, the more the effect on the current state is gradually reduced gradient (Vanishing Gradient Problem) occurs. Accordingly, there is a need to build a deep learning model using RNN, which has improved the processing speed while solving the problem of loss of gradient.

The technical problem of the present disclosure is to provide a deep learning model based on RNN and a query response device to which the model is applied.

The technical problem of the present disclosure is to provide a deep learning model that improves a computation speed and a query response device to which the model is applied while solving a gradient loss problem by using a simple recurrent unit (SRU).

The technical problem of the present disclosure is to provide an improved deep learning model and a query response device to which the model is applied by applying a plurality of qualities to questions and paragraphs.

The technical problems to be achieved in the present disclosure are not limited to the technical problems mentioned above, and other technical problems that are not mentioned will be clearly understood by those skilled in the art from the description below. Will be able to.

An apparatus and method for querying and answering based on machine reading according to an aspect of the present disclosure generate a feature vector for a question and a paragraph, encode a hidden state of a question based on the feature vector of the question, and hide the question. A question vector is generated based on a state, a hidden state of the paragraph is encoded based on the feature vector of the paragraph, a modeling vector is generated based on the hidden state of the paragraph, and the question vector and the modeling vector are generated. On the basis of it, the location of the correct answer can be predicted.

In an apparatus and method for querying and answering based on machine reading according to an aspect of the present disclosure, the feature vector may be generated in units of words.

In an apparatus and method for querying and answering based on machine reading according to an aspect of the present disclosure, the hidden state of the question may be encoded in units of sentences.

In a device and method for querying and answering based on machine reading according to an aspect of the present disclosure, the hidden state of the paragraph is encoded in units of words, and the hidden state of the word is configured to transfer the feature vector of the word to a bidirectional Recurrent Neural Network (RNN). It can be created as you type.

In an apparatus and method for querying and answering based on machine reading according to an aspect of the present disclosure, the bidirectional RNN may include a Simple Recurrent Unit (SRU).

In a question and answer apparatus and method based on machine reading according to an aspect of the present disclosure, the modeling vector may be generated based on context information regarding a hidden state sequence sequence of the paragraph.

In a question and answer apparatus and method based on machine reading according to an aspect of the present disclosure, the feature vector may be generated by using at least one feature in a word.

In a question and answer apparatus and method based on machine reading according to an aspect of the present disclosure, the qualities include at least one of word embedding, syllable embedding, exact match, token qualities, and sorted question expressions, wherein the question and the The number and type of qualities applied to a paragraph may be the same.

In a device and method for querying and answering based on machine reading according to an aspect of the present disclosure, the correct answer position may include a start position of a correct answer and an end position of a correct answer.

In a question and answer apparatus and method based on machine reading according to an aspect of the present disclosure, the correct answer position may be corrected according to whether a correct answer corresponding to the correct answer position is terminated in a finalized type.

The features briefly summarized above with respect to the present disclosure are merely illustrative aspects of the detailed description of the present disclosure described below, and do not limit the scope of the present disclosure.

According to the present disclosure, there is an effect that can provide a deep learning model based on RNN.

According to the present disclosure, by using a simple recurrent unit (SRU), while solving the problem of gradient loss, there is an effect that can provide a deep learning model with improved computation speed.

According to the present disclosure, by applying a plurality of qualities to the question and paragraph, there is an effect that can improve the performance of the deep learning model.

The effects obtainable in the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description. will be.

1 is a view showing a comparison of LSTM, GRU and SRU.
2 is a diagram hierarchically illustrating a deep learning model according to the present invention.
FIG. 3 is a block diagram of an MRC-based query response device using the deep learning model shown in FIG. 2.
4 is a flow chart showing an MRC-based query response method according to the present invention.
5 is a diagram illustrating a Korean machine reading data set applied to a deep learning model.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention. In the drawings, similar reference numerals refer to the same or similar functions throughout several aspects. The shape and size of elements in the drawings may be exaggerated for a clearer explanation. For detailed description of exemplary embodiments described below, reference is made to the accompanying drawings that illustrate specific embodiments as examples. These embodiments are described in detail enough to enable those skilled in the art to practice the embodiments. It should be understood that the various embodiments are different, but need not be mutually exclusive. For example, certain shapes, structures, and properties described herein may be implemented in other embodiments without departing from the spirit and scope of the invention in relation to one embodiment. In addition, it should be understood that the location or placement of individual components within each disclosed embodiment can be changed without departing from the spirit and scope of the embodiment. Therefore, the following detailed description is not intended to be taken in a limiting sense, and the scope of exemplary embodiments, if appropriately described, is limited only by the appended claims, along with all ranges equivalent to those claimed.

In the present invention, terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components. For example, the first component may be referred to as a second component without departing from the scope of the present invention, and similarly, the second component may be referred to as a first component. The term and / or includes a combination of a plurality of related described items or any one of a plurality of related described items.

When a component of the present invention is referred to or expressed as being "connected" or "connected" to another component, it may be directly connected to or connected to the other component, but other components in the middle It should be understood that may exist. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Components shown in the embodiments of the present invention are shown independently to indicate different characteristic functions, and each component may be implemented by hardware, software, or a combination thereof. For example, each component is at least one of hardware and / or software driven based on hardware such as a communication unit for performing data communication, a memory for storing data, a control unit (or processor) for performing data processing and calculation, and / or the like. It can be implemented through the above.

In addition, each component shown in the present embodiment does not have to be composed of separate hardware or one software component. That is, for convenience of description, each component is listed and included as each component, and at least two components of each component are combined to form one component, or one component is divided into a plurality of components to perform a function. The integrated and separated embodiments of the constituent parts may also be included in the scope of the present invention without departing from the essence of the present invention.

For example, in the example shown in FIG. 3, the operation performed in each component is performed through hardware such as a control unit (or processor), while data input / output between each component can be stored in a memory or the like.

The terms used in the present invention are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present invention, terms such as “comprise” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance. That is, in the present invention, description of "including" a specific configuration does not exclude configurations other than the corresponding configuration, and means that additional configurations may be included in the scope of the present invention or the technical spirit of the present invention.

Some of the components of the present invention are not essential components for performing essential functions in the present invention, but may be optional components for improving performance. The present invention can be implemented by including only components necessary for realizing the essence of the present invention, except components used for performance improvement, and structures including only essential components excluding optional components used for performance improvement. Also included in the scope of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In describing embodiments of the present specification, when it is determined that detailed descriptions of related well-known configurations or functions may obscure the subject matter of the present specification, detailed descriptions thereof will be omitted, and the same reference numerals will be used for the same components in the drawings. Used and duplicate description of the same components will be omitted.

To build a question-and-answer system based on machine reading, a data set is constructed and machine learning is performed by applying the given data set to a deep learning model. For example, Stanford's SQuAD, Facebook's bAbi, Microsoft's MS-MARCO and other data sets, and DrQA, fastQA, R-Net, AoA reader, BiDirectional Flow (BiDAF), match-LSTM, etc. end-to- end Deep learning models are being studied.

The pointer network-based deep learning model is an extension of the RNN Encoder-Decoder model and refers to a model that outputs a position corresponding to a given input sequence as a result. For example, when using the deep learning model, a boundary index (ie, a start position and an end position) of a correct answer similar to a question in context may be output.

A recurrent neural network (RNN), which is the basis of the deep learning model, is an artificial neural network (ANN) including at least one hidden layer between an input layer and an output layer. Artificial Neural Network). The cyclic neural network is an artificial neural network for processing sequential information or time-series data. The output (y (t)) in the current state (t) is not only the input (x (t)) in the current state (t), The information of the previous state (t-1) (specifically, the previous hidden state (h (t-1)) is also affected.) In order to transfer the information of the previous state to the current state, the RNN stores the information of the previous state. It may include a memory for.

However, in the RNN, a vanishing gradient problem may occur in which the influence of the past state on the present state decreases rapidly as the past distant from the present state. In order to solve the gradient loss problem, a method of configuring the hidden layer with a memory block such as a Long Short-Term Memeroy (LSTM) or a Gated Recurrent Unit (GRU) may be considered.

LSTM is a memory block including an input gate, an output gate, and a forget gate. The input gate may be used to determine whether to receive a new input, or the use of a forward gate to determine whether to transfer information from the previous state to the current state. The output gate may be used to determine whether to apply information transmitted from a previous state to the output value.

The GRU is a memory block in which the output gate is removed from the LSTM, and may include a reset gate and an update gate. That is, the GRU is a simplified structure of the LSTM, and has the advantage of being capable of fast processing due to less computational complexity than the LSTM. In the GRU, a reset gate can be used to determine how to combine the new input with the old memory, and an update gate can be used to determine whether to maintain the old memory information and compute a new state.

A simple recurrent unit (SRU) has a neural gate such as a GRU or LSTM, and like the LSTM or GRU, can solve the gradient loss problem. The SRU eliminates the previous hidden state at the gate input compared to the GRU, and has the advantage of less computation and faster parallel computation than the GRU.

1 is a view showing a comparison of LSTM, GRU and SRU.

1 (a) shows LSTM, (b) shows GRU, and (c) shows SRU. In addition, i is an input gate, f is a forward gate, o is an output gate, z is an update gate, and r is a reset gate. c and h may indicate each internal memory. The following equations (1) to (3) show the operations of LSTM, GRU and SRU, respectively.

In Equations 1 to 3, i is an input gate, f is a forward gate, r is a reset gate, z is an update gate, and o is an output gate. In Equation 1, x may be an input, g may be a candidate hidden state, s may be a hidden state, and c may indicate an internal state. In Equation 2, x may be an input, s may be a hidden state, and h may indicate an internal state. In Equation 3, x may be an input, h may be a hidden state, and c may be an internal state.

In Equations 1 to 3, subscript t indicates information of the current state, and subscript t-1 indicates information of the previous state. In addition, sigma () represents a sigmoid function, tanh () represents a hyperbolic tangent function, and g () represents an active function. The active function may include at least one of a sigmoid function, a hyperbolic function, a ReLU function, or a softmax function. ⊙ can represent elementwise multiplication.

Looking at Equation 3 representing the operation of the SRU, the current internal state (internal sate, i.e., the current memory) c _t can be derived by adjusting information transmission from the current input x _t and the previous internal state c _t-1 .

와 elementwise 곱을 수행하여 입력 정보 반영 여부를 결정하고, 포겟 게이트 f_t는 c_t-1와의 elementwise 곱을 수행하여, 이전 내부 상태(internel state)의 정보를 얼마나 반영할 것인지를 결정할 수 있다. 여기서,

는 입력 x_t와 가중치 W를 곱하여 선형 변환한 결과이고, i_t는 기준값과 포겟 게이트 f_t와의 차분값(예컨대, 1-f_t)으로 유도될 수 있다. 아울러, f_t는 입력 x_t에 대해 피드 포워드 신경망(FFNN, Feed-Forward Neural Network)을 적용한 뒤, 시그모이드(Sigmoid) 함수를 적용하여 유도될 수 있다.The input gate i _t is

And performing elementwise multiplication to determine whether input information is reflected, and forget gate f _t to perform elementwise multiplication with c _t-1 to determine how much to reflect information of a previous internal state. here,

Is a result of linear conversion by multiplying the input x _t and the weight W, and i _t may be derived as a difference value (eg, 1-f _t ) between the reference value and the target gate f _t . In addition, f _t may be derived by applying a feed-forward neural network (FFNN) to the input x _t and then applying a sigmoid function.

Referring to Equation 1, since the forward gate f _t of the LSTM is derived using the previous hidden state s _t-1 , it is considered that the current internal state c _t of the LSTM is also derived using the previous hidden state s _t-1 . Can be. Accordingly, it can be seen that the current hidden state derived based on the current internal state c _t is also derived using the previous hidden state s _t-1 . Referring to Equation 2, it can be seen that the current internal state of the GRU is also derived using the previous hidden state s _t-1 . Accordingly, it can be seen that the current hidden state derived based on the current internal state h _t is also derived using the previous hidden state s _t-1 .

On the other hand, since the SRU calculates the forward gate f _t without using the previous hidden state, it can be seen that the current internal state is determined without using the previous hidden state. Accordingly, the SRU can derive the current hidden state s _t without using the previous hidden state s _t-1 . That is, the SRU is characterized by reducing the amount of calculation by applying FFNN instead of using the previous hidden state, and enabling parallel calculation between layers.

The machine-reading deep learning model described below may include an RNN using at least one of LSTM, GRU, or SRU. Preferably, by configuring the RNN based on the SRU, it is possible to reduce complexity and computational complexity compared to using an LSTM or GRU.

2 is a diagram hierarchically illustrating a deep learning model according to the present invention.

In order to perform machine reading, the deep learning model can perform learning using the question (Q) and paragraph (P) data sets. In the example shown in FIG. 2, a question data set consisting of m words {q ₁ , q ₂ ,… q _m } and n words Paragraph data set {p ₁ , p ₂ ,… It is shown that learning is performed using p _n }.

Depending on the language you want to learn, you can define each word in the paragraph and question based on at least one of morphemes, tokens, spaces, or special characters. For example, when learning English, the learning unit is determined based on tokens, spaces, and special characters, whereas when learning Korean, the learning unit may be determined in units of morphemes. In addition, it is also possible to define a word of a paragraph and a question by pairing a part-of-speech tag indicating a predetermined word and part-of-speech.

For example, after classifying paragraphs or questions into units of morphemes, words may be defined by tagging parts of speech suitable for each morpheme. In the example shown in FIG. 2, nng may be a generic noun, ec is a connecting ending, sn is a number, nnb is a dependent noun, and sf is a part-of-speech tag indicating a punctuation mark indicating a period, question mark, or exclamation point.

Referring to FIG. 2, the deep learning model according to the present invention may include a feature layer 210, a hidden layer 220, a self-matching layer 230, a modeling layer 240, and an output layer 250. .

In the feature layer 210, feature embedding for paragraphs and questions is performed. Specifically, embedding may be performed on words included in the input paragraph and / or words included in the input question based on at least one feature. As a result of embedding, a feature vector for a paragraph and / or a feature vector for a question may be generated.

In the hidden layer 220, a hidden state for a paragraph and / or question may be encoded based on a feature vector generated in the feature layer 210. First, the hidden state for the question may be generated by combining the hidden state for each question word in units of sentences, and then encoding the hidden state for the question sentence into one vector. That is, the hidden state for the question may be encoded in units of sentences. When the hidden state of the question sentence is encoded, an alignment vector may be generated by normalizing the hidden state of the question sentence, and a question vector may be generated using the encoded hidden state vector and the alignment vector.

The hidden state of each word included in the paragraph can be encoded based on the bidirectional RNN. At this time, the bidirectional RNN may include a memory block of at least one of LSTM, GRU, or SRU. Preferably, an RNN can be configured using a bidirectional SRU.

In the self-matching layer 230, it is possible to compensate for a decrease in the influence of the past state word of the paragraph on the present state word. Specifically, it is possible to generate context information of a similar meaning by calculating an attention weight for the own own hidden state sequence. To this end, an alignment score may be calculated for an encoded hidden state sequence of a paragraph, and a context vector may be generated by multiplying the calculated alignment score by a hidden state vector.

In the modeling layer 240, encoding may be performed by concatenating the encoded hidden state vector of each word related to the paragraph and the Munmek vector. Specifically, encoding may be performed using a gated attention-based recurrent network and a modeling vector may be output as a result.

The output layer 250 may find and output a position of a correct answer suitable for a question in a paragraph based on a modeling vector and a question vector. Specifically, based on the pointing method of the pointer network, the modeling vector and the question vector can be calculated with bi-linear sequence attention, and the location of the correct answer can be found and output from the paragraph. At this time, the position of the correct answer may include a start position and an end position of the correct answer in the context.

FIG. 3 is a block diagram of an MRC-based query response device using the deep learning model shown in FIG. 2.

Referring to FIG. 3, the MRC-based query response device includes a feature vector generator 310, a hidden state encoding unit 320, a question vector generator 330, a self-matching attention unit 340, and a modeling vector generator. 350 and the correct answer position output unit 360.

First, the feature vector generator 310 may generate feature vectors for questions and / or paragraphs based on at least one feature in order to numerically transform the learning unit. The qualities used to generate feature vectors are: word embedding, syllable (or character embedding), exact match, token feature, or aligned question embedding. It may include at least one of.

The feature vector generator 310 may include a question feature vector generator 312 for generating a feature vector for a question and a paragraph feature vector generator 315 for generating a feature vector for a paragraph. The feature vector generator 310 may generate feature vectors by concatenating values generated by at least one feature.

Equation 4 below represents the feature vector of the word input at the time point t as an expression, and Equation 5 represents the output value of each feature for the paragraph as an expression. The output value of each feature for the question can be obtained by substituting the variables p and q in Equation 5 with q and p, respectively.

는 문단의 자질 벡터를 나타내고,

는 질문의 자질 벡터를 나타낸다. f_emb는 단어 표현 자질에 기초한 출력값, f_{c_emb}는 음절 표현(또는 문자 표현) 자질에 기초한 출력값, f_{exact_match}는 정확한 매치 자질에 기초한 출력값, f_tf는 토큰 자질에 기초한 출력값, f_align은 정렬된 질문 표현 자질에 기초한 출력값을 나타낸다.In Equation 4,

Denotes the feature vector of the paragraph,

Denotes the feature vector of the question. f _emb is the output value based on word expression qualities, f _{c_emb} is the output value based on syllable expression (or character expression) qualities, f _{exact_match} is output based on exact match qualities, f _tf is output based on token qualities, and f _align is sorted questions Represents an output value based on expression qualities.

Using the word expression qualities, the embedding value for the word can be output. The embedding value may be calculated using a pre-trained word embedding database. For example, a word expression quality value may be obtained using an embedding database learned using a Neural Network Language Model (NNLM).

Using the syllable expression quality, the embedding value for the word can be output. Specifically, after setting the syllable expression quality value to an arbitrary initial value, an embedding value of a word may be output using a convolutional neural network (CNN).

The output value based on the exact match quality may be determined based on whether the paragraph word p _t is included in the question or whether the question word q _t is included in the paragraph. Specifically, 0 or 1 may be output depending on whether the paragraph word p _t is included in the question or whether the question word q _t is included in the paragraph.

The output value of the token quality is determined by normalizing the frequency of each word. For example, the output value based on the token quality may be determined as the number of times the input word appears in the context or the question or a value corresponding to the number of times.

The output value of the aligned question expression feature may be calculated based on an alignment score for a paragraph expression and a question expression and a context encoder vector. Specifically, after calculating an alignment score between a sequence of words included in a question (ie, a question sequence) and a sequence of words included in a paragraph (ie, a paragraph sequence), the calculated result is multiplied by a context encoder vector to match the context You can output a vector (matching context vector).

The same number and type of qualities can be applied to questions and paragraphs. For example, as shown in Equation 4, word expressions, syllable (or letter) expressions, exact matches, token qualities, and aligned question expressions can be commonly applied to questions and paragraphs.

As another example, at least one of the number or type of qualities applied to the question and the paragraph may be set differently. For example, the feature vector of the question may be obtained using n of the listed features, while the feature vector of the paragraph may be obtained using m (different from n) of the listed features.

Alternatively, at least one of the number or types of qualities may be set differently according to the learning language. For example, in a question answering device based on Korean language, n qualities may be set, and in a question answering device based on English, m qualities may be set.

The hidden state encoding unit 320 may encode and output a hidden state for a paragraph and / or question based on the feature vector generated by the feature vector generator 310. The hidden state encoding unit 320 may include a question hidden state encoding unit 322 for encoding a hidden state related to a question and a paragraph hidden state encoding unit 325 for encoding a hidden state related to a paragraph.

The question hidden state encoding unit 322 may output the encoded hidden state for the question sentence by combining the hidden states for each question word in units of sentences. Specifically, when the hidden state is output by inputting the feature vector of each word into the bidirectional RNN, the hidden state for the question sentence can be obtained by combining the outputted hidden states into one. The bidirectional RNN may include a memory block of at least one of LSTM, GRU, or SRU, and preferably, may include SRU.

The paragraph hidden state encoding unit 325 may encode the hidden state for each word by inputting a feature vector of each word included in the paragraph into the bidirectional RNN. The bidirectional RNN may include a memory block of at least one of LSTM, GRU, or SRU, and preferably, may include SRU. Equation 6 below shows an example of encoding a hidden state for a paragraph.

In Equation 6, u ^P _t represents an encoded hidden state. As shown in Equation (6), an encoded hidden state u ^P _t may be generated by inputting a previously encoded hidden state u ^P _t-1 and a feature vector into a bidirectional SRU (BiSRU).

The question vector generator 330 may generate the alignment vector by normalizing the hidden state of the question sentence, and multiply the encoded hidden state of the question sentence by the alignment vector to generate the question vector. Equation 7 below shows an example in which a question vector is generated.

In Equation 7, q represents a question vector and b _j represents an alignment vector. As in the example shown in Equation 7 above, the alignment vector b _j is obtained by normalizing the hidden state q _j of the question sentence, and the hidden state q _j of the question sentence is multiplied by the alignment vector b _j to generate a question vector q. have.

The self-matching attention unit 340 may obtain an alignment score for a sequence of encoded hidden states of the paragraph. In addition, the context vector may be generated by multiplying the encoded hidden state vector and the alignment score. Equation 8 below shows an example in which a context vector is generated.

In Equation 8, c _t represents a context vector, and a _i ^t represents an alignment score. As shown in Equation 8, when an encoded hidden state sequence of a paragraph is input, an alignment score may be obtained using the encoded hidden state u _j ^P and u _t ^P of words in the sequence. Here, the sequence may be divided into sentence units or a predetermined number of units of data. Then, the context vector c _t may be calculated by multiplying the input hidden state sequence and the alignment score.

The modeling vector generation unit 350 may connect the encoded hidden state of the paragraph and the Munmek vector, and generate a modeling vector based on this. As an example, Equation 9 shows an equation for calculating a modeling vector.

In Equation 9, [u _t ^P ; c _t ] indicates that the hidden state is connected to the context vector, and [u _t ^P ; c _t ] * is the gated attention-based recurrent network (gated attention-based recurrent network) [u _t ^P ; c Displays output when _t ] is input. Specifically, in the circulating network based on the gated attention [u _t ^P ; When c _t ] is input, [u _t ^P ; c _t ] is a nonlinear gate layer produced by applying sigmoid (Sigmoid) g _t and [u _t ^P ; The elementwise sum between c _t ] may be output as an output value. As shown in Equation 9, the modeling vector h _t ^P is the modeling vector h _t-1 ^P and [u _t ^P ; c _t ] * to the bidirectional SRU. In Equation 9, it is shown that a modeling vector is generated using a bidirectional SRU, but it is also possible to generate a modeling vector using a bidirectional LSTM or bidirectional GRU.

The correct answer position prediction unit 360 may predict the position of the correct answer to the question entered in the paragraph using the question vector and the modeling vector. In this case, the output result of the correct answer position prediction unit 360 may include a start position and an end position of the correct answer. Equation 10 below shows an example of calculating the correct answer position.

In Equation 10 above, P _start (t) represents the start position of the correct answer, and P _end (t) represents the end position of the correct answer.

The length between the start position and the end position of the correct answer may be limited to a predefined maximum length or less. Here, the maximum length may indicate the number of letters or words. For example, when the maximum length represents 50 words, the length between the start position and the end position of the correct answer may be limited to 50 morphemes.

Also, a minimum length between the start position and end position of the correct answer may be defined. For example, when the distance between the start position and the end position of the correct answer is shorter than the minimum length of the correct answer, it may be corrected such that at least one of the starting position of the correct answer or the ending position of the correct answer is extended to adjacent words.

If the correct answer prediction is performed in units of words and the words are defined in morphological units, a problem may arise in which the correct answer is not terminated in a complete form. For example, if the words across the searched start position and end position are not terminated in a self-contained morpheme, it may be determined that the correct answer is not terminated in a complete form. In order to solve the above problems, after performing the correct answer prediction, when outputting the final correct answer, correct the position of the correct answer so that the independent morpheme is located at the ending of the correct answer, or a unit different from the morpheme (for example, blank, special Character or sentence) to correct the position of the correct answer. Alternatively, the correct answer may be found and output from the original paragraph using the predicted result, but the output correct answer may be set not to be limited to a morpheme unit.

Alternatively, a correct answer may be output by adding a predefined ending ending.

4 is a flow chart showing an MRC-based query response method according to the present invention.

Referring to FIG. 4, first, through the feature vector generator 310, feature vectors for questions and paragraphs may be generated (S410). In this case, the learning units of the questions and paragraphs may be word units, and accordingly, feature vectors for each word may be generated. A feature vector for each word can be generated by concatenating values calculated by at least one feature.

When a feature vector for a question and a paragraph is generated, a hidden state of the question and the paragraph may be encoded using the generated feature vector (S420). The hidden state of the question can be encoded in units of sentences. Specifically, the hidden state of the question sentence can be generated by combining the hidden state of the words included in the question in units of sentences.

The paragraph's hidden state can be encoded word by word. Specifically, the hidden state of each word can be obtained by inputting the feature vector of each word into the bidirectional RNN. In this case, the bidirectional RNN may include at least one memory block of LSTM GRU or SRU.

When the hidden state of the question sentence is encoded, a question vector may be generated based on the hidden state of the question sentence (S430). Specifically, the question vector can be generated by multiplying the alignment vector of the question sentence and the hidden state of the question sentence.

When the hidden state for the paragraph word is encoded, a context vector may be generated based on the hidden state of each word (S440). The context vector may be generated through a self-matching attention process that adjusts the attention weight based on the similarity between words included in the paragraph.

Thereafter, a modeling vector for each word may be generated based on the hidden state and context vector of each word (S450), and a correct answer position may be predicted based on the question vector and the modeling vector (S460). The predicted correct answer position may be corrected according to the minimum length or the maximum length. In addition, the correct answer position is predicted in the same unit as the learning unit.

As described above, the query response device based on the machine reading according to the present invention learns paragraphs and questions based on the RNN and the self-matching network. In this case, a deep learning model to which an RNN composed of SRUs and a self-matching network is applied may be referred to as S ² -Net.

The deep learning model based on MRC may include a question and answer data set. As an example, FIG. 5 is a diagram illustrating a Korean machine reading data set applied to a deep learning model.

The data set for machine reading can follow the JavaScript Object Notation (JSON) format. In FIG. 5, set represents a key including the entire data set, and Version represents a current version of the data set. Version can be displayed as a string.

Data represents a list including all data included in the current data set. In one example, title refers to the title of the document, and paragraphs refers to a list containing paragraph, question, and answer information against the document title.

context_original represents actual text. dp denotes a list containing the result of the dependent parsing of the text corresponding to context_original, and qas denotes a list containing question and answer information about the document. For example, dp may include a head (center word) and mods (modifiers) indicating a dependency relationship, a label indicating label information of a corresponding relationship, and id, text, and weight of each word. At this time, weight represents a weight result for a dependency relationship. qas may include a question_original which is the actual string of the question, the id of the question, a list in the question_ which is the result of parsing the dependency of the question, a question that includes the morpheme result of the actual question, and an answer key that includes the correct answer information accordingly. .

In relation to the correct answer corresponding to the question, it may include answer_start and answer_end indicating text_original that is the actual string of the correct answer, text list that is the morpheme information of the correct answer, and index information of the start and end in which the correct answer appears in the document.

In the above, the configuration of the query response device based on the machine reading described with reference to FIGS. 1 to 5 and the operation sequence thereof are not limited to the illustrated example. For example, the present invention may be performed while some of the illustrated systems or steps are omitted, or the present invention may be performed by adding components not shown.

Further, the learning system and the question and answer system according to the present invention may be implemented by hardware, software, or a combination thereof, as mentioned at the outset. For example, the systems may be implemented by computing devices capable of digital computing (eg, PC, smart phone, PDA, tablet PC, server, cloud computer, etc.). In addition, the system may be implemented based on at least one of software or hardware constituting these computing devices.

In the above, the present invention has been described by specific matters such as specific components, etc. and limited embodiments and drawings, which are provided to help the overall understanding of the present invention, but the present invention is not limited to the above embodiments , Those skilled in the art to which the present invention pertains can make various modifications and variations from these descriptions.

Therefore, the spirit of the present invention is not limited to the above-described embodiment, and should not be determined, and all claims that are equally or equivalently modified with the claims as described below belong to the scope of the spirit of the present invention. Would say

310: feature vector generator
320: hidden state encoding unit
330: question vector generator
340: self-matching attention section
350: modeling vector generator
360: correct answer position output

Claims (20)

In the query response method performed by the query response device,
Generating feature vectors for questions and paragraphs;
Encoding a hidden state of the question based on the feature vector of the question;
Generating a question vector based on the hidden state of the question;
Encoding a hidden state of the paragraph based on the feature vector of the paragraph;
Generating a modeling vector based on the hidden state of the paragraph; And
And predicting a correct answer position in the paragraph based on the question vector and the modeling vector,
The predicting step
A method for answering a question based on machine reading, wherein the correct answer position is corrected according to whether a correct answer corresponding to the correct answer position is terminated in a finalized type. According to claim 1,
Each of the feature vector of the question and the feature vector of the paragraph is generated in units of words. According to claim 2,
A method for answering a question based on machine reading, characterized in that the hidden state of the question is encoded in units of sentences. According to claim 2,
The hidden state of the paragraph is encoded in units of words, and the hidden state of the word is generated by inputting a feature vector of the word into a bidirectional Recurrent Neural Network (RNN), wherein the query response method is based on machine reading. According to claim 4,
The bidirectional RNN is characterized in that it comprises a Simple Recurrent Unit (SRU), a question and answer method based on machine reading. According to claim 4,
The modeling vector is generated based on context information about a hidden state sequence sequence of the paragraph, wherein the query response method based on machine reading. According to claim 2,
The feature vector is characterized by being generated by using at least one feature in a word, a method for answering a question based on machine reading. The method of claim 7,
The qualities include at least one of word embeddings, syllable embeddings, exact matches, token qualities and sorted question expressions,
A question and answer method based on machine reading, characterized in that the number and type of qualities applied to the question and the paragraph are the same. According to claim 1,
The correct answer position includes a start position of the correct answer and an end position of the correct answer,
The length between the start position and the end position, characterized in that less than the predefined maximum length, the machine-based query response method. The method of claim 1
The correct answer position includes a start position of the correct answer and an end position of the correct answer,
The predicting step
If the word spanning at least one of the start position and the end position does not end in a self-contained morpheme,
A method for answering a query based on machine reading, correcting at least one of the start position and the end position. A feature vector generator that generates feature vectors for questions and paragraphs,
A hidden state encoding unit encoding a hidden state of the question based on the feature vector of the question, and encoding the hidden state of the paragraph based on the feature vector of the paragraph;
A question vector generator that generates a question vector based on the hidden state of the question;
A modeling vector generator that generates a modeling vector based on the hidden state of the paragraph; And
And a correct answer position predicting unit predicting a correct answer position in the paragraph based on the question vector and the modeling vector,
The correct answer position prediction unit
A machine for reading and answering based on machine reading, correcting the position of the correct answer according to whether the correct answer corresponding to the correct answer position is terminated in a finalized type The method of claim 11,
Each of the feature vector of the question and the feature vector of the paragraph is generated in units of words, and a device for querying and answering based on machine reading. The method of claim 12,
And the hidden state encoding unit encodes the hidden state of the question in units of sentences. The method of claim 12,
The hidden state encoding unit encodes the hidden state of the paragraph in units of words,
The hidden state of the word is generated by inputting the feature vector of the word into a bidirectional Recurrent Neural Network (RNN), a query response device based on machine reading. The method of claim 14,
The bidirectional RNN is characterized in that it comprises a Simple Recurrent Unit (SRU), a query response device based on machine reading. The method of claim 14,
The modeling vector generating unit generates a modeling vector based on context information regarding a hidden state sequence sequence of the paragraph, and a machine for reading and answering based on machine reading. The method of claim 12,
The feature vector generating unit generates a feature vector by applying at least one feature to a word, and a device for querying and answering based on machine reading. The method of claim 17,
The qualities include at least one of word embeddings, syllable embeddings, exact matches, token qualities and sorted question expressions,
The number and types of qualities applied to the question and the paragraph are the same, characterized in that the machine, based on machine reading and answering questions. The method of claim 11,
The correct answer position includes a start position of the correct answer and an end position of the correct answer,
The length between the start position and the end position, characterized in that less than a predefined maximum length, machine-based query response device. The method of claim 19,
The correct answer position includes a start position of the correct answer and an end position of the correct answer,
The correct answer position prediction unit,
A machine based on machine reading, correcting at least one of the start position and the end position when a word spanning at least one of the start position and the end position is not terminated in a free-standing morpheme.

Images