KR20220056633A - Machine reading comprehension method and apparatus - Google Patents

Abstract

Disclosed are a machine reading comprehension method and apparatus. According to one embodiment of the present invention, the machine reading comprehension method comprises: a step of receiving a context and question consisting of a natural language; a step of generating an embedding vector based on the context and question; a step of performing encoding at least one time based on the embedding vector; and a step of predicting a right answer for the question based on the encoded embedding vector.

Description

MACHINE READING COMPREHENSION METHOD AND APPARATUS

Embodiments relate to a machine reading method and apparatus.

Machine reading comprehension is a technology that finds and returns the correct answer to a user's natural language query in context. With machine reading, a machine reading algorithm can find meaning in a sentence and answer it, just as a human reads a text and infers an answer to a question.

In the conventional machine reading method, not only the probabilities of the starting and ending words are analyzed independently, but also the boundary between sentences is not explicit, so the first and last words of the correct answer are not extracted from the same sentence, so the performance of the model is poor. There is a problem with degradation.

Embodiments may provide machine reading techniques. However, the technical tasks are not limited to the above-described technical tasks, and other technical tasks may exist.

A machine reading method according to an embodiment includes the steps of receiving a context and a question made of a natural language, generating an embedding vector based on the context and the question, and performing at least one encoding based on the embedding vector. and predicting a correct answer to the question based on the encoded embedding vector.

The generating of the embedding vector may include generating a first embedding vector based on the context and generating a second embedding vector based on the question.

The machine reading method may further include matching the context with the words included in the question by performing an attention based on the first embedding vector and the second embedding vector.

The performing of the encoding includes: performing positional encoding on the embedding vector; and extracting a local feature by performing a convolution operation on the embedding vector on which the positional encoding has been performed. and extracting a global feature by performing self-attention based on the local feature.

The performing of the at least one encoding may be performed by a neural network including a residual block.

The machine reading method may further include calculating a score for the sentence included in the context and the question by performing encoding based on the encoded embedding vector.

The predicting of the correct answer may include calculating a probability of the start word of the correct answer, calculating the probability of the end word of the correct answer, and the correct answer based on the probability of the start word and the probability of the end word It may include the step of predicting.

The predicting of the correct answer may further include correcting the predicted correct answer based on the context and scores for the sentences included in the question.

A machine reading apparatus according to an embodiment includes a receiver that receives a context and a question made of a natural language, generates an embedding vector based on the context and the question, performs at least one encoding based on the embedding vector, and encodes and a processor for predicting a correct answer to the question based on the embedded vector.

The processor may generate a first embedding vector based on the context, and generate a second embedding vector based on the question.

The processor may match the context with the words included in the question by performing an attention based on the first embedding vector and the second embedding vector.

The processor performs positional encoding on the embedding vector, and extracts a local feature by performing a convolution operation on the embedding vector on which the positional encoding is performed, and based on the local feature A global feature can be extracted by performing self-attention.

The processor may perform the at least one encoding by using a neural network including a residual block.

The processor may calculate a score for the context and the sentence included in the question by performing encoding based on the encoded embedding vector.

The processor may calculate a probability of a start word of the correct answer, calculate a probability of an end word of the correct answer, and predict the correct answer based on the probability of the start word and the probability of the end word.

The processor may correct the predicted correct answer based on the score for the context and the sentence included in the question.

1 is a schematic block diagram of a machine reading apparatus according to an embodiment.
FIG. 2 shows the operation of the machine reading apparatus shown in FIG. 1 .
FIG. 3 shows the structure of the encoding block shown in FIG. 2 .
4 shows an example of self-attention.
FIG. 5 is a flowchart of an operation of the machine reading apparatus shown in FIG. 1 .

Specific structural or functional descriptions of the embodiments are disclosed for purposes of illustration only, and may be changed and implemented in various forms. Accordingly, the actual implementation form is not limited to the specific embodiments disclosed, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical spirit described in the embodiments.

Although terms such as first or second may be used to describe various elements, these terms should be interpreted only for the purpose of distinguishing one element from another. For example, a first component may be termed a second component, and similarly, a second component may also be termed a first component.

When a component is referred to as being “connected to” another component, it may be directly connected or connected to the other component, but it should be understood that another component may exist in between.

The singular expression includes the plural expression unless the context clearly dictates otherwise. In this specification, terms such as "comprise" or "have" are intended to designate that the described feature, number, step, operation, component, part, or combination thereof exists, and includes one or more other features or numbers, It should be understood that the possibility of the presence or addition of steps, operations, components, parts or combinations thereof is not precluded in advance.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art. Terms such as those defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted in an ideal or excessively formal meaning unless explicitly defined in the present specification. does not

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In the description with reference to the accompanying drawings, the same components are assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

A module in the present specification may mean hardware capable of performing functions and operations according to each name described in this specification, or may mean computer program code capable of performing specific functions and operations, , or an electronic recording medium on which a computer program code capable of performing specific functions and operations is loaded, for example, may refer to a processor or a microprocessor.

In other words, a module may mean a functional and/or structural combination of hardware for carrying out the technical idea of the present invention and/or software for driving the hardware.

1 is a schematic block diagram of a machine reading apparatus according to an embodiment.

Referring to FIG. 1 , the machine reading apparatus 10 may predict a correct answer to a given question or query by analyzing text made in natural language. The machine reading apparatus 10 may analyze a natural language using a neural network and predict a correct answer to a question.

The machine reading device 10 may find and return the correct answer to the user's question in a related document, or notify the user if there is no proper answer to the question. The machine reading apparatus 10 may predict a correct answer to a question using at least one encoder block.

Unlike the existing information search and question-and-answer technology, the machine reading apparatus 10 may receive a query made in natural language and predict the correct answer regardless of the format of the question. Since the machine reading apparatus 10 directly returns the correct answer to the query to the user instead of providing a document related to the query, it is possible to provide great convenience to the user.

Neural networks (or artificial neural networks) can include statistical learning algorithms that mimic the neurons of biology in machine learning and cognitive science. A neural network may refer to an overall model having problem-solving ability by changing the bonding strength of synapses through learning in which artificial neurons (nodes) formed a network by bonding of synapses.

The neural network may include a deep neural network. Neural networks include Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Perceptron, Feed Forward (FF), Radial Basis Network (RBF), Deep Feed Forward (DFF), Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), Auto Encoder (AE), Variational Auto Encoder (VAE), Denoising Auto Encoder (DAE), Sparse Auto Encoder (SAE), Markov Chain (MC), Hopfield Network (HN), Boltzmann Machine (BM) ), Restricted Boltzmann Machine (RBM), Deep Belief Network (DBN), Deep Convolutional Network (DCN), Deconvolutional Network (DN), Deep Convolutional Inverse Graphics Network (DCIGN), Generative Adversarial Network (GAN), Liquid State Machine (LSM) ), Extreme Learning Machine (ELM), Echo State Network (ESN), Deep Residual Network (DRN), Differential Neural Computer (DNC), Neural Turning Machine (NTM), Capsule Network (CN), Kohonen Network (KN), and AN (Attention Network) may be included.

The machine reading apparatus 10 may be implemented in the form of a mobile terminal. For example, the machine reading device 10 may be implemented as an IoT device, a machine-type communication device, or a portable electronic device.

Portable electronic devices include laptop computers, mobile phones, smart phones, tablet PCs, mobile internet devices (MIDs), personal digital assistants (PDAs), and enterprise digital assistants (EDAs). ), digital still camera, digital video camera, PMP (portable multimedia player), PND (personal navigation device or portable navigation device), handheld game console, e-book (e-book), may be implemented as a smart device. For example, the smart device may be implemented as a smart watch or a smart band.

The machine reading apparatus 10 includes a receiver 100 and a processor 200 . The machine reading apparatus 10 may further include a memory 300 .

The receiver 100 may include a receiving interface. The receiver 100 may receive a context and a question (or a query) made of natural language. Context and questions may be in the form of text. The receiver 100 may output the received context and question to the processor 200 .

The processor 200 may process data stored in the memory 300 . The processor 200 may execute computer readable codes (eg, software) stored in the memory 300 and instructions induced by the processor 200 .

The “processor 200” may be a data processing device implemented in hardware having circuitry having a physical structure for performing desired operations. For example, desired operations may include code or instructions included in a program.

For example, a data processing device implemented as hardware includes a microprocessor, a central processing unit, a processor core, a multi-core processor, and a multiprocessor. , an Application-Specific Integrated Circuit (ASIC), and a Field Programmable Gate Array (FPGA).

The processor 200 may generate an embedding vector based on the context and the question. The processor 200 may generate the first embedding vector based on the context. The processor 200 may generate a second embedding vector based on the question.

The processor 200 may perform encoding at least once based on the embedding vector. The processor 200 may perform encoding at least once by using the neural network including the residual block.

The processor 200 may perform positional encoding on the embedding vector. The processor 200 may extract a local feature by performing a convolution operation on the embedding vector on which the positional encoding has been performed. The processor 200 may extract a global feature by performing self-attention based on the local feature.

The processor 200 may match the words included in the context and the question based on the generated embedding vector. For example, the processor 200 may match the context and words included in the question by performing an attention based on the first embedding vector and the second embedding vector.

The processor 200 may calculate a score for a sentence included in a context and a question by performing encoding based on the encoded embedding vector. The processor 200 may predict a correct answer to the question based on the encoded embedding vector.

The processor 200 may calculate the probability of the start word of the correct answer and calculate the probability of the end word of the correct answer. The processor 200 may predict the correct answer based on the probability of the start word and the probability of the end word.

The processor 200 may correct the predicted correct answer based on the context and the score for the sentence included in the question.

The memory 300 may store instructions (or programs) executable by the processor 200 . For example, the instructions may include instructions for executing an operation of a processor and/or an operation of each component of the processor.

The memory 300 may be implemented as a volatile memory device or a nonvolatile memory device.

The volatile memory device may be implemented as dynamic random access memory (DRAM), static random access memory (SRAM), thyristor RAM (T-RAM), zero capacitor RAM (Z-RAM), or twin transistor RAM (TTRAM).

Nonvolatile memory devices include EEPROM (Electrically Erasable Programmable Read-Only Memory), Flash memory, MRAM (Magnetic RAM), Spin-Transfer Torque (STT)-MRAM (Spin-Transfer Torque (STT)-MRAM), Conductive Bridging RAM (CBRAM) , FeRAM(Ferroelectric RAM), PRAM(Phase change RAM), Resistive RAM(RRAM), Nanotube RRAM(Nanotube RRAM), Polymer RAM(Polymer RAM(PoRAM)), Nano Floating Gate Memory (NFGM)), a holographic memory, a molecular electronic memory device, or an Insulator Resistance Change Memory.

FIG. 2 shows the operation of the machine reading apparatus shown in FIG. 1 .

Referring to FIG. 2 , the processor 200 may predict a correct answer to a question using a neural network or a neural network model 210 (hereinafter, referred to as a model 210 ).

The processor 200 may predict the correct answer to the question using two modules: a module that encodes information in units of sentences to calculate a probability that each sentence contains a correct answer, and a module that finds the position of a correct sentence.

The processor 200 may predict the correct answer by multiplying the probability that each sentence is finally the correct answer by the probability of the correct sentence. Through this, the processor 200 can predict the probability of the actual correct phrase very accurately.

In addition, the two modules described above can perform joint learning on a neural network by sharing and optimizing some free-parameters. The processor 200 may maximize the generalization effect of each module through joint learning.

The model 210 may include a baseline 211 and a sentiment scorer 213 . In other words, the two modules described above may be the baseline 211 and the intensity scorer 213 .

The baseline 211 may find the position of the correct answer phrase. The sense scorer 213 may calculate a probability that each sentence includes a correct answer.

The sense scorer 213 may output a probability of 0 when an arbitrary sentence includes a correct answer, and a probability of 1 when not. The processor 200 may correct the predicted correct answer using the output of the sense scorer 213 .

The processor 200 may generate an embedding vector by performing embedding on the words included in the context and the question. The processor 200 may map the word to a continuous vector space by performing embedding on the input word.

The processor 200 vectorizes and embeds words included in contexts and questions using a combination of word embedding that maps 1:1 with vectors and text-based word embedding that encodes the characters of the input word with a convolutional neural network (CNN). You can create vectors. In this case, the neural network that vectorizes the context and the question may share the same parameters.

The embedding encoder block may encode the context of the document and question transformed into the input embedding vector. The embedding encoder blocks may use the same neural network and share parameters. That is, the embedding encoder block corresponding to the context and the embedding encoder block corresponding to the question may share a parameter.

The processor 200 may calculate the weight of the word of the context for the query and the word of the question about the context from the embedding-encoded document and the embedding vector corresponding to the query using the context-query attention structure.

The processor 200 may match the context and words included in the question by performing a context-query attention based on the first embedding vector and the second embedding vector. In other words, the processor 200 may determine to which words the words of the context and the words of the question are matched with each other.

The processor 200 may separate a hidden matrix of a context for each sentence. The processor 200 may construct sentence embedding, which is a vector expression for a sentence, by extracting features from words belonging to the sentence using max pooling.

The processor 200 may analyze the correlation between sentences through the final encoder block, and represent the probability that each sentence becomes a correct answer as between 0 and 1 through a sigmoid function.

The processor 200 may combine the calculated weight results to extract the words included in the context and the words included in the question and a structure. The processor 200 may re-encode the extracted structure using the encoder block.

The processor 200 may generate each sentence vector by performing sentence-wise max-pooling as well as encoding for each word, and may calculate a sense score indicating fitness for each sentence.

The sense score indicates the degree of relevance of a sentence to a question, and may indicate a value between 0 and 1 using a sigmoid function. The processor 200 may correct the predicted correct answer based on the calculated sense score.

The processor 200 may finally calculate a correct answer probability of each word using a feed forward neural network (FFNN) for each word, and calculate a final probability by multiplying each word by a previously calculated sense score to predict a correct answer.

The processor 200 may predict which phrase in the context will be the correct answer to the question by calculating the probability distribution of the start word and the end word of the correct phrase. For example, the processor 200 may predict the correct answer to the question by multiplying the probability that each sentence has a correct answer and the probability that each paragraph is the correct answer.

The encoder block may include a passage encoder block and a sentence encoder block. The passage encoder block can find the correct answer by encoding the relevant paragraph. The sense encoder block can independently analyze the position of the sentence in which the correct answer exists by encoding the related sentence. The encoder block will be described in detail with reference to FIG. 3 .

The processor 200 may train a neural network for machine reading. The processor 200 may simultaneously learn the syntax prediction result of the correct answer and the loss function for the correct sentence sentence prediction result. Through this, the processor 200 may accurately know the boundary of the sentence of the correct sentence.

The processor 200 may improve machine reading performance by sharing free variables in the process of training the neural network for extracting correct answer candidate phrases and predicting correct answer sentences, and flawing the loss function.

The processor 200 may perform machine reading or learning of a neural network using the following script.

1. Main.py: define train and test codes

2. Config.py: Defines hyperparameters required for model configuration

3. Layer.py: Defines the modules required to construct a neural network

4. Model.py: define MRC neural network model

5. Prepro.py: Tokenize, data processing necessary for text analysis, training, and inference such as embedding

6. Util.py: Defines other modules necessary for learning inference such as Evaluation and batch

For learning, the processor 200 may first perform preprocessing based on Prepro.py to process the learning data into an input/output form required for learning and inference.

The processor 200 may perform learning and testing based on the contents of main.py, and the neural network model and configuration may be changed through model.py and config.py.

3 shows the structure of the encoding block shown in FIG. 2, and FIG. 4 shows an example of self-attention.

3 and 4 , the model 210 may include at least one encoder block 215 . According to the example of FIG. 3 , the model 210 may include an embedding encoder block, a passage encoder block, and a sense encoder block.

The encoder block 215 may perform encoding on the input. The encoder block 215 may include a positional encoding block, a CNN, a self-attention block, and an FFNN.

The positional encoding block may give a signal according to the absolute position of the input embedding vector. The positional encoding block may transform the relative position of the word converted into the embedding vector into the absolute position of each word.

For example, the positional encoding block may indicate a value obtained by normalizing the absolute position of each word using a sin or cos function. The positional encoding block may perform positional encoding through a process of converting sin(x+y) into a linear combination of sin(x) and sin(y).

The positional encoding block may express the meaning and position of each word by adding a value obtained by normalizing the absolute position of each word to the input embedding vector.

The CNN may extract a local feature of a word corresponding to the embedding vector by performing at least one convolution operation on the embedding vector on which the positional encoding has been performed.

As in the example of FIG. 4 , the self-attention block may represent each word in a context in a graph structure, and extract global features from the input embedding vector.

FFNN can perform nonlinear transformation on local and global features extracted from CNN and self-attention blocks.

The processor 200 may improve the efficiency of deep learning by applying a residual block that adds an output to an input in a CNN, a self-attention block, and an FFNN.

FIG. 5 is a flowchart of an operation of the machine reading apparatus shown in FIG. 1 .

Referring to FIG. 5 , the receiver 100 may receive a context and a question made of natural language ( 510 ).

The processor 200 may generate an embedding vector based on the context and the question ( 530 ). The processor 200 may generate the first embedding vector based on the context. The processor 200 may generate a second embedding vector based on the question.

The processor 200 may match the context and words included in the question by performing an attention based on the first embedding vector and the second embedding vector.

The processor 200 may perform encoding at least once based on the embedding vector. The processor 200 may perform encoding at least once using a neural network including the residual block.

The processor 200 may perform positional encoding on the embedding vector. The processor 200 may extract a local feature by performing a convolution operation on the embedding vector on which the positional encoding has been performed. The processor 200 may extract the global feature by performing self-attention based on the local feature.

The processor 200 may predict a correct answer to the question based on the encoded embedding vector. The processor 200 may calculate the probability of the starting word of the correct answer. The processor 200 may calculate the probability of the last word of the correct answer. The processor 200 may predict the correct answer based on the probability of the start word and the probability of the end word.

The processor 200 may calculate a score for a sentence included in a context and a question by performing encoding based on the encoded embedding vector. The processor 200 may correct the predicted correct answer based on the context and the score for the sentence included in the question.

The embodiments described above may be implemented by a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the apparatus, methods, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA) array), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using a general purpose computer or special purpose computer. The processing device may execute an operating system (OS) and a software application running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or apparatus, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in a computer-readable recording medium.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination, and the program instructions recorded on the medium are specially designed and configured for the embodiment, or are known and available to those skilled in the art of computer software. may be Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like.

The hardware devices described above may be configured to operate as one or a plurality of software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited drawings, those of ordinary skill in the art may apply various technical modifications and variations based thereon. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (17)

receiving a context and a question in natural language;
generating an embedding vector based on the context and question;
performing at least one encoding based on the embedding vector; and
predicting the correct answer to the question based on the encoded embedding vector
A machine reading method comprising:
According to claim 1,
The step of generating the embedding vector comprises:
generating a first embedding vector based on the context; and
generating a second embedding vector based on the question
machine reading method comprising
3. The method of claim 2,
matching the context with the words included in the question by performing an attention based on the first embedding vector and the second embedding vector
A machine reading comprehension method further comprising a.
According to claim 1,
The encoding step includes:
performing positional encoding on the embedding vector;
extracting a local feature by performing a convolution operation on the embedding vector on which the positional encoding has been performed; and
extracting a global feature by performing self-attention based on the local feature
A machine reading method comprising:
According to claim 1,
The at least one encoding step comprises:
performed by a neural network including a residual block
machine reading method.
According to claim 1,
calculating a score for the sentence included in the context and the question by performing encoding based on the encoded embedding vector
A machine reading comprehension method further comprising a.
The method of claim 1,
The step of predicting the correct answer is,
calculating a probability of the starting word of the correct answer;
calculating the probability of the end word of the correct answer; and
predicting the correct answer based on the probability of the start word and the probability of the end word
A machine reading method comprising:
8. The method of claim 7,
The step of predicting the correct answer is,
correcting the predicted correct answer based on the score for the sentence included in the context and the question
A machine reading comprehension method further comprising a.
A computer program stored in a medium for executing the method of any one of claims 1 to 8 in combination with hardware.
a receiver for receiving context and questions in natural language; and
A processor that generates an embedding vector based on the context and the question, performs encoding at least once based on the embedding vector, and predicts a correct answer to the question based on the encoded embedding vector
A machine reading device comprising a.
11. The method of claim 10,
The processor is
generate a first embedding vector based on the context;
generating a second embedding vector based on the question
mechanical reading device.
12. The method of claim 11,
The processor is
matching the context and words included in the question by performing an attention based on the first embedding vector and the second embedding vector
mechanical reading device.
11. The method of claim 10,
The processor is
performing positional encoding on the embedding vector,
Extracting a local feature by performing a convolution operation on the embedding vector on which the positional encoding has been performed,
Extracting global features by performing self-attention based on the local features
mechanical reading device.
11. The method of claim 10,
The processor is
Performing the at least one encoding using a neural network including a residual block
mechanical reading device.
11. The method of claim 10,
The processor is
calculating a score for the sentence included in the context and the question by performing encoding based on the encoded embedding vector
mechanical reading device.
11. The method of claim 10,
The processor is
Calculate the probability of the starting word of the correct answer,
Calculate the probability of the end word of the correct answer,
predicting the correct answer based on the probability of the start word and the probability of the end word
mechanical reading device.
11. The method of claim 10,
The processor is
Correcting the predicted correct answer based on the score for the sentence included in the context and the question
mechanical reading device.

Images