KR102292040B1 - System and method for extracting knowledge based on machine reading - Google Patents

Abstract

A knowledge extraction method for extracting knowledge from a document includes, according to an exemplary embodiment of the present invention, extracting an entity from an input document, based on an attribute included in an attribute list of the entity, including the entity generating a query from an input document, extracting an answer to the query from the input document based on machine reading comprehension, and generating a knowledge instance from entities, attributes and answers based on the format of the knowledgebase may include the step of

Description

SYSTEM AND METHOD FOR EXTRACTING KNOWLEDGE BASED ON MACHINE READING

The technical idea of the present invention relates to knowledge extraction, and more particularly, to a system and method for extracting knowledge based on machine reading comprehension.

The present invention is derived from research conducted by IPL Co., Ltd. and jointly researched by Saltlux Co., Ltd. as part of the Robot Industry Core Technology Development Project-Artificial Intelligence Convergence Robot System Technology of the Ministry of Trade, Industry and Energy. [Research period: 2019.01.01~2019.12.31, specialized research institute: Korea Institute of Industrial Technology Evaluation and Planning, research project name: home social robot and service development system, project identification number: 10077633]

A knowledgebase that stores knowledge in a computer-recognizable form enables various services utilizing the stored knowledge. For example, a question and answering system that provides an answer to a user's query may provide an answer by referring to knowledge stored in a knowledge base after analyzing the user's query. The quality and scope of a service using such a knowledge base may depend on the accuracy and amount of knowledge included in the knowledge base, and accordingly, it may be important to secure accurate knowledge for reinforcing the knowledge base.

The technical idea of the present invention provides a system and method for automatically extracting accurate knowledge based on machine reading.

In order to achieve the above object, according to an aspect of the technical idea of the present invention, a knowledge extraction system for extracting knowledge from a document includes an entity extractor for extracting an entity from an input document, and an entity attribute list Based on the properties included in the query generator configured to query including the object, a machine reading comprehension engine that extracts the answer to the query from the input document, and based on the format of the knowledge base, the object, property and It may include a knowledge generator that generates a knowledge instance from the answer.

According to an exemplary embodiment of the present invention, the entity extraction unit may extract the entity from the title of the input document.

According to an exemplary embodiment of the present invention, the query generating unit sequentially selects each of a plurality of attributes included in the attribute list, and generates a first word vector corresponding to the entity and a second word vector corresponding to the selected attribute. a preprocessor that is trained to generate sample queries according to sample objects and sample properties, a first deep learning network to generate a third word vector corresponding to the query from a first word vector and a second word vector, and a second It may include a post-processing unit that generates a query from a three-word vector.

According to an exemplary embodiment of the present invention, the machine reading engine is trained to output sample answers according to a natural language processing unit generating input data by natural language processing an input document and query, natural language processing sample documents and sample queries, and , a second deep learning network that generates output data from the input data, and an answer generator that generates an answer based on the output of the second deep learning network, wherein the output data includes a correct answer in the input document , the position of the correct answer, and may include at least one of the reliability of the correct answer.

According to an exemplary embodiment of the present invention, the answer generating unit may determine an answer extraction failure when the correct answer is not included in the input document or the reliability is less than a predefined threshold.

According to an exemplary embodiment of the present invention, the query generator may generate a query based on the next attribute included in the attribute list in response to the failure to extract the answer.

According to an exemplary embodiment of the present invention, the knowledge base may include a triple including a subject, a predicate, and an object, and the knowledge generating unit includes an object, an attribute, and an answer. A triple including , as a subject, a predicate, and an object can be created as a knowledge instance.

According to an exemplary embodiment of the present invention, the knowledge extraction system retrieves the document based on at least a portion of the knowledge instance over the network, provides the retrieved document and query to the machine reading engine, and based on the output of the machine reading engine to further include a knowledge verification unit that verifies the knowledge instance.

According to an exemplary embodiment of the present invention, the knowledge generating unit includes: a candidate instance generating unit that generates a candidate knowledge instance by post-processing an entity, an attribute, and an answer based on the format of the knowledge instance included in the knowledge base, and the candidate knowledge instance may include an instance comparison unit for generating a knowledge instance by comparing .

According to an aspect of the technical concept of the present invention, a knowledge extraction method for extracting knowledge from a document includes the steps of extracting an entity from an input document, based on an attribute included in the attribute list of the entity, including the entity generating a query from an input document that asks, based on machine reading comprehension, extracting an answer to the query from the input document, and based on the format of the knowledgebase, generating a knowledge instance from objects, attributes, and answers. It may include the step of generating.

According to the system and method according to the technical idea of the present invention, knowledge can be easily extracted from a document based on a technically mature machine reading.

In addition, according to the system and method according to the technical idea of the present invention, extracted knowledge can be verified, and finally, accurate knowledge can be extracted.

In addition, according to the system and method according to the technical idea of the present invention, the knowledge base can be effectively reinforced due to easily and accurately extracted knowledge, and accordingly, the quality of the services based on the knowledge base can be improved and the scope can be expanded. can

The effects obtainable in the exemplary embodiments of the present invention are not limited to the above-mentioned effects, and other effects not mentioned are common knowledge in the technical field to which exemplary embodiments of the present invention pertain from the following description. It can be clearly derived and understood by those who have That is, unintended effects of practicing the exemplary embodiments of the present invention may also be derived by those of ordinary skill in the art from the exemplary embodiments of the present invention.

1 is a block diagram showing a knowledge extraction system and its input/output according to an exemplary embodiment of the present invention.
2 is a diagram showing examples of an input document and an object extracted from the input document according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating an example of a query generator according to an exemplary embodiment of the present invention.
4 is a diagram illustrating an example of an operation of generating a query according to an exemplary embodiment of the present invention.
5 is a block diagram illustrating an example of a machine reading comprehension engine according to an exemplary embodiment of the present invention.
6 is a diagram showing an example of the operation of the natural language processing unit of FIG. 5 according to an exemplary embodiment of the present invention.
7 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention.
8 is a block diagram illustrating an example of a knowledge generating unit according to an exemplary embodiment of the present invention.
9 is a diagram illustrating an example of an operation of a knowledge generating unit according to an exemplary embodiment of the present invention.
10 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention.
11 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art. Since the present invention may have various changes and may have various forms, specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to the specific disclosed form, it should be understood to include all modifications, equivalents and substitutes included in the spirit and scope of the present invention. In describing each figure, like reference numerals are used for like elements. In the accompanying drawings, the dimensions of the structures are enlarged or reduced than the actual size for clarity of the present invention.

The terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as “comprise” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that it does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with the meanings in the context of the related art, and unless explicitly defined in the present application, they should be interpreted in an ideal or excessively formal meaning. doesn't happen

In the drawings and description below, a component indicated or described as one block may be a hardware block or a software block. For example, each of the components may be an independent hardware block that sends and receives signals with each other, or may be a software block executed by one processor. Also, as used herein, “system” or “database” may refer to a computing system including at least one processor and memory accessed by the processor.

1 is a block diagram showing a knowledge extraction system and its input/output according to an exemplary embodiment of the present invention. As shown in FIG. 1 , the knowledge extraction system 100 may receive an input document DIN and an attribute list PL, and communicate with a knowledgebase 200 , and a network 300 . can connect to

The knowledge extraction system 100 may extract knowledge from the input document DIN based on machine reading comprehension, as will be described later with reference to the drawings, and use the extracted knowledge in the knowledge base 200 ) can be reinforced. As shown in FIG. 1 , the knowledge extraction system 100 includes an object extraction unit 110 , a query generation unit 130 , a machine reading comprehension engine 150 , a knowledge generation unit 170 , and a knowledge verification unit 190 . may include In some embodiments, unlike as shown in FIG. 1 , the knowledge extraction system 100 , instead of communicating with the knowledgebase 200 , transfers the extracted knowledge to another system, such as a system for augmenting the knowledgebase 200 . may provide. In addition, in some embodiments, differently as shown in FIG. 1 , the knowledge verification unit 190 may be external to the knowledge extraction system 100 , and the knowledge extraction system 100 is configured with an external knowledge verification unit ( Verification of knowledge may also be performed by communicating (eg, over a network). Also, in some embodiments, different from that shown in FIG. 1 , the machine reading engine 150 may be external to the knowledge extraction system 100 , and the knowledge extraction system 100 may be configured with an external machine reading engine. It is also possible to provide a query (QUE) or receive an answer (ANS) by communicating (eg, via a network).

The knowledge base 200 may include structured knowledge based on an ontology, that is, knowledge instances. An ontology is an expression of real or human-recognizable things in a form that can be handled by a computer, and ontology components, for example, may include entities, classes, properties, values, etc. . Additionally, the ontology components may further include a relation, a function term, a restriction, a rule, an event, and the like. Specific information about an entity, ie, knowledge, may be referred to as a knowledge instance (or simply an instance), and the knowledge base 200 may store vast amounts of knowledge instances. In some embodiments, the knowledge base 200 may include knowledge instances expressed based on a Resource Description Framework (RDF), and the knowledge instances may be expressed as a triple. The knowledge base 200 may return a knowledge instance, that is, a triple in response to a query, for example, a SPARQL Protocol and RDF Query Language (SPARQL) query. A triple may consist of "subject (S)-predicate (P)-object (O)", and a knowledge instance can be the subject as well as the object of the triple, and can be a predicate. For example, the knowledge base 200 may have a triple of “Sun-Shin Yi (S)-Nationality (P)-Joseon (O)” as a knowledge instance including the entity “Sun-Shin Yi”.

The entity extraction unit 110 may extract the entity ENT from the input document DIN. The entity ENT is a target of knowledge and may constitute a knowledge instance included in the knowledge base 200 . The input document DIN may refer to an entity ENT and any data including information about the entity ENT. For example, the input document (DIN) may be a document provided from an encyclopedia service (or server) such as Wikipedia (wikipedia.org), articles provided from a newspaper or portal, etc., and a social network service ( They may be documents created on Social Network Service (SNS). Also, unlike the above-described examples obtained through a network such as the Internet, the input document DIN may be data stored in a local storage. The input document DIN may be structured data or unstructured data. As shown in FIG. 1 , the input document DIN may be provided not only to the entity extraction unit 110 , but also to the machine reading engine 150 . An example of the input document DIN will be described later with reference to FIG. 2 .

The object extraction unit 110 may extract an object from the input document DIN in an arbitrary manner. In some embodiments, the entity extraction unit 110 may perform morpheme analysis on text included in the input document DIN, and extract one of words corresponding to a noun among morphemes as an entity ENT. can do. In some embodiments, the entity extraction unit 110 may filter words corresponding to nouns by referring to entities and/or knowledge instances included in the knowledge base 200 , and select the filtered word as an entity (ENT). can be extracted as In some embodiments, the entity extraction unit 110 may extract the entity through sentence analysis included in the text of the input document DIN based on dependency parsing and/or Semantic Role Labeling (SRL). have. In some embodiments, the entity extraction unit 110 may extract the entity ENT from the title of the input document DIN, as shown in FIG. 2 .

The query generator 130 may receive the entity ENT extracted from the entity extractor 110 and receive the attribute list PL. The attribute list PL may define attributes each class has in the knowledge base 200 . For example, in the attribute list (PL), the class "person" may define attributes such as "age", "gender", "name", "birth", "occupation", "nationality", "spouse", etc. , and thus a knowledge instance including “Sun-Shin Yi” as an entity belonging to the class “person” is an attribute of one of the aforementioned attributes and another entity corresponding to the attribute (eg, an entity corresponding to occupation, nationality or spouse) or It may include a value (eg, an entity corresponding to an age, gender, name, or birth). In some embodiments, the query generating unit 130 may obtain a class to which the entity ENT provided from the entity extraction unit 110 belongs by referring to the input document DIN and/or the knowledge base 200 , , an attribute group corresponding to the acquired class may be extracted from the attribute list PL. The query generator 130 may generate a query QUE based on the entity ENT and one property PRO among a plurality of properties included in the property group. 1 , the query generating unit 130 may provide a query QUE to the machine reading comprehension engine 150 , and may provide the attribute PRO to the knowledge generating unit 170 . As will be described later, the query (QUE) provided to the machine reading engine 150 may be used to generate an answer (ANS) or may be used to verify the knowledge instance (INS) by the knowledge verification unit 190 . In some embodiments, the query generator 130 may receive the attribute list PL from the knowledge base 200 . Examples of the query generator 130 will be described later with reference to FIGS. 3 and 4 .

The machine reading engine 150 may receive the input document DIN, receive the query QUE from the query generator 130, and answer the query QUE from the input document DIN (ANS) ) can be extracted. Machine reading can refer to machines reading texts on various topics and understanding their meaning. As shown in FIG. 1 , the machine reading engine 150 may include a deep learning network 154 , which provides sample answers according to sample documents and sample queries. It may be in a state that has been learned to output the values. The machine reading engine 150 may obtain an answer (ANS) by providing an input document (DIN) including a query (QUE) and an answer to the query to the deep learning network 154 . For example, Stanford Question Answering Dataset (SQuAD) may provide a data set for machine reading in an English environment, and Korea Question Answering Dataset (KorQuAD) may provide a data set for machine reading in a Korean environment. Machine reading provides very good performance that exceeds human ability. For example, the human score presented by KorQuAD is EM (Exact Matching) 80.17 and F1 score 91.2, while the highest performance is EM 86.84 and F1 score 94.75. reaches to The knowledge extraction system 100 may accurately and easily extract knowledge from the input document DIN based on the mature machine reading comprehension.

1 , the machine reading engine 150 may provide an answer (ANS) to the knowledge generating unit 170 . In some embodiments, the machine reading engine 150 may be used to verify the extracted knowledge, and for this purpose, as shown in FIG. 1 , may communicate with the knowledge verification unit 190 . In some embodiments, the machine reading engine 150 may generate additional information related to an input document (DIN) and a query (QUE) as well as an answer (ANS), and the knowledge generator 170 and/or knowledge verification Unit 190 may use additional information. Examples of the machine reading engine 150 will be described below with reference to FIGS. 5 , 6 and 7 .

The knowledge generating unit 170 may receive the entity ENT from the entity extracting unit 110 , the attribute PRO from the query generating unit 130 , and an answer ( ANS) can be received. The knowledge generator 170 may generate a knowledge instance INS (eg, triple) from the entity ENT, the attribute PRO, and the answer ANS. For example, the knowledge generating unit 170 may post-process the entity (ENT), the attribute (PRO), and the answer (ANS) according to the format of the knowledge base 200 , and the entity (ENT), the attribute (PRO) And by extracting at least one identifier of the answer (ANS) from the knowledge base 200 may generate the knowledge instance (INS). In addition, the knowledge generating unit 170 may verify the knowledge instance INS by providing the knowledge instance INS to the knowledge verification unit 190 , and reinforce the verified knowledge instance INS to the knowledge base 200 . can do. Examples of the knowledge generating unit 170 will be described with reference to FIGS. 8 and 9 .

The knowledge verification unit 190 may receive the knowledge instance INS from the knowledge generation unit 170 , and provide the verification result to the knowledge generation unit 170 by verifying the knowledge instance INS. As shown in FIG. 1 , the knowledge verification unit 190 may connect to the network 300 and communicate with the machine reading engine 150 . The network 300 may include a local network as well as a wide area network such as the Internet, and the knowledge verification unit 190 communicates with other systems connected to the network 300 to verify the knowledge instance INS. Data (eg, documents) may be obtained. The knowledge verification unit 190 may provide the obtained data to the machine reading engine 150 , and based on the information provided by the machine reading engine 150 in response thereto, whether the verification of the knowledge instance INS succeeds. can be determined. An example of the knowledge verification unit 190 will be described later with reference to FIG. 10 .

Fig. 2 is a diagram showing examples of an input document and an object extracted from the input document according to an exemplary embodiment of the present invention. As described above with reference to FIG. 1 , the entity extraction unit 110 may extract the entity ENT' from the input document DIN'. Hereinafter, FIG. 2 will be described with reference to FIG. 1 .

Referring to FIG. 2 , the input document DIN' may include information about the person "Son Heung-min". For example, the input document DIN' may be provided from an encyclopedia service such as Wikipedia, Namu.wiki, and the like. As shown in FIG. 2 , the input document DIN' may include "Son Heung-Min" indicating the name of the person as a title, and includes the text "Son Heung-Min is from Korea..." can do. The contents of the input document DIN' may be updated, and accordingly, even after the knowledge extraction from the input document DIN' is completed, the object extraction unit 110 may extract the same or different objects from the updated input document again. have. Accordingly, the entity ENT' extracted by the entity extraction unit 110 may not be limited to the entities included in the knowledge base 200 , and as a result, information on entities included in the knowledge base 200 may be retrieved. More effective knowledge extension than methods of generating knowledge by retrieving from a document can be achieved.

In some embodiments, the entity extraction unit 110 may extract the entity ENT' from the title of the input document DIN'. For example, the input document DIN' may include information about the title, and accordingly, various knowledge instances including the entity extracted from the title may be extracted from the input document DIN'. Accordingly, as shown in FIG. 2 , the entity extraction unit 110 may extract “Heung-Min Son” as the entity ENT' from the input document DIN'.

Fig. 3 is a block diagram showing an example of a query generating unit according to an exemplary embodiment of the present invention, and Fig. 4 is a diagram showing an example of an operation for generating a query according to an exemplary embodiment of the present invention. As described above with reference to FIG. 1 , the query generator 130 ′ of FIG. 3 may receive the entity ENT and the attribute list PL, and may generate a query QUE. Hereinafter, FIGS. 3 and 4 will be described with reference to FIG. 1 .

Referring to FIG. 3 , the query generator 130 ′ may include a preprocessor 132 , a deep learning network 134 , and a postprocessor 136 . The preprocessor 132 may receive an entity ENT and a property PRO. In some embodiments, the preprocessor 132 may sequentially select each of the attributes included in the attribute group corresponding to the class of the entity ENT. For example, when it is determined that the generation of the knowledge instance INS is completed by the knowledge generation unit 170 of FIG. 1 or that the generation of the knowledge instance INS has failed, the preprocessor 132 determines that the same entity ENT ) properties, that is, properties different from the existing properties among properties included in the property group may be obtained from the property list PL.

The preprocessor 132 may generate a first word vector V1 and a second word vector V2 corresponding to the entity ENT and the property PRO with reference to the word vector model 400 . The word vector model 400 may refer to a multidimensional space in which a word (or token, word, etc.) having a meaning is expressed in one coordinate, that is, a word vector, or a system including word vectors and updating the word vectors. . Semantically similar words may be arranged adjacently in a multidimensional space, and thus word vectors corresponding to semantically similar words may have similar values. Accordingly, the first word vector V1 may have coordinate values corresponding to the entity ENT, the second word vector V2 may have coordinate values corresponding to the property PRO, and the first word Based on the vector V1 and the second word vector V2, the deep learning network 134 may perform mathematical operations. The word vector model 400 may be included in the knowledge extraction system 100 of FIG. 1 , and the query generator 130 ′ may access the word vector model 400 external to the knowledge extraction system 100 . .

The deep learning network 134 may receive the first word vector V1 and the second word vector V2 from the preprocessor 132 , and may output the third word vector V3 . The deep learning network 134 may be in a learned state, such as based on reinforcement learning (RL), to generate sample queries according to sample objects and sample properties, and may have any structure. For example, the deep learning network 134 may be implemented as hardware or a combination of hardware and software, and may be referred to as an artificial neural network (ANN). The deep learning network 134 is, as a non-limiting example, a Deep Neural Network (DNN), a Convolution Neural Network (CNN), a Recurrent Neural Network (RNN), and a Restricted Boltzmann Machine (Restricted Boltzmann). Machine (RBM), Deep Belief Network (DBN), and Deep Q-Network. Accordingly, the deep learning network 134 may generate the third word vector V3 for generating the query QUE from the first word vector V1 and the second word vector V2. In this specification, the deep learning network 134 included in the query generator 130 ′ may be referred to as a first deep learning network.

The post-processing unit 136 may generate a query QUE from the third word vector V3 with reference to the word vector model 400 . For example, as shown in FIG. 4 , the deep learning network 134 generates a first word vector V1 and a second A word vector V2 may be received. The deep learning network 134 may generate the third word vector V3 in response to the first word vector V1 and the second word vector V2, and the post-processing unit 136 may generate the third word vector V3. ) can be obtained from a series of words W1 to W4, and by combining the series of words W1 to W4, "What is Heung-Min Son's job?" can be generated as a query (QUE).

5 is a block diagram illustrating an example of a machine reading comprehension engine according to an exemplary embodiment of the present invention, and FIG. 6 is a diagram illustrating an example of the operation of the natural language processing unit of FIG. 5 according to an exemplary embodiment of the present invention. As described above with reference to FIG. 1 , the machine reading engine 150 ′ of FIG. 5 may receive a query (QUE) and an input document (DIN), and may generate an answer (ANS). The machine reading engine 150' may generate an answer (ANS) based on machine reading in any manner that is currently used or will appear later, and is not limited to the content described with reference to FIG. 5 below. It is noted that no Hereinafter, FIGS. 5 and 6 will be described with reference to FIG. 1 .

Referring to FIG. 5 , the machine reading engine 150 ′ may include a natural language processing unit 152 , a deep learning network 154 , and an answer generating unit 156 . The natural language processing unit 152 may generate the input data IN by natural language processing the input document DIN and the query QUE. In some embodiments, the natural language processing unit 152 may perform morphological analysis on the input document DIN and the query QUE. For example, as shown in FIG. 6 , the natural language processing unit 152 may receive “What is Heung-Min Son’s job?” as a query QUE, and through morphological analysis of the query QUE, the The first input data IN1 as shown in FIG. In the first input data IN1, a diagonal line (/) and an alphabet following a word indicate morphemes of the corresponding word. For example, in the first input data IN1, “Heung-Min Son/NNP” may indicate that the word “Heung-Min Son” is a proper noun, and the word “job/NNG” may indicate that “job” is a general noun. Also, "of/JKG" may indicate that the word "of" is a possessive proposition. Similarly, the natural language processing unit 152 may generate the second input data IN2 as shown in FIG. 6 through morphological analysis of the input document DIN, and the first input data IN1 and the second input Input data IN' including data IN2 may be generated. In some embodiments, the natural language processing unit 152 may be external to the knowledge extraction system 100 , different from that shown in FIG. 5 , and the machine reading engine 150 ′ may include input documents ( DIN) and query (QUE) may receive input data (IN).

The deep learning network 154 may receive the input data IN from the natural language processing unit 152 and may generate the output data OUT in response to the input data IN. The deep learning network 154 may be in a state that has been trained to output sample answers according to sample documents and sample queries, and accordingly, the contents of the query (QUE) and the input document (DIN) embedded in the input data (IN). It is possible to generate output data OUT with the contents of the answer (ANS) embedded therein. As described above, knowledge may be extracted based on machine reading in consideration of context, and accordingly, more accurate knowledge may be extracted than methods of generating knowledge by searching for words in a document. The output data OUT may further include additional information as well as content corresponding to the answer ANS. For example, the output data OUT may include information indicating the location of the answer ANS in the input document DIN, while whether the answer ANS is included in the input document DIN as a correct answer, the answer ( ANS) may further include reliability and the like. In this specification, the deep learning network 154 included in the machine reading engine 150 ′ may be referred to as a second deep learning network.

The answer generator 156 may receive the output data OUT from the deep learning network 154 and generate the answer ANS based on the output data OUT. For example, the output data OUT may include information indicating the position of “soccer/NNG player/NNG” in the second input data IN2 of FIG. 6 , and the answer generating unit 156 may include the output data ( OUT) based on the location information included in the "soccer player" can be generated as the answer (ANS). Also, the answer generator 156 may generate a determination result DET indicating whether answer extraction is successful based on additional information included in the output data OUT. In some embodiments, the determination result DET is provided to other components of the knowledge extraction system 100 , such as at least one of the entity extraction unit 110 , the query generation unit 130 , and the knowledge generation unit 170 . can be An example of the operation of the answer generating unit 156 will be described later with reference to FIG. 7 .

7 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention. Specifically, the flowchart of FIG. 7 shows a method of generating an answer for knowledge extraction based on the result of machine reading. In some embodiments, the method of FIG. 7 may be performed by the answer generator 156 of FIG. 5 and may be referred to as an operation method of the answer generator 156 . As shown in FIG. 7 , the method of FIG. 7 may include a plurality of steps S72 , S74 , S76 and S78 , which will be described below with reference to FIG. 5 .

In step S72, an operation of determining whether a correct answer is included may be performed. For example, the deep learning network 154 may generate output data OUT including information indicating whether the input document DIN contains a correct answer to the query QUE, and the answer generating unit ( 156 may determine whether a correct answer is included based on information included in the output data OUT. 7 , when it is determined that the input document DIN does not contain the correct answer, step S76 may be performed subsequently, while when it is determined that the input document DIN contains the correct answer, step S74 This can be done subsequently.

In step S74, an operation of comparing the reliability of the answer (ANS) with a predefined threshold may be performed. For example, the deep learning network 154 may generate output data OUT including the reliability of the answer ANS together with the location information of the answer ANS, and the answer generator 156 may generate the output data ( OUT) can be compared with a threshold value. As shown in FIG. 7 , if the reliability is less than the threshold, step S76 may be performed subsequently, while if the reliability is greater than or equal to the threshold, step S78 may be performed subsequently.

If it is determined that the correct answer is not included in the input document DIN or the reliability of the answer ANS is less than a threshold, an operation of determining the extraction failure of the answer may be performed in step S76. For example, the answer generating unit 156 may generate a determination result (DET) indicating an extraction failure of an answer, and provide the determination result (DET) to other components included in the knowledge extraction system 100 . can Components included in the knowledge extraction system 100 may perform an operation for extracting the next knowledge in response to the determination result DET indicating the extraction failure. For example, the object extraction unit 110 may extract an object different from the previous object from the input document DIN, or may receive a different input document DIN. The query generator 130 may generate a query QUE by acquiring an attribute different from the previous attribute from the attribute list PL. The knowledge generator 170 may stop generating a knowledge instance for the current entity ENT and the property PRO.

On the other hand, if it is determined that the correct answer is included in the input document DIN and the reliability of the answer ANS is equal to or greater than the threshold, the operation of generating the answer ANS may be performed in step S78. For example, as described above with reference to FIGS. 5 and 6 , the answer generator 156 generates an answer ANS from the input document DIN based on the location information of the correct answer included in the output data OUT. can be extracted.

Fig. 8 is a block diagram showing an example of a knowledge generating unit according to an exemplary embodiment of the present invention, and Fig. 9 is a diagram showing an example of an operation of the knowledge generating unit according to an exemplary embodiment of the present invention. As described above with reference to FIG. 1 , the knowledge generating unit 170 ′ of FIG. 8 may generate a knowledge instance INS (eg, triple) from an entity ENT, a property PRO, and an answer ANS. have. Hereinafter, FIGS. 8 and 9 will be described with reference to FIG. 1 .

Referring to FIG. 8 , the knowledge generating unit 170 ′ may include a candidate instance generating unit 172 and an instance comparing unit 174 . The candidate instance generator 172 may receive the entity ENT, the attribute PRO, and the answer ANS, and may generate the candidate knowledge instance CAN. In some embodiments, the candidate instance generating unit 172 may generate an entity (ENT), an attribute based on the format of the knowledge base 200 , that is, the format of knowledge instances (eg, triples) included in the knowledge base 200 . (PRO) and answer (ANS) can be post-processed. For example, when "Sun-Shin Yi", "Birthday" and "April 28, 1545" are received as the entity (ENT), attribute (PRO) and answer (ANS), the candidate instance creation unit 172 is the knowledge base In (200), based on the format "YYYY-MM-DD" for representing the date, the answer (ANS) "April 28, 1545" may be converted into "1545-04-28".

The instance comparison unit 174 may receive the candidate knowledge instance CAN and generate the knowledge instance INS based on the knowledge instances included in the knowledge base 200 . In some embodiments, the instance comparison unit 174 is a "similarity calculation unit" and the "similarity calculation unit" described in Korean Patent Application No. 10-2018-0151222 filed by the same applicant as the present application and incorporated herein by reference in its entirety. Similarly, the degree of similarity between the candidate knowledge instance CAN and the knowledge instances included in the knowledge base 200 may be calculated. The instance comparison unit 174 may detect entities in the knowledge base 200 corresponding to the subject, predicate, and object included in the candidate instance based on the calculated similarity, and based on the detection result, the subject, predicate, and object A knowledge instance (INS) may be generated by extracting a corresponding identifier, for example, a Uniform Resource Identifier (URI) from the knowledge base 200 .

Referring to FIG. 9 , when "Son Heung-min", "job" and "soccer player" are received as an entity (ENT'), a property (PRO'), and an answer (ANS'), the candidate instance creation unit 172 is A candidate knowledge instance (CAN') corresponding to "Heung-Min Son-Occupation-Soccer Player" may be created. In addition, the knowledge base 200 includes all of "Son Heung-Min", "Occupation" and "Soccer player", whereas the knowledge that the occupation of the character "Son Heung-Min" is a soccer player, that is, knowledge instances may not be included. have. In this case, the instance comparison unit 174 may determine knowledge instances similar to the candidate knowledge instance CAN' among the knowledge instances included in the knowledge base 200, for example, a knowledge instance having the same or similar name, and a candidate knowledge instance. A knowledge instance including entities of a similar type (eg, class) to (CAN') may be extracted. The instance comparison unit 174 determines the identifiers “addr:0001”, “addr:1012”, and “addr:0156” respectively corresponding to “Heung-Min Son”, “Occupation” and “Soccer player” based on the extracted knowledge instances. ", and thus create "addr:0001-addr:1012-addr:0156" as the knowledge instance INS'. As such, the knowledge instance INS may define a relationship between entities or between entities and values. Accordingly, the knowledge base 200 including knowledge instances may be referred to as including a knowledge graph.

Unlike the example of FIG. 9 , when at least one of a subject, a predicate, and an object of a candidate knowledge instance (CAN) is an object that is not included in the knowledge base 200 , the instance comparison unit 174 is included in the knowledge base 200 . A knowledge instance (INS) may be created by generating a new identifier that has not been identified. Accordingly, the instance comparison unit 174 may generate the knowledge instance INS' including the identifiers. In addition, when a knowledge instance very similar to the candidate knowledge instance CAN' is searched for in the knowledge base 200, the instance comparison unit 174 may determine that the knowledge extracted from the input document DIN is duplicate knowledge, It is also possible to stop the creation of knowledge instances. In some embodiments, the instance comparison unit 174 may reinforce the knowledge base 200 by adding non-overlapping knowledge instances INS to the knowledge base 200 .

10 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention. Specifically, the flowchart of FIG. 10 shows a method of verifying a knowledge instance corresponding to the extracted knowledge. In some embodiments, the method of FIG. 10 may be performed by the knowledge verification unit 190 of FIG. 1 , and may also be referred to as an operation method of the knowledge verification unit 190 . As shown in Fig. 10, the method of Fig. 10 may include a plurality of steps (S101, S102, S103, S104, S105, S106), below which Fig. 10 will be described with reference to Fig. 1, Among the descriptions of FIG. 10 , content overlapping with those of FIG. 7 will be omitted.

In step S101, an operation of searching for a document through the network 300 based on the knowledge instance INS may be performed. For example, the knowledge verification unit 190 may include a document including at least one of the names of the knowledge instance (INS), for example, “Heung-Min Son”, “Occupation” and “Soccer player” in the system connected to the network 300 . can be searched in In some embodiments, similar to sources providing an input document DIN, the knowledge verification unit 190 may search for a document by accessing systems that provide documents including various information.

In step S102, an operation of providing the retrieved document and the query QUE to the machine reading engine 150 may be performed. For example, the knowledge verification unit 190 may provide the document retrieved in step S101 to the machine reading engine 150 , while returning the query QUE used to generate the knowledge instance INS to the machine reading engine 150 . ) or have the query generating unit 130 provide it. Accordingly, the machine reading engine 150 may find the correct answer to the query QUE in the searched document.

In step S103, an operation of determining whether a correct answer is included may be performed. For example, the knowledge verification unit 190 may directly receive the output data OUT of FIG. 5 , and may determine whether a correct answer is included based on the output data OUT. As another example, the knowledge verification unit 190 may determine whether a correct answer is included based on the determination result DET provided by the answer generation unit 156 of FIG. 5 . 10 , when it is determined that the searched document does not contain the correct answer, step S105 may be performed subsequently, while if it is determined that the searched document contains the correct answer, step S106 may be performed subsequently have.

In step S104, an operation of comparing the reliability of the answer extracted from the searched document with a predefined threshold may be performed. In some embodiments, the threshold of FIG. 10 may be different from the threshold of FIG. 7 , eg, the threshold of FIG. 10 may be higher than the threshold of FIG. 7 . As shown in FIG. 10 , if the reliability is less than the threshold, step S105 may be performed subsequently, while if the reliability is greater than or equal to the threshold, step S106 may be performed subsequently.

When a correct answer is not included in the searched document or the reliability of an answer extracted from the searched document is low, it may be determined in step S105 that the verification of the knowledge instance INS fails. For example, when it is determined that the verification fails, the knowledge verification unit 190 may re-verify the knowledge instance INS using another searched document. When the verification of the knowledge instance INS fails using all of the found documents or a predefined amount of documents, the knowledge verification unit 190 may finally determine the verification failure of the knowledge instance INS, The knowledge generating unit 170 may be notified.

On the other hand, when a correct answer is included in the retrieved document or the reliability of an answer extracted from the retrieved document is high, verification success of the knowledge instance INS may be determined in step S106. For example, when verification success is determined using one searched document, the knowledge verification unit 190 may finally determine verification success of the knowledge instance INS. As another example, the knowledge verification unit 190 may finally determine the verification success of the knowledge instance INS when verification success is determined using a predefined number or ratio of retrieved documents.

11 is a flowchart illustrating a method for knowledge extraction according to an exemplary embodiment of the present invention. In some embodiments, the method of FIG. 11 may be performed by the knowledge extraction system 100 of FIG. 1 . 11 , the method of FIG. 11 may include a plurality of steps S200 , S400 , S600 , and S800 , and FIG. 11 will be described below with reference to FIG. 1 .

In step S200, an operation of extracting the entity ENT from the input document DIN may be performed. For example, the input document DIN may include a title, and the entity extraction unit 110 may extract the entity ENT from the title of the input document DIN.

In step S400, an operation of generating a query (QUE) including the entity (ENT) may be performed. For example, the query generator 130 may select one of the attributes that the class of the entity ENT may have from the attribute list PL. The query generator 130 may generate word vectors (eg, V1 and V2 in FIG. 3 ) from the selected attribute PRO and entity ENT with reference to the word vector model, and a deep learning network (eg, FIG. 3 ) A query QUE may be generated based on a word vector (eg, V3 of FIG. 3 ) obtained by providing the word vectors to 134 of 3 .

In step S600, an operation of extracting an answer (ANS) from the input document (DIN) based on the machine reading may be performed. For example, the machine reading engine 150 may extract an answer (ANS) corresponding to the query (QUE) from the input document (DIN) using the deep learning network (154).

In operation S800 , an operation of generating a knowledge instance INS based on the entity ENT, the attribute PRO, and the answer ANS may be performed. For example, the knowledge generator 170 may generate a candidate knowledge instance (eg, CAN of FIG. 8 ) based on the entity ENT, the attribute PRO, and the answer ANS. The knowledge generating unit 170 may generate a knowledge instance INS from the candidate knowledge instance by comparing the candidate knowledge instance with the knowledge instances included in the knowledge base 200 , or a knowledge base based on the knowledge instance INS. (200) may determine whether reinforcement. In some embodiments, following step S800, an operation of verifying the knowledge instance INS generated in step S800 may be further performed.

Exemplary embodiments have been disclosed in the drawings and specification as described above. Although the embodiments have been described using specific terms in the present specification, these are used only for the purpose of explaining the technical idea of the present invention and not used to limit the meaning or the scope of the present invention described in the claims. . Therefore, it will be understood by those skilled in the art that various modifications and equivalent other embodiments are possible therefrom. Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

Claims (10)

A knowledge extraction system for extracting knowledge from a document, comprising:
an entity extraction unit configured to extract an entity from the input document;
a query generator configured to generate a query including the entity based on the attribute included in the attribute list of the entity;
a machine reading comprehension engine configured to extract an answer to the query from the input document; and
a knowledge generating unit configured to generate a knowledge instance from the entity, the attribute, and the answer based on the format of the knowledge base;
The query generator,
a preprocessor configured to sequentially select each of a plurality of properties included in the property list, and generate a first word vector corresponding to the entity and a second word vector corresponding to the selected property;
a first deep learning network trained to generate sample queries according to sample objects and sample properties, the first deep learning network to generate a third word vector corresponding to the query from the first word vector and the second word vector; and
and a post-processing unit configured to generate the query from the third word vector. The method according to claim 1,
and the entity extraction unit is configured to extract the entity from the title of the input document. delete The method according to claim 1,
The machine reading engine is
a natural language processing unit configured to generate input data by natural language processing the input document and the query;
a second deep learning network trained to output sample answers according to natural language processed sample documents and sample queries, and configured to generate output data from the input data; and
an answer generator configured to generate the answer based on the output of the second deep learning network,
The output data is a knowledge extraction system, characterized in that it includes at least one of whether the correct answer is included in the input document, the position of the correct answer, and the reliability of the correct answer. 5. The method according to claim 4,
The answer generating unit is configured to determine the extraction failure of the answer when the correct answer is not included in the input document or the reliability is less than a predefined threshold. 6. The method of claim 5,
The knowledge extraction system according to claim 1, wherein the query generating unit is configured to generate the query based on a next attribute included in the attribute list in response to the failure to extract the answer. The method according to claim 1,
The knowledge base includes a triple including a subject, a predicate, and an object,
The knowledge generating unit is configured to generate a triple including the entity, the attribute, and the answer as the subject, the predicate, and the object as the knowledge instance. The method according to claim 1,
knowledge validation configured to retrieve a document based on at least a portion of the knowledge instance over a network, provide the retrieved document and the query to the machine reading engine, and validate the knowledge instance based on an output of the machine reading engine A knowledge extraction system further comprising wealth. The method according to claim 1,
The knowledge generating unit,
a candidate instance generating unit configured to generate a candidate knowledge instance by post-processing the entity, the attribute, and the answer based on the type of the knowledge instance included in the knowledge base; and
and an instance comparison unit configured to generate the knowledge instance by comparing the candidate knowledge instance with knowledge instances included in the knowledge base, and to reinforce the knowledge base. A knowledge extraction method for extracting knowledge from a document, comprising:
extracting an entity from the input document;
generating a query including the entity based on attributes included in the attribute list of the entity;
extracting an answer of the query from the input document based on a machine reading comprehension; and
generating a knowledge instance from the entity, the attribute and the answer based on the format of the knowledgebase;
Creating the query includes:
sequentially selecting each of a plurality of attributes included in the attribute list;
generating a first word vector corresponding to the entity and a second word vector corresponding to the selected attribute;
providing the first word vector and the second word vector to a first deep learning network trained to generate sample queries according to sample objects and sample properties;
receiving a third word vector corresponding to the first word vector and the second word vector from the deep learning network; and
and generating a query from the third word vector.

Images