KR101986345B1 - Apparatus for generating meta sentences in a tables or images to improve Machine Reading Comprehension perfomance - Google Patents

Abstract

According to the present invention, an apparatus for generating meta sentences in tables or images to improve machine reading comprehension (MRC) performance comprises: a meta sentence data set (200) in which example data converting information included in tables or images into meta sentences are stored; and a meta sentence generation unit (100) to use the example data stored in the meta sentence data set (200) to generate meta sentences from information included in tables or images from an original document including the tables or the images, and then supply a document with the meta sentences to an MRC processing unit (300) for performing MRC. According to the present invention, meta sentences for tables or images are generated in an original document including the tables or the images to supply the meta sentences to the MRC processing unit to allow questions and answers for information about the tables or the images while using a vacant MRC system without modifying the vacant MRC system to drastically improve machine reading comprehension performance.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and a method for generating meta sentences in a table or an image,

The present invention relates to a machine readable (MRC) system, and more particularly, to a machine readable system (MRC) system for generating a meta sentence in an original document including a table or an image, To the apparatus.

Machine Reading Comprehension (MRC), as we know, is the machine reading ability of Deep Learning based AI which reads and interprets a given document like a human and finds the correct answer to the question . Machine Reading Comprehension (MRC) refers to artificial intelligence technology that derives natural language question answering from a text document. As a person reads newspapers and magazines and deduces question answers, artificial intelligence (AI) It is possible to find and answer.

As the MRC technology develops, the accuracy and level of answers increase. At the core of the MRC is the data set. In other words, how to obtain and analyze various data is important for MRC performance improvement.

On the other hand, the document in the practical area includes various information such as a table or an image (picture) in addition to the text. However, the conventional MRC system finds information in the text document and answers the question. Therefore, in order to improve the performance of MRC system, it is necessary to process information such as tables and images.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points, and it is an object of the present invention to improve the performance of an original document including a table or an image by using the known MRC system without modifying the information other than plain text such as a table or an image And to provide a meta sentence to the MRC system by generating a meta sentence in a table or an image.

According to another aspect of the present invention, there is provided an apparatus for generating a meta sentence in a table and an image, the apparatus comprising: a meta sentence data set in which information included in a table is converted into an example sentence, (200): generating information from the original document including the table as meta sentences using the example sentence data stored in the meta sentence data set (200) And a meta sentence generator 100 for providing a meta sentence generated document to an MRC processor 300 that performs machine reading comprehension (MRC).

According to an embodiment of the present invention, the meta-sentence data set 200 includes a meta- meta-sentence data set 220 in which example sentence data converted into an example sentence is stored, The generating unit 100 generates a table meta-sentence generating unit for generating information included in the table as a meta sentence from the original document including the table using the example sentence data of the table meta-sentence data set 220 120). The table meta-sentence generation unit 120 generates at least one of a meta sentence for each column of the table and a meta sentence for each line, and attaches the generated meta sentence to the original document.

According to another embodiment of the present invention, the table meta-sentence data set 220 includes a table pattern template 222 that is stored in a plurality of patterns in accordance with the type and the type of the table, 220) stores example sentence data converted into an example sentence for each column or each row corresponding to each pattern of the table pattern template 222. The table meta sentence generation unit 120 generates a table meta sentence A corresponding pattern among a plurality of patterns stored in the table pattern template 222 is searched according to the type and form of the table and is included in the table of the original document using the example sentence data for each column or each row corresponding to the pattern Information is created as a meta sentence and attached to the original document.

According to another embodiment of the present invention, the table meta-sentence generation unit 120 notifies a user when a table included in an original document can not find a corresponding pattern among a plurality of patterns stored in the table pattern template 222 And stores the new pattern corresponding to the table included in the original document and the example sentence data converted into the example sentence corresponding to the new pattern in the table meta sentence data set 220 from the user, Uses the new pattern and the corresponding sentence data corresponding thereto to generate information included in the table of the original document as a meta sentence and attaches the information to the original document.

According to another embodiment of the present invention, the meta-sentence data set 200 may further include an image meta-sentence data set 240 in which example data converted into meta data is stored, The meta sentence generation unit 100 may generate the meta sentence data set 240 using the image meta sentence data set 240 and an image captioning engine 250 that automatically generates a meta sentence for the image, And an image meta sentence generation unit 140 for generating information included in the image from the original document as a meta sentence.

According to another embodiment of the present invention, the meta-sentence generation unit 100 uses the table meta-sentence data set 220 and the image meta-sentence data set 240 to generate an original document And a table / image meta sentence generator 160 for generating information included in the table and the image as a meta sentence.

According to the present invention, a meta sentence about a table or an image is generated in an original document including a table or an image and provided to the MRC processing unit, so that it is possible to inquire about information about the table or image while using the known MRC system without modification As a result, the improvement of machine reading performance is remarkably improved.

FIG. 1 is a block diagram of an apparatus for generating a meta sentence in a table / image in order to improve the machine reading performance according to an embodiment of the present invention.
Fig. 2 is an enlarged view of the main part of Fig. 1,
FIG. 3 is a view for explaining an example of a table pattern stored in the table pattern template of FIG. 1,
FIGS. 4 to 7 are views for explaining the operation of an apparatus for generating a meta sentence in a table / image in order to improve the machine reading performance according to the embodiment of the present invention.

These and other objects, features and other advantages of the present invention will become more apparent by describing in detail preferred embodiments of the present invention with reference to the accompanying drawings. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an apparatus for generating a meta sentence in a table and an image will be described in detail with reference to the accompanying drawings in order to improve the machine reading performance according to an embodiment of the present invention.

1 and 2, an apparatus for generating a meta sentence in a table image according to an exemplary embodiment of the present invention includes a meta sentence generation unit 100 and a meta sentence data set 200 .

In the meta-sentence data set 200, the example sentence data converted from the information included in the table or the image into the example sentence is stored. Specifically, the meta-sentence data set 200 includes a table meta-sentence data set 220 in which the example sentence data converted from the information included in the table is converted into an example sentence, And an image meta sentence data set 240 in which the example sentence data is stored.

The table meta-sentence data set 220 includes a table pattern template 222 that is stored in a plurality of patterns in accordance with the type and shape of the table. The type and form of a table is the type and form of the table (for example, whether a column and a row are formed as a single cell or divided into a plurality of cells), the contents of the table (for example, the first row and the first column The contents described in each cell) are stored in a predetermined pattern, and the pattern may be several tens to several thousands. As described above, since the performance of the MRC improves as the number of data sets increases, the number of the table patterns increases.

The table meta-sentence data set 220 stores the example sentence data converted into the example sentences corresponding to the respective patterns of the table pattern template 222. That is, corresponding example sentences corresponding to each table pattern are stored. The number of example sentences corresponding to one table pattern may be one or more. That is, one table pattern may have a plurality of rows and columns, and the corresponding example sentences corresponding to the plurality of rows and columns may be plural because a sentence can be generated by a corresponding combination of each row and column. The details of this will be described later.

The image meta-sentence data set 240 stores the example sentence data converted from the information included in the image into an example sentence.

The meta sentence generation unit 100 generates information included in the table or image as a meta sentence from the original document 10 including the table or image by using the example sentence data stored in the meta sentence data set 200 To the original document. Then, the meta sentence generation unit 100 provides the MRC processing unit 300, which performs machine reading comprehension (MRC), with the document in which the meta sentence is generated. That is, the meta-sentence generation unit 100 generates a sentence having a context so that the MRC processing unit 300 can recognize the table or the image. These meta-sentences can be attached around the table in the original document.

The meta sentence generation unit 100 includes a table meta sentence generation unit 120, an image meta sentence generation unit 140, and a table / image meta sentence generation unit 160.

The table meta sentence generation unit 120 attaches the original document including the table to the original document that generates the information included in the table as a meta sentence using the example sentence data of the table meta sentence data set 220 .

The table meta sentence generation unit 120 recognizes a table included in the original document and finds a corresponding pattern among a plurality of patterns stored in the table pattern template 222 according to the type and the type of the table, , The information included in the table is created as a meta sentence and attached to the original document.

On the other hand, the table meta-sentence generation unit 120 notifies the user when a table included in the original document can not find a corresponding pattern among a plurality of patterns stored in the table pattern template 222. [ Accordingly, the user can save the new pattern corresponding to the table included in the original document and the example data converted into the corresponding sentence corresponding to the table in the table meta sentence data set 220, and the table meta sentence generation unit 120 A meta sentence is generated using the new pattern stored in the meta sentence data table 220 and the corresponding sentence data corresponding to the new pattern and attached to the original document.

The image meta sentence generation unit 140 generates an image meta sentence using the image meta sentence data set 240 and the image captioning engine 250 that automatically generates a meta sentence for the image, The information included in the image is generated as a meta sentence and attached to the original document. The image capturing engine 250 may use a known artificial intelligence engine such as GAN (Generative Adversarial Networks), which converts images into text sentences. According to an embodiment of the present invention, the image meta sentence generation unit 140 may include an image captioning engine 250.

The table / image meta-sentence generation unit 160 generates a table meta-sentence data set 220 and an image meta-sentence data set 240 in the form of an image Information included in the table and the image is generated as a meta sentence from an original document including an image in the table and is attached to the original document.

The MRC processing unit 300 reads the machine using the document 50 in which the meta sentence generated by the meta sentence generation unit 100 is generated, reads and comprehends the document, and finds and provides an answer to the question. A known MRC model can be used as the MRC processing unit 300. Various models such as BiDaF, Document-QA, ReasoNet, R-NET, SynNet, and OpenNMT can be applied.

3 to 7, the operation of the apparatus for generating meta sentences in tables and images will be described in detail in order to improve the machine reading performance according to the embodiment of the present invention.

First, the operation of generating the meta sentence in the table will be described.

4, the table meta-sentence generation unit 120 of the meta-sentence generation unit 100 generates a table meta sentence having the size of 3 * And automatically generates meta statements for the rows and columns (using the sentence data set 220). Meta sentences can be general sentences or they can be in the form of specific patterns.

FIGS. 4 and 5 show the most typical and typical table form (i.e., each row and column are composed of only one cell). FIGS. 4 and 5 can be divided into a single pattern, Assume the pattern of FIG. That is, the table pattern of FIG. 3 is stored in the table pattern template 222, which may be one of dozens to thousands of patterns stored in the table pattern template 222.

Normally, the contents in the cells in the first row (top row) and the first column (left column) are item displays such as table classification and type, and the contents in the cells in the remaining rows and columns indicate the contents of each yield do.

On the other hand, the table meta-sentence data set 220 stores the example data sentences converted into the example sentences corresponding to the table patterns.

For example, the example sentence for a row of the table pattern of FIG. 3 is '[Header 1]' [a] is [or Header 2] [Header 3] is (or is) [a2], and [Header 1] [b] is [or Header 2] ), And [Header 3] is [b2], and the example sentence for the column of the table pattern of FIG. 3 is' [Headr 1] And [Header 2] are [a1] and [or b1], and [Headr 3] and [Header 2] Can be [a2] and (or), [b2] (or).

Usually, the contents written in the cells in the first column are often given as subjects. Therefore, it is often the case that the contents written in the cells in the last column are often used as descriptive words, such as',,,, To create an example sentence with an investigation such as 'is'. The contents in the cells in the middle column are analyzed to analyze the contents of each cell, and then, for example, "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~" .

According to the preferred embodiment of the present invention, each pattern stored in the table pattern template 222 includes a sum of the number of rows (two in FIG. 3) and the number of columns (three in FIG. 3) (Five in the case of FIG. 3) are stored in the table meta-sentence data set 220 at least.

4 and 5, when the table as shown in FIG. 4 exists in the original document 10, the table meta-sentence generation unit 120 of the meta-sentence generation unit 100 first recognizes the table of FIG. 3 A table pattern corresponding to the type and form of the table is found in the table pattern template 220. [ As described above, the table pattern corresponding to FIG. 4 is illustrated in FIG. 3. The table meta sentence generation unit 120 generates a meta pattern using the information included in the table using the example sentence corresponding to the corresponding table pattern, ) Sentence.

As described above, the example sentence of each line for the pattern of FIG. 3 is '[Header 1]' [a] is [or Header 2] Header 2] is [b1], [Header 3] is [a2], and [Header 1] is [b2] 4, the meta-sentence for the row of the table in Fig. 4 is as follows: "Product name (person) A has a monthly salary of A1, data of A2", " And the data of B is B2. &Quot;

In addition, the example sentence of each column for the pattern of FIG. 3 is '[Headr 1]' (or silver) [a] and (or), [b] Or [a1], [b1], and [Headr 3] are [a2] and [or] and [b2], respectively. , The meta sentence for the column of the table in FIG. 4 can be generated as 'Product Name is A and B,' 'Monthly Amount is A1 and B1', 'Data is A2 and B2', etc. have.

On the other hand, in addition to these basic meta sentences, a meta sentence for a question that a user may frequently ask may be generated as follows. For example, a product name having a monthly salary of A is A, a product name of A2 is A, and so on. Also, it is possible to generate a sentence having a complex condition such as 'a product whose monthly amount is X or more and data is Y or less is A'. Of course, in order to generate a meta sentence containing such a complicated condition, it is necessary to store some similar sentence data in the table meta sentence data set 220. However, basically, the table meta sentence generation unit 120 generates a meta sentence data set The information included in the table can be generated as a meta sentence from the original document 10 including the table by using the table 220.

Similarly, when a 4 * 4 size table as shown in FIG. 5 exists in the original document 10, such a table can be recognized as a pattern similar to the table type of FIG. 4, And generates a meta sentence using the example sentence data corresponding to the pattern. For example, the meta-sentence for the second row can be generated as 'volcano name Mt.Lassen is California, most recent eruption is 1914-17, and eruption type is explosive eruption'.

 Referring to FIGS. 6A and 6B, the table has patterns different from those of the preceding tables. For example, in the case of a table having merged cells as shown in FIG. 5A, all the merged cells are found , And after the merging is canceled, each row or column is made into a sentence.

Accordingly, the table pattern is stored in the table pattern teletext 222, and the corresponding sentence is stored in the table meta data set 220. For example, the example sentence for the second row is' [Header 1] This (or a) [first cell content] is an example sentence such that [Header 2] is (or a) [second cell content], and [Header 3] is (or a) [third cell content] . Then, the table meta sentence generation unit 120 generates a meta sentence using the example sentence data. For example, the meta sentence can be generated in such a manner that 'the category is apple and the color is red' have.

The table meta-sentence generation unit 120 recognizes such a table, and generates a list of the rows, columns, cells, and the like that form the table. The contents inside the cell can be recognized.

Next, the operation of generating the meta sentence in the figure will be described.

First, the meta sentence about the photograph is automatically generated using the image capturing engine 250. The image meta sentence generation unit 140 of the meta sentence generation unit 100 generates a meta sentence using the meta sentence generated by the image meta sentence data set 220 and the image captioning engine 250 Generate a meta sentence.

For example, the image captioning engine 250 generates a sentence 'Two people are shaking hands' for the image shown in FIG. 7, and the image meta sentence generating unit 140 generates' This handshake is being done. ' Of course, in order to generate meta sentences including such a complex condition, an example sentence similar to a certain degree should be stored in the image meta sentence data set 240. Basically, the image meta sentence generation unit 140 generates an image meta sentence data set 240), information included in the table or image can be generated as a meta sentence from the original document 10 including the image.

Here, the image includes not only the photograph as shown in FIG. 7 but also a graph or an image.

The table / image meta generator 160 includes a table meta sentence data set 220, an image meta sentence data set 240, an image captioning engine 210, The meta sentence can be generated by using the meta sentence.

In addition, various meta sentences generated by the meta sentence generation unit 100 can be stored in the meta sentence data set 200, so that the data of the data set can be secured.

In this way, a meta sentence is generated and attached to a table or an image of an original document by the meta sentence generation unit 100, and the document 20 in which the meta sentence is generated is provided to the MRC processing unit 300. The MRC processing unit 300 may be able to perform a query response to information represented by a table or figure using the document 20 in which the meta sentence is generated.

As described above, according to the present invention, a known MRC system (engine or model) can be provided by generating a meta sentence in a table or an image and providing it to the MRC processing unit by generating a sentence having a context so that the MRC processing unit can recognize the table or image, It is possible to inquire about the information of the table or the image without using the correction, so that the improvement of the MCR performance is remarkably improved. MCR performance improvement has been dramatically improved to overcome the disadvantages of the application fields of the conventional MCR system and to overcome all the areas where the content of the document should be understood and an answer should be provided to the user query (for example, Such as identification of toxin clauses, intelligent chatbots, educational robots based on book content, etc.).

Although the preferred embodiments of the present invention have been described, the present invention is not limited to the specific embodiments described above. It will be apparent to those skilled in the art that numerous modifications and variations can be made in the present invention without departing from the spirit or scope of the appended claims. And equivalents should also be considered to be within the scope of the present invention.

10. Original document 20. Meta sentence generated document
100. Meta sentence generation unit 120. Table meta sentence generation unit
140. An image meta sentence generation unit 200. A meta sentence data set
220. Table meta-sentence data set 222. Table pattern template
Image meta-sentence data set 300. MRC processing unit

Claims (6)

A meta-sentence data set 200 in which example sentence data converted into a meta sentence is stored,
Using the example sentence data stored in the meta sentence data set 200, information included in the table is generated as a meta sentence from an original document including the table, attached to the original document, and then a meta sentence is generated To a MRC processing unit (300) for performing a machine reading comprehension (MRC);
Lt; / RTI >
The meta-sentence data set 200 includes a table meta-sentence data set 220 in which example data converted into an example sentence is stored,
The meta-sentence generation unit 100 generates a meta-sentence sentence using meta sentence data of the meta-sentence data set 220 to generate information included in the meta- Generating unit 120,
The table meta sentence generation unit 120 generates at least one of a meta sentence for each column of the table and a meta sentence for each row and attaches the meta sentence to each of the original documents. / RTI > delete The method according to claim 1,
The table meta-sentence data set 220 includes a table pattern template 222 that is stored in a plurality of patterns in accordance with the type and the type of the table, and the table meta-sentence data set 220 includes the table- ), The example sentence data converted into the example sentence for each column or for each row is stored,
The table meta-sentence generation unit 120 finds a corresponding pattern among a plurality of patterns stored in the table pattern template 222 according to the type and the type of the table included in the original document, A meta sentence is created in a table or image in order to improve the machine reading performance by generating the meta information in the original document using the example sentence data for the meta sentence. Device. The method of claim 3,
If the table included in the original document does not find a corresponding pattern among a plurality of patterns stored in the table pattern template 222, the table meta-sentence generation unit 120 notifies the user of the pattern, And stores the new pattern corresponding to the table and the example sentence data converted into the corresponding example sentence into the table meta sentence data set 220. The table meta sentence generation unit 120 generates the new pattern and the corresponding example And the information included in the table of the original document is generated as a meta sentence using the sentence data and attached to the original document. The apparatus for generating a meta sentence in a table / image for improving the machine reading performance. The method according to claim 1,
The meta-sentence data set 200 further includes an image meta-sentence data set 240 in which example sentence data converted into an example sentence is stored,
The meta sentence generator 100 generates a meta sentence using the image meta sentence data set 240 and an image captioning engine 250 that automatically generates a meta sentence for the image, Further comprising an image meta sentence generation unit (140) for generating information included in the image from a document as a meta sentence, wherein the image meta sentence generation unit (140) generates the meta sentence in the table / image. 6. The method of claim 5,
The meta sentence generation unit 100 generates the meta sentence data set 220 and the image meta sentence data set 240 using the table meta sentence data set 220 and the image meta sentence data set 240, And a table / image meta sentence generation unit (160) for generating a meta sentence in a meta (meta) sentence.

Images