KR102593463B1 - Apparatus and method for generating language based on commonsense reasoning - Google Patents

Abstract

Pertaining to a language generation device and method based on general knowledge reasoning, the language generation device based on general knowledge reasoning inputs at least one concept, at least one image, and at least one document. Using a receiving input unit and the at least one concept, scene knowledge is extracted from the at least one image, relationship knowledge is extracted from the at least one document, and a language model is created using the scene knowledge and the relationship knowledge as input values. It may include a processor that performs learning processing.

Description

Apparatus and method for generating language based on common sense reasoning {APPARATUS AND METHOD FOR GENERATING LANGUAGE BASED ON COMMONSENSE REASONING}

It relates to a language generation device and method based on general knowledge reasoning.

Artificial Intelligence (AI) allows mechanical devices such as computers to process human learning and reasoning abilities, and can be implemented in the form of devices, systems, or programs. Recently, artificial intelligence technology has been growing rapidly due to the development of machine learning technology, which trains models based on various information to find the optimal model and uses it to make decisions, and is having a significant impact on a wide range of industrial fields. . These artificial intelligence technologies can also be applied in the field of natural language processing to more accurately and effectively perform natural language processing such as text analysis process, voice recognition and conversion into text, or text recognition and translation into other languages. It is being done. Recently, technology that uses artificial intelligence technology to generate new, precise sentences in response to sentences or concepts given to them has been researched, developed, and disclosed. It is believed that this sentence generation technology can be used not only for simple text such as articles or emails, but also for the creation of literary works such as literature. However, due to technical limitations, these sentence generation technologies often have problems such as generating sentences by omitting input concept information or generating awkward sentences that are difficult for humans to understand. Accordingly, the need for research and development of language generation technology that naturally generates sentences as if they were written by humans has increased.

The goal is to provide a language generation device and method based on general knowledge reasoning that can strengthen general knowledge reasoning ability by reflecting retrieved external knowledge and generate more natural sentences based on the strengthened general knowledge reasoning ability. do.

In order to solve the above-mentioned problems, a language generation device and method based on common sense reasoning are provided.

The general knowledge inference-based language generation device includes an input unit that receives at least one concept, at least one image, and at least one document, and extracts scene knowledge from the at least one image using the at least one concept, It may include a processor that extracts relationship knowledge from the at least one document and performs a language model learning process using the scene knowledge and the relationship knowledge as input values.

A general knowledge inference-based language generation method includes receiving at least one concept, at least one image, and at least one document as input, extracting scene knowledge from the at least one image using the at least one concept, It may also include extracting relationship knowledge from the at least one document using the at least one concept and performing learning processing of a language model using the scene knowledge and the relationship knowledge as input values.

According to the above-described general common sense reasoning-based language generation device and method, the general common sense reasoning ability is strengthened by reflecting the retrieved external knowledge, and the effect of being able to generate more natural sentences based on the strengthened general common sense reasoning ability is obtained. You can.

According to the above-described common sense inference-based language generation device and method, it is possible to assume a general situation that can be implemented by a combination of given conceptual information using scene knowledge, and using relational knowledge. By making it possible to infer a general relationship between scene knowledge and one or more conceptual information, the advantage of being able to construct more valid sentences during the language generation process can also be obtained.

According to the above-described general knowledge inference-based language generation device and method, it is possible to reinforce the low performance and insufficient general knowledge inference ability of language models such as conventional autoregressive-based generative models, thereby omitting given conceptual information and generating sentences. By creating or deleting sentences with insufficient information, it is possible to prevent the creation of sentences that cannot be understood through common sense.

According to the above-described common sense reasoning-based language generation device and method, there is also the advantage of being able to implement a language generation model, device, and method that have versatility applicable to most language generation models regardless of whether there is an encoder or not.

1 is a block diagram of an embodiment of a language generation device based on common sense reasoning.
Figure 2 is a block diagram for explaining an embodiment of the operation of a processor.
Figure 3 is a block diagram of an embodiment of the language model processing unit.
Figure 4 is a flowchart of an embodiment of a language generation method based on common sense reasoning.

Throughout the specification below, the same reference signs refer to the same components unless there are special circumstances. Terms with the addition of 'unit' used below may be implemented as software and/or hardware, and depending on the embodiment, one 'unit' may be implemented as one physical or logical part, or a plurality of 'units' may be implemented. It is also possible to be implemented with one physical or logical part, or one 'part' to be implemented with a plurality of physical or logical parts. When a part is said to be connected to another part throughout the specification, this may mean that the part and the other part are physically connected to each other and/or electrically connected. In addition, when a part includes another part, this does not mean excluding another part other than the other part unless specifically stated to the contrary, and means that another part may be included depending on the designer's choice. do. Expressions such as the first to Nth (N is a natural number greater than or equal to 1) are intended to distinguish at least one part(s) from other part(s), and do not necessarily mean that they are sequential unless otherwise specified. Additionally, singular expressions may include plural expressions, unless the context clearly makes an exception.

Hereinafter, an embodiment of a language generation device based on common sense reasoning will be described with reference to FIGS. 1 to 3.

1 is a block diagram of an embodiment of a language generation device based on common sense reasoning.

According to one embodiment shown in Figure 1. The general knowledge inference-based language generation device 100 (hereinafter referred to as language generation device) may include a processor 110, and, if necessary, an input unit 101, a storage unit 105, and an output unit 109. It may include at least one more. At least two of the input unit 101, storage unit 105, output unit 109, and processor 110 are configured to transmit data or commands/instructions one-way or two-way through a circuit, cable, or wireless communication network. It may have been prepared.

The input unit 101 may receive data or programs (which may be referred to as apps, software, or applications) required for the operation of the language generating device 100 from a user or another device (not shown).

For example, the input unit 101 may receive the concept set 91 itself, or at least one concept(s) (91-1 in FIG. 2) that may be included in the concept set 91. . Here, at least one concept 91-1 may include one or more words, for example, at least one semantic word (content word). If necessary, at least one concept 91-1 may include at least one functional word. At least one word may include a word from a predetermined part of speech, for example, a verb (e.g., possess, have, wear, etc.) and a noun (e.g., rose, etc.) to create a sentence that reflects general knowledge in an objective form. It may also include at least one of the following: a garden, a dress, etc.). Here, the verb may include a verb base form, a gerund, a present or past tense, and/or a past participle, and the noun may include a singular common noun, a plural common noun, a singular proper noun, and/or a plural proper noun. there is. Depending on the embodiment, at least one word may include an adverb or an adjective in addition to a verb or noun. The concept set 91 refers to a set formed by including one or more of these concepts (91-1). The concept set 91 may be prepared in the form of a database with individual concepts 91-1 as elements. In addition, the input unit 101 inputs an image (93, which may include a still image, a video, or one or more individual frames within a video) used to extract scene knowledge, or a document (95, text, etc.) used to extract relationship knowledge. Additional information (which may also include tables, images, or symbols) may be received as needed. At least one of the image 93 and the document 95 may be input directly according to the operation of the user, etc., or may be transmitted and input through a separately provided memory device or other information processing device (for example, a server hardware device or a smartphone, etc.). It may be possible. At least one of the image 93 and the document 95 may be obtained from another external website accessible through a wired or wireless communication network. For example, the image 93 may be obtained by collecting from an external search site or an image database site. Additionally, the image 93 may be obtained from a separately prepared image data set (eg, VATEX or Visual Genome). For example, the document 95 may be obtained by collecting from an external search site or an information database site (eg, a dictionary or wiki site, etc.), and may include a web document. Collection of at least one of these images 93 and documents 95 may be performed manually or automatically according to an operation of a user or the like or a control operation of the processor 110.

A program input through the input unit 101 may include an algorithm for operating the processor 110, etc. Depending on the embodiment, the input unit 101 may include a keyboard, mouse, tablet, touch screen, touch pad, track ball, track pad, scanner device, image capture module, ultrasound scanner, motion detection sensor, vibration sensor, light receiving sensor, and pressure sensitive sensor. A data input/output terminal (such as a universal serial bus terminal, a high-definition multimedia interface terminal, or a slot) may include sensors, proximity sensors, and/or microphones, and can receive data, etc. from other devices (e.g., portable memory devices, etc.). (M.2 or PCI express), etc.) or a communication module (for example, a LAN card, short-distance communication module, or mobile communication module) that is connected to another external device through a wired or wireless communication network.

The storage unit 105 can store data or programs necessary for the operation of the language generating device 100. For example, the storage unit 105 may temporarily or non-temporarily store at least one of a concept set 91, an image 93, and a document 95 to be used by the processor 110. Data stored in the storage unit 105 may be input through the input unit 101 or may be generated during processing by the processor 110. Data (at least one of the concept set 91, image 93, and document 95) stored in the storage unit 105 may be transmitted to the processor 110 as needed, such as a call to the processor 110. . Depending on the embodiment, the storage unit 105 may include at least one of a main memory and an auxiliary memory. The main memory may include semiconductor storage media such as ROM and/or RAM, and the auxiliary memory may include a flash memory device, Secure Digital (SD) card, or solid state drive (SSD). Drive, hard disk drive (HDD), optical media such as magnetic drum, compact disk (CD), DVD or laser disk, magnetic tape, magneto-optical disk and/or floppy disk It may include a disk, etc.

The output unit 109 may output the processing results of the processor 110 or data stored in the storage unit 105 to the outside. For example, the output unit 109 may visually or auditorily output at least one sentence generated by the processor 110 in response to the processed concept 91. In this case, at least one sentence may be output in the form of text, etc. on the display screen. In addition, for another example, the output unit 109 transmits one or more sentences generated by the processor 110 to another external device (smart phone, server hardware device, memory device, etc.) via a cable or wireless communication network. It can also be provided through delivery. Depending on the embodiment, the output unit 109 may be provided integrally with the language generating device 100, or may be provided to be physically separable, for example, a display, a printer device, a speaker device, an image device, etc. It may include an output terminal, a data input/output terminal, and/or a communication module. However, the output unit 109 is not limited to this.

The processor 110 uses the concept set 91 to enhance the performance of the language model (e.g., general knowledge reasoning ability and/or sentence generation ability based on it) or generates one or more sentences using the language model. can be created. The processor 110 may execute a program stored in the storage unit 105 to perform these operations. The processor 110 may include, for example, a central processing unit (CPU), a microcontroller unit (MCU), an application processor (AP), and an electronic control unit (ECU). ), a Baseboard Management Controller (BMC), a Micro Processor (Micom), and/or at least one electronic device capable of performing various calculations and control processing. These processing or control devices may be implemented, for example, by using one or more semiconductor chips, circuits, or related components alone or in combination.

As shown in FIG. 1, the processor 110 may include an extraction unit 120, an encoding unit 130, and a language model processing unit 140 in one embodiment. At least two of the extraction unit 120, the encoding unit 130, and the language model processing unit 140 may be physically separated or logically separated. If they are physically separated, the extraction unit 120, At least two of the encoding unit 130 and the language model processing unit 140 may, for example, be implemented by different physical processing devices, and when logically separated, the extraction unit 120, the encoding unit 130, and the language All of the model processing units 140 may be implemented by one physical processing device.

Figure 2 is a block diagram for explaining an embodiment of the operation of a processor.

As shown in Figures 1 and 2, the extraction unit 120 may include a scene knowledge extraction unit 121 and a relationship knowledge extraction unit 123.

The scene knowledge extraction unit 121 selects at least one concept 91-1 of the concept set 91 input through the input unit 101 or stored in the storage unit 105, for example, flower, dress, garden, holding and wearing. etc.), and obtain at least one image 93 received through the input unit 101 (for example, received through a search of a website, etc.) or stored in the storage unit 105, and at least Scene knowledge 121a can be acquired based on one concept 91-1 and at least one image 93. The scene knowledge 121a may include, for example, one or more languages (eg, words or combinations of words, etc.), and more specifically, one or more words extracted from at least one image 93. It can be included.

The scene knowledge extraction unit 121 may use the image 93 as a database to search and extract content depicting a general situation for a specific concept 91-1 to obtain scene knowledge 121a. For example, when concepts (91-1) such as [rose], [garden], and [clothing] are input, the corresponding image 93 is displayed such as [woman wearing a black dress] or [woman You can acquire scene knowledge (121a) such as [I am holding this flower]. Additionally, the scene extractor 121 is also capable of extracting scene knowledge 121a from a data set (for example, VATEX or Visual Genome, etc.) that includes a description of at least one image. According to one embodiment, the scene knowledge extraction unit 121 may use Wittgenstein picture theory to extract the scene knowledge 121a. Wittgenstein's picture theory is a theory that language (propositions) correspond one-to-one to the world (facts), and that language can describe the world like a picture. The scene extraction unit 121 may acquire scene knowledge 121a by performing generative commonsense reasoning about the scene according to Wittgenstein's picture theory. More specifically, for example, the scene extraction unit 121 performs part-of-speech tagging and stemming to extract a description highly related to the concept 91-1 to extract at least one image 93 ), you can perform scene knowledge search. Part-of-speech tagging can be performed by selecting the concept (91-1) according to the part of speech of the corresponding concept (91-1). For example, in the part-of-speech tagging process, only the concept (91-1) corresponding to a verb or noun can be selected among all concepts (91-1). Part-of-speech tagging can be performed, for example, using a part-of-speech tagger (POS tagger) of the Natural language Toolkit (NLTK). Stemming may be performed by extracting stems from all or some of the token(s) within the description of concept 91-1 and image 93. Grammatical differences can be ignored by stemming. According to one embodiment, when a sentence is extracted, the scene knowledge extraction unit 121 treats sentences containing at least two concepts with the same stem present in the selected concept 91-1 among all extracted sentences as valid. You can. Sentences treated as valid can be reconstructed into scene knowledge-based sentences that describe the external world. Scene knowledge 121a can be acquired according to this process. A scene knowledge-based sentence may include, for example, a set of concepts with a verb and a noun. Such scene knowledge-based sentence learning allows one to improve sentence generation ability based on understanding of general situations obtained from external clock knowledge, just as when explaining a picture.

The relationship knowledge extraction unit 123 may acquire relationship knowledge 123a based on at least one concept 91-1 and at least one document 95. The relational knowledge 123a may include external visual knowledge of the predetermined concept 91-1 and non-visual common sense that is difficult to derive only from the searched or used scene knowledge 121a. This kind of relationship knowledge (123a) allows information about the general relationship of each concept (91-1) to be inferred based on Wittgenstein's usage theory. Specifically, the relational knowledge 123a further uses external knowledge (e.g., a document 95) in one or more general situations assumed by each given concept 91-1 to form a relationship between each concept 91-1. 1) It is possible to expand information about the general relationship. For example, when the sentence [A woman is holding a flower] is extracted from the image 93, if [Congratulations] is further used as the relationship knowledge 123a, the language model processing unit 140 determines that [The woman is holding a flower]. [He is celebrating with flowers] The sentence can be created. According to one embodiment, in order to acquire the relationship knowledge 123a, the relationship knowledge extraction unit 123 first calculates a relevance score for one or more documents 95 and selects at least one document 95 using this. can do. In this case, a document 95 with a relatively high relevance score, that is, a document 95 with high relevance, may be selected. The relevance score for one or more documents (95) is, for example, the frequency with which the concept (91-1) is derived from one or more concepts (91-1) that are the same as a given one or more concepts (91-1) among one or more documents (95). (91-1) can be calculated based on the length of the corresponding document 95 from which it is derived and the average length of all documents 95. According to one embodiment, the score may be given as Equation 1 below.

[Equation 1]

In Equation 1, score(D, Q) is the score according to the given document (D) and concept (Q, query), IDF() is the inverse document frequency, and f() is the term frequency. k and b are hyperparameters and can be given as 1.2 and 0.75, respectively, for example. |D| is the length of the document (D), and avgDL means the average length of all documents.

When at least one document 95 is selected, the selected document 95 is decomposed into at least one sentence, and each sentence can be used as a relative knowledge base sentence. In this case, depending on the embodiment, a sentence containing at least two concepts (91-1) among sentences obtained through decomposition may be used as a relative knowledge-based sentence.

If necessary, a noise removal process may be performed on multiple sentences to remove irrelevant information. For example, noise due to irrelevant information can be removed by selecting highly relevant sentences from among a plurality of sentences in one or more selected documents 95 and excluding other sentences. According to one embodiment, the noise removal process may be performed using the GloVe word embedding method. Specifically, for example, the similarity (e.g., cosine similarity) between a given concept (91-1) and concepts extracted from a sentence (e.g., a sentence extracted from a document with a high relevance score) is calculated first, and the relatively high By extracting and selecting only sentence(s) with similarity, sentences with high relevance may be selected.

The scene knowledge 121a and relationship knowledge 123a obtained as described above may be transmitted to the encoding unit 130. The encoding unit 130 may encode the scene knowledge 121a and the relationship knowledge 123a and merge the encoding results 131 and 132 to generate an input value 135 for the language model processing unit 140. Here, the input value 135 may be given as a pair of at least one concept and a knowledge-based sentence. According to one embodiment, the encoding unit 130 converts the scene knowledge 121a to generate the first sequence 131 corresponding to the scene knowledge 121a, and/or converts the relationship knowledge 123a to create a relationship A second sequence 132 corresponding to the knowledge 123a may be generated. More specifically, each sequence 131, 132 includes an initial token (SOS), an end token (EOS), at least one concept disposed between the initial token (SOS) and an end token (EOS), and these concepts It may be implemented by including extracted and derived sentences, where the sentence from which the concept is extracted may belong to either scene knowledge 121a or relationship knowledge 123a. The encoding unit 130 may merge each sequence 131 and 132 to generate an input value 135 for the language model processing unit 140 and transmit it to the language model processing unit 140.

The language model processing unit 140 may perform learning processing based on the input value 135. For example, the language model processing unit 140 may train a language model using the input value 135 and/or use the input value 135 to obtain a result corresponding to the concept 91-1 (i.e. sentences, etc.) can also be obtained. According to one embodiment, the language model processing unit 140 is an autoregressive language model such as a generative pre-trained transformer (GPT), generative pre-trained transformer 2 (GPT2), BART, or T5. It can also be implemented using an autoregressive model. A generative pre-trained transformer (GPT) contains a transform decoder with a very large number of parameters, such as . Generative Pre-Trained Transformer 2 (GPT2) is a transformer that has a normalization layer in front of each self-attention block, just like the pre-activation residual network. It is a decoder-only, transformer-based autoregressive model that is pre-trained using massive text data. am. BART is pre-trained using variation removal methods such as token masking, deletion or filling, and consists of a transformer encoder and decoder structure. T5 is also formed with a converter encoder and decoder structure, and uses a relative position encoding method instead of the typical positional encoding. In addition to these, the language model processing unit 140 includes, for example, a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), and a convolutional neural network ( CRNN: Convolutional Recurrent Neural Network, Deep Belief Network (DBN), Deep Q-Networks, Long short term memory (LSTM), Multi-layer Perceptron, Training and inference based on at least one learning model, such as a support vector machine (SVM), a generative adversarial network (GAN), and/or a conditional generative adversarial network (cGAN). You can also perform .

Figure 3 is a block diagram of an embodiment of the language model processing unit.

For example, if the language model processing unit 140 is implemented using generative pre-training transformer 2 (GPT2), the language model processing unit 140 uses the masked multi-head attention 141 as shown in FIG. 3. , may include a multi-head attention 143, a feed forward network (145), and an output processing unit 147. The masked multi-head attention 141 performs self-attention based on only some words (some tokens) among the words in the input sentence. The multi-head attention 143 may receive the processing result of the masked multi-head attention 141, perform multiple self-attention on the processing results, and transmit the processing results to the forward network 145. The forward network 145 performs learning based on the processing result of the multi-head attention 143, and the output processing unit 147 calculates the final result (149) from the result value of the forward network 145 using a softmax function, etc. ), that is, an improved sentence is generated and obtained based on scene knowledge (121a) and relationship knowledge (123a).

Accordingly, the language generating device 100 can generate more natural sentences based on enhanced general knowledge reasoning ability using image and relationship knowledge.

The above-described language generating device 100 may be implemented using a device specifically designed to perform the above-described operations, or may be implemented by using one or more information processing devices alone or in combination. Here, one or more information processing devices include, for example, a desktop computer, a laptop computer, a server hardware device, a scanner device, a printer device, a three-dimensional printer device, a smart phone, a tablet PC, a smart watch, a smart tag, a smart band, Head Mounted Display (HMD) devices, portable game consoles, personal digital assistants (PDAs), navigation devices, smart keys, digital televisions, set-top boxes, digital media player devices, media streaming devices, DVD playback devices, compact disc playback devices, sound reproduction devices (artificial intelligence speakers, etc.), home appliances (e.g. refrigerators, fans, air conditioners, ovens or washing machines, etc.), manned or unmanned vehicles (e.g. vehicles such as cars, buses or two-wheelers, etc.) Mobile robots, wireless model vehicles, robot vacuum cleaners, etc.), manned or unmanned flying vehicles (e.g. aircraft, helicopters, drones, model airplanes, model helicopters, etc.), medical devices (X-ray imaging devices, computed tomography devices (CT) Tomography or magnetic resonance imaging (MRI) devices, etc.), household, industrial or military robots, industrial or military machines, or traffic controllers, but are not limited thereto. Depending on the situation or conditions, designers, users, etc. may consider and employ at least one of various devices for computational processing and control of information in addition to the above-described information processing device as the above-described language generation device 100.

Hereinafter, several embodiments of a language generation method based on common sense reasoning will be described with reference to FIG. 4.

Figure 4 is a flowchart of an embodiment of a language generation method based on common sense reasoning.

Referring to Figure 4, in order to perform a language generation method (hereinafter referred to as a language generation method) based on common sense reasoning, one or more concepts may first be input from a user or another device (memory device, information processing device, etc.). (200). The input concept may be stored in the form of a concept set in the storage unit.

When the language generation operation is initiated according to the user's operation or predefined settings, the concept input by the processor is acquired or at least one concept in the concept set stored in the storage is acquired, and simultaneously or sequentially, images and At least one of the documents may be obtained (202). Acquisition of at least one of images and documents may be performed through at least one of manipulating an input unit such as a user, searching another website, receiving from another external storage medium, etc., and may also be performed through various methods in addition to these. there is. Here, the image may include a still image or a moving image, and may also include at least one frame of the moving image. A document can contain text and, if necessary, one or more images.

Once at least one image is acquired, scene knowledge can be extracted and acquired from the at least one image (204). Additionally, when at least one document is acquired, relationship knowledge can be extracted and obtained from the at least one document. Extraction of scene knowledge and extraction of relationship knowledge may be initiated simultaneously or at different times. Therefore, depending on the embodiment, it is possible that the extraction of scene knowledge is performed first and the extraction of relationship knowledge is performed later, or it is also possible that the extraction of scene knowledge and the extraction of relationship knowledge are performed simultaneously. Specifically, for example, the acquisition of scene knowledge may be performed by conducting mutualistic common sense inference about the scene according to Wittgenstein's picture theory. In this case, the process of acquiring scene knowledge may be performed based on part-of-speech tagging, which selects a given concept according to the part of speech of the concept, and stemming, which extracts stems from all token(s) within the concept and description. Additionally, acquisition of relational knowledge can be performed, for example, by finding word(s) corresponding to a concept in a given document and summarizing the distance or frequency between them. Specifically, the similarity between a given concept and a concept extracted from a sentence is calculated using the GloVe word embedding method, and based on this, sentences with high relevance are selected, and sentences with a high relevance score are scored using the score for each document. Relational knowledge can also be obtained by extracting one or more items (e.g., five items) in descending order.

Once scene knowledge and relationship knowledge are acquired, encoding processing is performed (208). Specifically, the encoding process is performed by generating a first sequence corresponding to scene knowledge and a second sequence corresponding to relationship knowledge, and merging the generated first and second sequences to generate an input value to be input to the language model. You can.

The input value obtained through encoding processing is input to the language model, and the language model can be trained based on the input value or generate a result value corresponding to the input value (i.e., a sentence corresponding to the input concept). (210). Here, the language model may include a generative pre-training transformer (GPT), generative pre-training transformer 2 (GPT2), BART, or T5, and may also include various learning models that designers, etc. can consider. .

Depending on the learning process for the language model, the language model may be trained and updated and/or a processing result (i.e., a sentence) corresponding to a given concept may be obtained (212). The obtained sentence may be presented visually or auditorily to the user through an output unit, or may be transmitted to another external information processing device or memory device.

The language generation method according to the above-described embodiment may be implemented in the form of a program that can be driven by a computer device. A program may include instructions, libraries, data files, and/or data structures, etc., singly or in combination, and may be designed and produced using machine code or high-level language code. The program may be specially designed to implement the above-described method, or may be implemented using various functions or definitions that are known and available to those skilled in the art in the computer software field. In addition, here, the computer device may be implemented by including a processor or memory that enables the function of the program, and may further include a communication device if necessary. A program for implementing the above-described language generation method may be recorded on a computer-readable recording medium. Recording media readable by a computer include, for example, semiconductor storage media such as a solid state drive (SSD), ROM, RAM, or flash memory, magnetic disk storage media such as a hard disk or floppy disk, or a compact disk or DVD. It may include at least one type of physical storage medium capable of temporarily or non-temporarily storing one or more programs executed in response to a call from a computer, such as an optical recording medium such as an optical recording medium, or a magneto-optical recording medium such as a floptical disk. .

Although several embodiments of the language generation device and the language generation method based on common sense reasoning have been described above, the language generation device or the language generation method is not limited to the above-described embodiments. Various other devices or methods that can be implemented by those skilled in the art by modifying and modifying the above-described embodiments based on the above-described embodiments may also be examples of the above-described language generation device or language generation method. For example, the described method(s) may be performed in an order other than as described, and/or component(s) of the described system, structure, device, circuit, etc. may be combined, connected, or otherwise in a form other than as described. Even if combined or replaced or replaced by other components or equivalents, it can be an embodiment of the above-described language generation device and/or language generation method.

91: Concept set 93: Image
95: Document 100: Language generation device
101: input unit 105: storage unit
109: output unit 110: processor
120: extraction unit 121: scene knowledge extraction unit
123: Relational knowledge extraction unit 130: Encoding unit
140: Language model processing unit

Claims (14)

an input unit that receives at least one concept, at least one image, and at least one document; and
Using the at least one concept, extracting scene knowledge from the at least one image, extracting relationship knowledge from the at least one document, and learning a language model using the scene knowledge and the relationship knowledge as input values. Includes a processor that performs,
The processor performs generative common sense reasoning according to Wittgenstein picture theory to obtain scene knowledge from the at least one image,
The processor extracts the scene knowledge by selecting concepts that are verbs and nouns among the at least one concept, or extracting stems from all or some tokens in a description of the at least one concept or the at least one image. ,
A language generation device based on common sense reasoning. delete delete According to paragraph 1,
The processor extracts information about a general relationship from the at least one document to expand information about the relationship between the concepts according to Wittgenstein's usage theory. A language generation device based on common sense reasoning. an input unit that receives at least one concept, at least one image, and at least one document; and
Using the at least one concept, extracting scene knowledge from the at least one image, extracting relationship knowledge from the at least one document, and learning a language model using the scene knowledge and the relationship knowledge as input values. Includes a processor that performs,
the processor extracts information about general relationships from the at least one document to expand information about relationships of the concepts according to Wittgenstein's usage theory,
The processor obtains a relevance score for the at least one document, selects a document with a high relevance score from among the at least one document, and decomposes the sentence with the high relevance score into at least one sentence. language production device. According to clause 5,
The processor calculates the similarity between the at least one concept and the concept extracted from the sentence of the document with the high relevance score, and selects sentences with relatively high similarity, thereby selecting sentences based on common sense reasoning. Generating device. According to paragraph 1,
The language model is a general sense inference-based language generation device including a generative pre-trained transformer (GPT), generative pre-trained transformer 2 (GPT2), or BART. In a general sense inference-based language generation method performed by a general sense inference-based language generation device implemented as a computing device including at least a processor,
Receiving at least one concept, at least one image, and at least one document as input;
extracting scene knowledge from the at least one image using the at least one concept;
extracting relational knowledge from the at least one document using the at least one concept; and
Comprising: performing learning processing of a language model using the scene knowledge and the relationship knowledge as input values,
The step of extracting scene knowledge from the at least one image using the at least one concept includes:
Comprising: obtaining scene knowledge from the at least one image by performing generative common sense reasoning according to Wittgenstein picture theory,
The step of performing generative common sense reasoning according to the Wittgenstein picture theory to obtain scene knowledge from the at least one image,
A step of extracting the scene knowledge by selecting concepts that are verbs and nouns among the at least one concept, or extracting stems from all or some tokens in a description of the at least one concept or the at least one image. doing,
A language generation method based on general knowledge reasoning. delete delete According to clause 8,
The step of extracting relational knowledge from the at least one document using the at least one concept includes:
A language generation method based on common sense reasoning, comprising: extracting information about a general relationship from the at least one document in order to expand information about the relationship between the concepts according to Wittgenstein's usage theory. In a general sense inference-based language generation method performed by a general sense inference-based language generation device implemented as a computing device including at least a processor,
Receiving at least one concept, at least one image, and at least one document as input;
extracting scene knowledge from the at least one image using the at least one concept;
extracting relational knowledge from the at least one document using the at least one concept; and
Comprising: performing learning processing of a language model using the scene knowledge and the relationship knowledge as input values,
The step of extracting relational knowledge from the at least one document using the at least one concept includes:
A step of extracting information about a general relationship from the at least one document to expand information about the relationship of the concepts according to Wittgenstein's usage theory,
The step of extracting information about a general relationship from the at least one document includes:
Obtaining a relevance score for the at least one document;
Selecting a document with a high relevance score from among the at least one document; and
Decomposing the sentence with the high relevance score into at least one sentence. A language generation method based on common sense reasoning, including: According to clause 12,
The step of extracting information about a general relationship from the at least one document includes:
calculating a degree of similarity between the at least one concept and a concept extracted from a sentence of a document with a high relevance score; and
A language generation method based on common sense reasoning, further comprising: selecting highly relevant sentences by selecting sentences with relatively high similarity. According to clause 8,
The language model is a general sense inference-based language generation method including a generative pre-training transformer, generative pre-training transformer 2, or BART.