KR20220141521A - Apparatus and method for generating query - Google Patents

Abstract

A method for generating a query performed by a device for generating a query according to an embodiment comprises the steps of: receiving an image; inputting the image into a pre-trained deep learning model to extract features of the image; generating at least one word using the features; generating a first query having the at least one word combined therein; and generating a second query so that the first word used in the first query cannot be included with a preset probability, wherein the pre-trained deep learning model may be pre-trained with a plurality of images and a plurality of questions for each of the plurality of images, which are label data for each of the plurality of images, as input. Therefore, various kinds of creative queries can be generated for an input image.

Description

Apparatus and method for generating a query statement {APPARATUS AND METHOD FOR GENERATING QUERY}

The present invention relates to an apparatus and method for generating a query.

Image captioning technology is an artificial intelligence technology that receives given visual information (image or video) as input and creates a natural language sentence describing the visual information.

Such image captioning technology is being utilized in technologies such as automatic news article generation, document summary, and photo information description.

However, since the current image captioning technology generates only simple types of questions focused only on the description of the image, such as a general description of the image, it is not possible to generate various kinds of creative questions.

Korean Patent Publication No. 10-2014-0066476 (published on Jun. 2, 2014)

SUMMARY OF THE INVENTION It is an object of the present invention to provide an apparatus and method for generating a query.

In addition, generating various kinds of creative questions with respect to the input image may be included in the task to be solved by the present invention.

However, the problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be.

A method for generating a query according to an embodiment of the present invention comprises the steps of: receiving an image; inputting the image into a pre-trained deep learning model; extracting a feature of the image, generating at least one word by using the feature, generating a first query sentence combining the at least one word, and the first query used in the first query sentence. generating a second query so that the word is not included with a preset probability, wherein the pre-trained deep learning model is a plurality of images for each of the plurality of images as label data of each of the plurality of images It may be pre-learned by inputting a question of.

In addition, the generating of the at least one word may include generating the first word using the feature, and generating a second word using the first word and features of the image.

In addition, the generating of the at least one word may include generating the second word to be time-series connected to the first word.

Also, in the generating of the at least one word, when a word including a punctuation mark indicating the end of a predefined sentence is generated, word generation may be terminated.

In addition, the pre-trained deep learning model may be a deep learning model to which a transformer network structure is applied.

An apparatus for generating a query according to an embodiment of the present invention includes an input/output unit for receiving an image; Memory; and a processor electrically connected to the memory, wherein the processor inputs the image to a pre-trained deep learning model, extracts features of the image, uses the features to generate at least one word, , generating a first query by combining the at least one word, generating a second query so that the first word used in the first query is not included with a preset probability, and the pre-trained deep learning model includes a plurality of It may be pre-learned by inputting a plurality of questions for each of the plurality of images as the image of and label data of each of the plurality of images.

The apparatus for generating a query according to an embodiment of the present invention may generate various types of creative questions with respect to an input image.

In addition, the apparatus for generating a question for an embodiment of the present invention uses various types of generated creative questions as a learning data set of a deep learning learning model used in Visual Question Answering (VQA) based on visual information. can be used, thereby reducing the cost of constructing the training data set.

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present disclosure belongs from the description below. will be able

1 is a block diagram of an apparatus for generating a query according to an embodiment of the present invention.
2 is a diagram for explaining an example of a method of generating a query using a pre-trained deep learning model according to an embodiment of the present invention.
3 is an exemplary flowchart of a procedure of a method for generating a query according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

In describing the embodiments of the present invention, if it is determined that a detailed description of a well-known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. In addition, the terms to be described later are terms defined in consideration of functions in an embodiment of the present invention, which may vary according to intentions or customs of users and operators. Therefore, the definition should be made based on the content throughout this specification.

1 is a block diagram of an apparatus for generating a query according to an embodiment of the present invention.

Referring to FIG. 1 , an apparatus 100 for generating a query according to an embodiment of the present invention may include an input/output unit 101 , a communication unit 102 , a memory 110 , and/or a processor 120 .

The input/output unit 101 transmits, for example, a command or data input from a user or other external device to other component(s) of the query generating apparatus 100 according to an embodiment, or, according to an embodiment A command or data received from other component(s) of the apparatus 100 for generating a query according to the above may be output to a user or an external device.

For example, the input/output unit 101 may receive an image, but is not limited thereto, and may receive an image as a dynamic image.

The communication unit 102 may support establishment of a wired or wireless communication channel between the query generating apparatus 100 and an external device and performing communication through the established communication channel.

The memory 110 includes various data used by at least one component (the processor 120, the input/output unit 101, and/or the communication unit 102) of the query generating apparatus 100, for example, software ( Example: program) and input data or output data for commands related thereto can be stored. The memory 110 may include a volatile memory or a non-volatile memory.

The processor 120 (also referred to as a control unit, a control device, or a control circuit) includes at least one other component (eg, a hardware component (eg, an input/output unit 101 , a communication unit 102 ) of the connected query generating device 100 . and/or the memory 110) or a software component), and may perform various data processing and operations.

In addition, the processor 120 may load and process commands or data received from at least one of the other components into the volatile memory, and store various data in the non-volatile memory.

To this end, the processor 120 executes one or more software programs stored in a dedicated processor (eg, an embedded processor) or a memory device for performing the corresponding operation, thereby performing the corresponding operation (generic-purpose processor). processor) (eg, a CPU or an application processor or a micro controller unit (MCU), etc.).

More specifically, the processor 120 may input the image received from the input/output unit 101 into the pre-trained deep learning model to extract features of the image.

Here, the pre-trained deep learning model may be a deep learning model to which a transformer network structure based on a multi-head attention layer is applied.

Hereinafter, the processor 120 may generate at least one word related to the image input from the input/output unit 101 using the pre-learned deep learning model.

In this case, the processor 120 generates at least one word related to the image input from the input/output unit 101 using a pre-learned machine learning model stored in the memory 110 , or the processor 120 uses another external word. After loading the pre-learned machine learning model from the device, at least one word related to the image input from the input/output unit 101 may be generated using the pre-learned machine learning model.

In more detail, the processor 120 may generate at least one word using the features of the extracted image, but is not limited thereto, and the processor 120 may generate at least one word using the features of the extracted image. .

For example, the processor 120 may generate a first word using the features of the extracted image, and may generate the second word using the generated first word and the features of the image.

In this case, the processor 120 may generate the second word to be time-series connected to the first word.

That is, the processor 120 may first generate a word using the features of the extracted image, and then generate a plurality of words using the previously generated word and the features of the extracted image. Also, the processor 120 may generate a plurality of generated words to be time-series connected to previously generated words.

Meanwhile, when a word including a punctuation mark indicating the end of a predefined sentence is generated, the processor 120 may end word generation.

Here, the punctuation marks indicating the end of a predefined sentence may include a period (.), a question mark (?), an exclamation mark (!), and the like, but is not limited thereto.

The processor 120 may generate a first query by combining at least one generated word, and then generate a second query such that the first word used in the first query is not included with a preset probability.

For example, the processor 120 may generate a preset number of queries in addition to the first and second queries. For example, the processor 120 includes the first word used in the second query with a preset probability. You can create a third query to prevent this from happening.

In more detail, after the first query is generated, the processor 120 generates a first word to be included in the second query using the features of the extracted image, and the first word to be included in the second query has a preset probability (eg, 85% chance), the first word used in the first query may not be generated.

For example, when the processor 120 generates the first word to be included in the second query, since the probability that the first word used in the first query is not generated is set to 85%, the processor 120 generates It is highly likely that the first word of the second query does not overlap with the first word of the first query.

Therefore, the possibility that the query generating apparatus 100 according to an embodiment of the present invention generates a duplicate query is low, and the first word of the query generated after the previous query is generated is the first word of the previously generated query. and redundancy is low, so various types of query statements can be created.

Hereinafter, an example of a method of generating a query using a pre-learned deep learning model will be described in detail with reference to FIG. 2 .

2 is a diagram for explaining an example of a method of generating a query using a pre-trained deep learning model according to an embodiment of the present invention.

Referring to FIG. 2 , when an image 200 is first inputted from the input/output unit 101 , the received image is input to the pre-trained deep learning model 220 , so that the image is divided into preset sizes. A pre-processing process 210 is performed.

Thereafter, the image on which the data preprocessing has been performed is input to the pre-trained deep learning model 220 , and the pre-trained deep learning model 220 receives the image on which the data pre-processing has been performed, and outputs at least one word. (Output Probabilities) is possible.

Here, the pre-trained deep learning model 220 may be pre-learned by inputting a plurality of questions for each of a plurality of images as label data of a plurality of images and each of the plurality of images.

In more detail, the pre-trained deep learning model 220 may include an encoder unit 213 and a decoder unit 225 , and the encoder unit 213 receives an image on which data pre-processing has been performed, and features of the image may be extracted, and the decoder unit 235 may receive the features extracted from the encoder unit 213 and generate at least one word.

Here, the decoder unit 235 receives the characteristics of the image extracted from the encoder unit 213 and passes through each layer of the LSTM (Long Short Term Memory) network structure, which is a time series-based network structure after the Transformer network structure. Accordingly, it is possible to generate at least one word most related to the characteristics of the input image based on the time series.

For example, when input/output unit 101 receives an image 200 in which a banana is located on a woman's face and the location of the banana has a mustache shape on the woman's face, the input image 200 is the image A data pre-processing process 210 such as division into a preset size is performed, and the image on which the data pre-processing process is performed may be input to the pre-trained deep learning model 220 .

In this case, the pre-learned deep learning model 220 may receive an image on which data preprocessing has been performed, and may output at least one word (Output Probabilities).

For example, when the pre-trained deep learning model 220 generates “any” as the first word, “any” is the second word in consideration of the first word “any” and the characteristics of the image extracted from the encoder unit 233 . Colored” may be generated, and “mustache” may be generated as the third word in consideration of the first word (“some”), the second word (“colored”), and features of the image extracted from the encoder unit 233 . Thereafter, the pre-trained deep learning model 220 considers the first word (“some”), the second word (“colored”), the third word (“mustache”), and the features of the image extracted from the encoder unit 233 . to produce "is it?" as the fourth word.

Here, the pre-trained deep learning model 220 is a punctuation mark indicating the end of a predefined sentence, and since a word including a question mark (?) is generated in the fourth word (“is it?”), word generation is terminated. can do.

Thereafter, the first query may be generated by sequentially combining the four words generated by the deep learning model 220 previously learned by the processor 120 .

Thereafter, the pre-trained deep learning model 220 may again generate the first word for the second query, where the first word of the second query has a preset probability (eg, for example, 85% chance).

3 is an exemplary flowchart of a procedure of a method for generating a query according to an embodiment of the present invention. The method for generating a query of FIG. 3 may be performed by the apparatus 100 for generating a query shown in FIG. 1 . In addition, the method of generating a query shown in FIG. 3 is merely exemplary.

Referring to FIG. 3 , the input/output unit 101 may receive an image (step S10 ).

Thereafter, the processor 120 may input the image received from the input/output unit 101 into the pre-trained deep learning model to extract features of the image (step S20 ).

Here, the pre-trained deep learning model may be pre-trained by inputting a plurality of images and a plurality of questions for each of the plurality of images as label data of each of the plurality of images.

Thereafter, the processor 120 may generate at least one word by using the extracted features (step S30 ).

For example, the processor 120 may generate a first word using the features of the extracted image, and may generate a second word using the generated first word and the features of the image. In this case, the processor 120 may generate a second word Words can be generated to be connected in time series with the first word.

Also, when a word including a punctuation mark indicating the end of a predefined sentence is generated, the processor 120 may end word generation.

Thereafter, the processor 120 may generate a first query by combining at least one generated word (step S40 ).

Finally, the processor 120 may generate the second query so that the first word used in the first query is not included with a preset probability (step S50 ).

For example, the processor 120 may generate a preset number of queries in addition to the first and second queries. For example, the processor 120 determines that the third query is the first word used in the second query. A third query can be created so that it is not included with a set probability.

As described above, the apparatus for generating a query according to an embodiment of the present invention may generate various types of creative questions with respect to an input image.

In addition, the apparatus for generating a question for an embodiment of the present invention uses various types of generated creative questions as a learning data set of a deep learning learning model used in Visual Question Answering (VQA) based on visual information. can be used, thereby reducing the cost of constructing the training data set.

Meanwhile, the apparatus 100 for generating a question according to an embodiment of the present invention can be used in various platforms, for example, an artificial intelligence service system that can provide a question that can be recognized (or thought) by viewing visual information. etc. can be used.

Combinations of each block in the block diagram attached to the present invention and each step in the flowchart may be performed by computer program instructions. These computer program instructions may be embodied in the encoding processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions executed by the encoding processor of the computer or other programmable data processing equipment may correspond to each block or Each step of the flowchart creates a means for performing the functions described. These computer program instructions may also be stored in a computer-usable or computer-readable memory that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable memory. The instructions stored in the block diagram may produce an article of manufacture containing instruction means for performing the functions described in each block in the block diagram or in each step in the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for carrying out the functions described in each block of the block diagram and each step of the flowchart.

Further, each block or each step may represent a module, segment, or portion of code that includes one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in blocks or steps to occur out of order. For example, it is possible that two blocks or steps shown one after another may in fact be performed substantially simultaneously, or that the blocks or steps may sometimes be performed in the reverse order according to the corresponding function.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations will be possible without departing from the essential quality of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: query generating device
101: input/output unit
110: memory
120: processor

Claims (8)

A method for generating a query performed by an apparatus for generating a query, the method comprising:
receiving an image, and
inputting the image to a pre-trained deep learning model, and extracting features of the image;
generating at least one word using the feature;
generating a first query sentence combining the at least one word;
generating a second query so that the first word used in the first query is not included with a preset probability;
The pre-trained deep learning model is,
Pre-learned by inputting a plurality of questions for each of the plurality of images as label data of a plurality of images and each of the plurality of images
How to create a query. The method of claim 1,
The generating of the at least one word comprises:
generating the first word using the feature, and generating a second word using the first word and features of the image
How to create a query. 3. The method of claim 2,
The generating of the at least one word comprises:
The second word is generated to be time-series connected with the first word
How to create a query. The method of claim 1,
The generating of the at least one word comprises:
When a word including a punctuation mark indicating the end of a predefined sentence is generated, the word generation is terminated.
How to create a query. The method of claim 1,
The pre-trained deep learning model is,
It is a deep learning model to which the Transformer network structure is applied.
How to create a query. an input/output unit for receiving an image;
Memory; and
a processor electrically connected to the memory;
The processor is
Input the image to a pre-trained deep learning model, extract features of the image, use the features to generate at least one word, and generate a first query that combines the at least one word, generating a second query so that the first word used in the first query is not included with a preset probability;
The pre-trained deep learning model is,
Pre-learned by inputting a plurality of questions for each of the plurality of images as label data of a plurality of images and each of the plurality of images
Query generator. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
receiving an image, and
inputting the image to a pre-trained deep learning model, and extracting features of the image;
generating at least one word using the feature;
generating a first query sentence combining the at least one word;
and instructions for causing the processor to perform a method including generating a second query so that the first word used in the first query is not included with a preset probability;
The pre-trained deep learning model is,
Pre-learned by inputting a plurality of questions for each of the plurality of images as label data of a plurality of images and each of the plurality of images
computer readable recording medium. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
receiving an image, and
inputting the image to a pre-trained deep learning model, and extracting features of the image;
generating at least one word using the feature;
generating a first query sentence combining the at least one word;
and instructions for causing the processor to perform a method including generating a second query so that the first word used in the first query is not included with a preset probability;
The pre-trained deep learning model is,
Pre-learned by inputting a plurality of questions for each of the plurality of images as label data of a plurality of images and each of the plurality of images
computer program.

Images