KR102622958B1 - System and method for automatic generation of image caption - Google Patents

Abstract

The present invention relates to an automatic image caption generation system and method that generates captions by extracting attribute information and object information in images using deep learning, and reconstructs the generated captions by predicting relationships between object information.
In an automatic caption generation system for automatically generating captions describing images for images according to an embodiment of the present invention. A client that provides an image for generating the caption, analyzes an image provided by the client to generate a caption describing the image, and transmits the generated caption and the basis for generating the caption to the client. Includes generator.

Description

System and method for automatic generation of image caption}

The present invention relates to an automatic image caption generation system and method. More specifically, the present invention relates to a system and method for automatically generating image captions. More specifically, the present invention relates to a caption generated by extracting attribute information and object information in an image using deep learning, and a caption generated by predicting the relationship between object information. It relates to a system and method for automatically generating image captions for restructuring.

Image captioning involves generating natural language sentences that describe a provided image. Recently, with the advancement of artificial intelligence technology, technology to automatically generate captions using machines has been developed.

In this way, the technology to automatically generate captions using a machine searches for images with the same label or uses the information on many existing images and the label (one word that describes the image) attached to each image. Labels were assigned to an image to create a caption for the image.

Image captioning is the process of creating a caption that describes a provided image using natural language sentences. Recently, with the advancement of artificial intelligence technology, technology to automatically generate captions using machines is being developed.

Automatically generating captions using a machine can be done using many existing images and the label (one word that describes the image) information for each image. In other words, it is possible to create a caption for an image by searching for images with the same label or assigning labels from similar images to one image.

However, in this method, captions are generated using only the image and label data stored for the new image, so it is difficult to generate captions in natural language sentences, and even if generated, the quality of the sentences is poor.

The present invention is intended to solve the problems described above, and is an image caption that generates captions by extracting attribute information and object information in images using deep learning, and reconstructs the generated captions by predicting relationships between object information. The purpose is to provide an automatic generation system and method.

In addition to the technical problems of the present invention mentioned above, other features and advantages of the present invention are described below, or can be clearly understood by those skilled in the art from such description and description.

An automatic caption generation system for automatically generating a caption describing an image according to an embodiment of the present invention to achieve the purpose described above includes a client providing an image for generating a caption, and an image provided by the client. It may include a caption generator that analyzes the image and generates a caption that describes the image, and transmits the generated caption and the basis for generating the caption to the client.

Meanwhile, the automatic caption generation method for automatically generating a caption describing an image according to an embodiment of the present invention to achieve the purpose described above uses deep learning in the caption generation module to generate attribute information and objects within the image. A step of extracting information and generating a caption using attribute information and object information, predicting relationships between objects in the image in the relationship creation module, and creating a tuple set that structures the predicted relationships in the form of a tuple. You can create extended captions by restructuring the captions using the captions and tuple sets created in the description generation module, and visualize the graph for the extended captions and tuple sets.

The automatic image caption generation system and method according to an embodiment of the present invention uses deep learning to generate captions by reflecting attribute information and object information in the image, thereby improving the performance of caption generation for images.

In addition, other features and advantages of the present invention may be newly understood through embodiments of the present invention.

1 is a diagram showing the configuration of an automatic image caption generation system according to an embodiment of the present invention.
Figure 2 is a diagram showing the configuration of a caption generator according to an embodiment of the present invention.
Figure 3 is a diagram showing the configuration of a caption creation module according to an embodiment of the present invention.
Figure 4 is a diagram showing the configuration of a relationship creation module according to an embodiment of the present invention.
Figure 5 is a diagram showing the configuration of a description generation module according to an embodiment of the present invention.
Figure 6 is a diagram showing caption creation for an image according to an embodiment of the present invention.
Figure 7 is a diagram showing extended caption creation according to an embodiment of the present invention.
Figure 8 is a diagram showing a method for automatically generating image captions according to an embodiment of the present invention.
Figure 9 is a diagram showing a method for generating captions according to an embodiment of the present invention.
Figure 10 is a diagram showing a method for generating extended captions according to an embodiment of the present invention.

In order to clearly explain the present invention, parts that are not relevant to the description are omitted, and identical or similar components are assigned the same reference numerals throughout the specification.

The terminology used herein is only intended to refer to specific embodiments and is not intended to limit the invention. As used herein, singular forms include plural forms unless phrases clearly indicate the contrary. As used in the specification, the meaning of "comprising" refers to specifying a particular characteristic, area, integer, step, operation, element and/or ingredient, and the presence or presence of another characteristic, area, integer, step, operation, element and/or ingredient. This does not exclude addition.

Although not defined differently, all terms including technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art in the technical field to which the present invention pertains. Terms defined in commonly used dictionaries are further interpreted as having meanings consistent with related technical literature and currently disclosed content, and are not interpreted in ideal or very formal meanings unless defined.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein.

1 is a diagram showing the configuration of an automatic image caption generation system according to an embodiment of the present invention.

Referring to FIG. 1, the automatic image caption generation system 1000 according to an embodiment of the present invention may include a client 100 and a caption generator 200.

The client 100 may provide an image for generating a caption. The client 100 may provide images to the caption generator 200 through a user device such as a smartphone or tablet PC.

Additionally, the caption generator 200 may analyze an image provided by the client 100, generate a caption describing the image, and transmit the generated caption and the basis for generating the caption to the client 100.

Here, the caption generator 200 can analyze the image through deep learning. Specifically, the caption generator 200 may be learning the image and the correct caption for the image.

The caption generator 200 can generate a caption for a new image using the learned image and the correct captions for the image. The caption generator 200 may generate a caption for an image provided from the client 100 using the learned image and correct captions for the image. Here, the correct caption may be a sentence containing five or more phrases arbitrarily set by the user for the image. Additionally, the caption generator 200 can generate a more expanded caption by extracting objects from a provided image, predicting relationships between objects, and applying the predicted relationships to the generated caption.

The caption generator 200 may transmit the extended caption and the basis for generating the caption to the client 100, and the client 100 may transmit the caption for the image delivered from the caption generator 200 and the basis for generating the caption. You can interpret the results of deep learning. Here, the client 100 and the caption generator 200 may be connected wired or wirelessly.

Figure 2 is a diagram showing the configuration of a caption generator according to an embodiment of the present invention.

Referring to FIG. 2, the caption generator 200 according to an embodiment of the present invention may include a caption generation module 210, a relationship generation module 220, and a description generation module 230.

The caption generation module 210 is learning images and correct captions for images, and can generate captions for provided images using the learned images and correct captions for images. The caption creation module 210 may extract attribute information and object information within an image and generate a caption using the extracted attribute information and object information. Here, attribute information may be words related to the image, and object information may be the core object of the provided image. For example, in the case of an image containing a dog in front of a sofa, attribute information may be 'dog' or 'sofa', and object information may be 'dog' or 'sofa' in the image.

The relationship creation module 220 can predict relationships between objects in an image and generate a tuple set in which the predicted relationships are structured in the form of a tuple. Here, the tuple form is a list of elements, which may be separated by commas ',' within parentheses '( )'. For example, when an image including a dog in front of a sofa is provided, the relationship creation module 220 can predict the relationship between the dog, which is an object, and the sofa. That is, the relationship creation module 220 can predict that a dog is in front of the sofa, and structure the predicted relationship as (sofa, front, dog). At this time. '(sofa, front, dog)' can be a set of tuples.

The description generation module 230 may restructure the caption using the caption generated by the caption generation module 210 and the tuple set generated by the relationship generation module 220 to generate an extended caption. That is, the description generation module 230 may generate a more expanded caption by reflecting the relationships between objects predicted by the relationship generation module 220 in the caption generated by the caption generation module 210. Additionally, the description generation module 230 may visualize an extended caption and a graph of the tuple set that is the basis for generating the caption and transmit it to the client 100.

Figure 3 is a diagram showing the configuration of a caption creation module according to an embodiment of the present invention.

Referring to FIG. 3, the caption creation module 210 according to an embodiment of the present invention may include an attribute extraction model 212, an object recognition model 214, and an image caption model 216.

The attribute extraction model 212 can extract attribute information of a provided image and convert the attribute information into a tuple form. Here, the attribute extraction model 212 may have learned images and captions for the images. That is, the attribute extraction model 212 may have many images and words related to each image mapped and stored in one vector space. Accordingly, the attribute extraction model 212 can output words related to a new image using the stored information and use the output words for learning.

In addition, the attribute extraction model 212 extracts the captions for each image from the captions for each image using words in the form of verbs (or gerunds and participles) in the captions and words in the form of nouns that occur identically three or more times. Words can be extracted from . The attribute extraction model 212 can learn to embed the image and extracted words into one vector space using a deep learning model.

Accordingly, the attribute extraction model 212 can extract words most related to the provided image using the learned image and caption data for the image.

The object recognition model 214 can extract important objects from the image and convert the object area including the extracted objects into a tuple form. The object recognition model 214 may utilize a deep learning-based object recognition model such as the Mask R-CNN algorithm to extract areas corresponding to a predefined object area in the provided image as the object area of the provided image.

The image caption model 216 may generate a caption for a provided image using each word extracted from the attribute extraction model 212 and object regions extracted from the object recognition model 214.

The image caption model 216 is performed using a deep learning algorithm and may be performed based on a recurrent neural network (RNN). Accordingly, the image caption model 216 can predict the relationship between objects in the image in a time series manner.

The image caption model 216 according to an embodiment of the present invention may include an attribute attention model 216a, an object attention model 216b, a grammar learning model 216c, and a language generation model 216d.

The attribute attention model 216a may assign an attention score to words extracted from the attribute extraction model 212. The attribute attention model 216a may assign word attention to a word order that is highly related to the word tag generated by the language generation model 216d at the current time. Here, the word attention level is a value between 0 and 1, and the higher the relevance to the word tag, the closer it may be to 1.

The object attention model 216b may assign region attention to regions of the object extracted from the object recognition model 214. The object attention model 216b may assign region attention to a word order that is highly related to the word tag generated by the language generation model 216d at the current time. Here, the region attention is a value between 0 and 1, and the higher the relevance to the word tag, the closer it may be to 1.

The grammar learning model 216c can learn the grammar of sentences for images and image captions. The grammar learning model 216c can tag each word in the sentence using a grammar tagging tool such as EasySRL for the correct caption sentence of the image and learn the grammar of the correct caption sentence of the image. By learning the grammar of caption sentences, the grammar learning model 216c can ensure that grammatical aspects are taken into consideration when generating captions for provided images.

The language generation model 216d includes words extracted from the attribute extraction model 216a, object regions extracted from the object recognition model 216b, word attention generated from the attribute attention model 216c, and object attention model 216d. ), word tags and grammar tags for captions can be created at each time step based on the region attention generated in ).

The language generation model 216d includes a word attention value, a region attention value, an average vector of words converted into a tuple form by the attribute extraction model 212, and an average of object regions converted into a tuple form by the object recognition model 214. Word tags and grammar tags can be predicted at the current time by considering all compressed information about the vector, words generated by the language generation model 216d at a previous time, and all words generated by the language generation model 216d. The language generation model 216d can compare the predicted word tags and grammar tags with the correct caption sentences and calculate loss values for the generated word tags and grammar tags, respectively. The language generation model 216d may update the learning parameters of the caption generation module 210 by reflecting loss values for word tags and grammar tags.

Accordingly, the language generation model 216d can generate a caption sentence that takes grammar into account for the provided image using word tags and grammar tags.

Figure 4 is a diagram showing the configuration of a relationship creation module according to an embodiment of the present invention.

Referring to FIG. 4, the relationship creation module 220 according to an embodiment of the present invention may include an object extraction model 222, a relationship prediction model 224, and a relationship graph creation model 226.

The object recognition model 222 can extract important object areas within the provided image. The object recognition model 222 can extract important objects from the provided image and extract object areas including the extracted objects.

The relationship prediction model 224 can predict relationships between extracted object areas and structure the relationships between predicted object areas in a tuple form. Here, the relationship prediction model 224 may structure the relationship between predicted object areas in the form of (first noun, predicate, second noun).

The relationship graph generation model 226 can create a graph for the generated tuple set. The relationship graph creation model 226 can generate a graph for sets of tuples, such as displaying an arrow from a first noun to a predicate, and displaying an arrow from a predicate to a second noun.

Figure 5 is a diagram showing the configuration of a description generation module according to an embodiment of the present invention.

Referring to FIG. 5, the description generation module 230 according to an embodiment of the present invention may include a sentence restructuring model 232 and a visualization model 234.

The sentence restructuring model 232 replaces some words with phrases for the tuples according to an algorithm using the caption generated in the caption generation module 210 and the set of tuples generated in the relationship generation module 220, and the generated caption can be expanded. That is, the sentence restructuring model 232 can further expand the caption by reflecting the tuple set generated by the relationship generation module 220 in the caption generated by the caption generation module 210.

The sentence restructuring model 232 may remove tuple sets included in the caption generated by the caption generation module 210 from among the tuple sets generated by the relationship generation module 220. Here, removing a tuple set means that if the first noun, second noun, and predicate in the tuple set are all included in the caption generated by the caption generation module 210, it is determined to be a duplicate tuple set and this duplicate tuple set can be deleted.

The sentence restructuring model 232 can remove duplicate tuple sets and convert the remaining tuple sets into sentence format. Here, when the predicate of the tuple set is a preposition, the sentence restructuring model 232 can be converted into a sentence format by listing the items in the following order: first noun - preposition - second noun. On the other hand, the sentence restructuring model 232 can be converted into a sentence format by listing the predicate of the tuple set in the order of second noun - verb - first noun when the predicate of the tuple set is a verb.

For example, if the tuple set is (sofa, in front, dog), the predicate of the tuple set is a preposition, so the sentence restructuring model 232 can convert the tuple set into 'dog in front of the sofa'. As another example, if the tuple set is (person, lie down, bed), the predicate of the tuple set is a verb, so the sentence restructuring model 232 can convert the tuple set to 'person lying in bed'.

The sentence restructuring model 232 can convert a set of tuples into a sentence format and reflect the converted sentence in the caption. Afterwards, the caption (extended caption) reflecting the converted sentence is compared with the correct caption to calculate the score, and the phrase with the highest score can be selected. The sentence restructuring model 232 can repeat the process of converting a set of tuples into a sentence format - applying it to a caption - selecting a phrase with the highest score until there are no more sets of tuples remaining. Thereafter, the sentence restructuring model 232 may select the last selected phrase as the final extended caption.

The visualization model 234 can visualize the caption expanded from the sentence restructuring model 232 by matching it with a tuple set. The visualization model 234 can generate a graph representing the relationship between the tuple sets by matching the extended caption in the sentence restructuring model 232 with the tuple set. Additionally, the visualization model 234 may transmit a graph representing the relationship between the generated tuple sets to the client 100 so that the user can check the basis for generating the extended caption.

The visualization model 234 may display the object area corresponding to the set of tuples reflected in the caption on the provided image. At this time, the visualization model 234 may display each object area using different colors or different lines (line type or thickness, etc.). Additionally, the visualization model 234 may display phrases corresponding to the object area in the final caption in the same color as the object area. For example, if the final caption sentence is 'a dog in front of a sofa lying on the floor and a cat around a laptop', the visualization model 234 may display the sofa and the dog in the provided image as one object area using a red line. . Additionally, the visualization model 234 may display 'dog in front of the sofa' in red in the final caption sentence. In this way, by displaying corresponding passages and object areas in the same color, the user can recognize them at a glance.

Figure 6 is a diagram showing caption creation for an image according to an embodiment of the present invention.

Referring to FIG. 6, when an image 10 is provided from the client 100, the attribute extraction model 212 may extract attribute information 1 within the provided image 10. The attribute extraction model 212 can extract attribute information (1) in the image 10 provided based on the learned image and the correct caption of the image. As an example, the attribute extraction model 212 can extract a dog, a cat, a floor, etc. as attribute information (1).

Additionally, the object recognition model 214 may extract the object information and object area 2 including the object within the provided image 10 at the same time as the attribute extraction model 212 extracts the attribute information 1. The object recognition model 214 may extract the object area 2 within the provided image 10 based on the learned image and the correct caption of the image. As an example, the object recognition model 214 can extract a dog, a cat, a floor, etc. as object information, and can extract an object area 2 including object information.

Additionally, the image caption model 216 may generate a caption 3 for the provided image 10 using attribute information extracted from the attribute extraction model 212 and object information extracted from the object recognition model 214. As an example, the image caption model 216 may generate a caption 3 that reads 'living room photo of a dog and cat lying on the floor.'

Figure 7 is a diagram showing extended caption creation according to an embodiment of the present invention.

Referring to FIG. 7, the object extraction model 222 can extract object information and an object area 2 including the object within the provided image 10. The object extraction model 222 can extract object information in the provided image 10 based on the learned image and the correct caption of the image. As an example, the object extraction model 222 can extract a dog, a cat, a sofa, a laptop, a door, etc. as object information, and can extract an object area 2 including the object information. At this time, the object extraction model 222 may extract the object area 2 to include two or more extracted objects. Through this, the relationship prediction model 224 can predict the relationship between objects in the object area 2.

The relationship prediction model 224 can predict relationships between objects extracted from the object extraction model 222 and generate relationships between objects as a tuple set 4. As an example, the relationship prediction model 224 can predict that the relationship between 'sofa' and 'dog' extracted as objects is that there is a dog in front of the sofa, and accordingly, (sofa, front, dog) is a tuple set (4 ) can be created. As another example, the relationship prediction model 224 can predict that the relationship between 'cat' and 'door' extracted as objects is that the cat is next to the door, and accordingly, a tuple set (door, next to, cat) is created ( 4) can be created.

The sentence restructuring model 232 can use the tuple set 4 generated by the relationship prediction model 224 to replace some words with phrases for the tuple set according to an algorithm and expand the generated caption 3. there is. That is, the sentence restructuring model 232 can further expand the caption by reflecting the tuple set 4 generated by the relationship generating module 220 in the caption 3 generated by the caption generating module 210. As an example, the sentence restructuring model 232 can expand the caption to 'living room photo of a dog in front of the sofa lying on the floor and a cat around a laptop by the door.'

The relationship graph generation model 226 may generate a relationship graph for the tuple set 4 generated by the relationship prediction model 224. Here, the relationship graph creation model 226 may express the predicate of the tuple set 4 as a square box and the nouns of the tuple set as circular boxes. The relationship graph creation model 226 can connect each box in the following order: first noun - predicate - second noun.

The visualization model 234 may display extended caption passages on the image as object areas, and in this case, each object area may be displayed in a different color. Additionally, the visualization model 234 can visualize passages of expanded captions corresponding to each object area by displaying them in the same color as the corresponding object area.

Figure 8 is a diagram showing a method for automatically generating image captions according to an embodiment of the present invention.

Referring to FIG. 8, the caption creation module 210 may extract attribute information and object information of a provided image and generate a caption by reflecting the attribute information and object information of the extracted image (S100).

The caption creation module 210 may extract attribute information and object information within an image and generate a caption using the extracted attribute information and object information. Here, attribute information may be words related to the image, and object information may be the core object of the provided image. Here, the caption generation module 210 may generate captions for provided images based on images learned through deep learning and captions for each image.

The relationship creation module 220 may predict relationships between objects in an image and generate a set of tuples for the predicted relationships (S200). The relationship creation module 220 may represent the relationship between objects in the image as a tuple set consisting of (first noun, predicate, second noun).

The description generation module 230 may generate an extended caption using the caption generated by the caption generation module 210 and the tuple set generated by the relationship generation module 220 (S300). The description generation module 230 can expand the caption by converting the tuple set into a sentence and reflecting it in the caption.

The description generation module 230 can visualize the extended caption and the relationships between objects by graphing them (S400). The description generation module 230 may generate a graph by matching extended captions and relationships between objects. The description generation module 230 may transmit the generated graph to the client 100 so that the user can check the basis for generating the extended caption.

Figure 9 is a diagram showing a method for generating captions according to an embodiment of the present invention.

Referring to FIG. 9, the attribute extraction model 212 can extract attribute information of an image (S110). Here, the attribute extraction model 212 may have learned images and captions for the images. Accordingly, the attribute extraction model 212 can output attribute information related to a new image using the learned information.

The object recognition model 214 may extract important objects in the image and convert the object area including the extracted objects into a tuple form (S120). The object recognition model 214 may utilize a deep learning-based object recognition model such as the Mask R-CNN algorithm to extract areas corresponding to a predefined object area in the provided image as the object area of the provided image.

The image caption model 216 can assign word attention and area attention to attribute information and object areas extracted from the provided image (S130). The image caption model 216 can assign word attention in the order of words that are highly related to the word tag created at the current time. Here, the word attention level and area attention level are values between 0 and 1, and the higher the correlation with the word tag, the closer to 1.

The image caption model 216 generates word tags for captions at each time step based on attribute information extracted from the attribute extraction model 212, object area extracted from the object recognition model 214, word attention, and area attention. Grammar tags can be predicted (S140). The image caption model 216 can calculate loss values for the generated word tags and grammar tags by comparing the predicted word tags and grammar tags with the correct caption sentences.

The image caption model 216 can generate a caption by reflecting loss values for word tags and grammar tags (S150). Accordingly, the image caption model 216 can generate a caption sentence that takes grammar into account for a provided image using word tags and grammar tags, and learn it.

Figure 10 is a diagram showing a method for generating extended captions according to an embodiment of the present invention.

Referring to FIG. 10, the description generation module 210 may remove tuple sets included in the caption generated by the caption generation module 210 from among the tuple sets generated by the relationship generation module 220 (S310). Here, removing a tuple set means that if the first noun, second noun, and predicate in the tuple set are all included in the caption generated by the caption generation module 210, it is determined to be a duplicate tuple set and this duplicate tuple set can be deleted.

The explanation generation module 210 may remove duplicate tuple sets and convert the remaining tuple sets into sentence format (S320). Here, if the predicate of the tuple set is a preposition, the description generation module 210 can convert it into a sentence format by listing the predicates in the following order: first noun - preposition - second noun. On the other hand, if the predicate of the tuple set is a verb, the description generation module 210 can convert it into a sentence format by listing the tuple set in the following order: second noun - verb - first noun.

The description generation module 210 may reflect the sentences converted from tuple sets into the caption (S330). Afterwards, the caption (extended caption) reflecting the converted sentence is compared with the correct caption to calculate the score, and the phrase with the highest score can be selected. The sentence restructuring model 232 can repeat the process of converting a set of tuples into a sentence format - applying it to a caption - selecting a phrase with the highest score until there are no more sets of tuples remaining. Thereafter, the description generation module 210 may select the last selected phrase as the final extended caption.

The description generation module 210 can visualize the caption expanded from the sentence restructuring model 232 by matching it with a tuple set (S340). The description generation module 210 may generate a graph representing the relationship between the tuple sets by matching the extended caption with the tuple set. Additionally, the visualization model 234 may transmit a graph representing the relationship between the generated tuple sets to the client 100 so that the user can check the basis for generating the extended caption.

As described above, according to an embodiment of the present invention, an image caption is created by extracting attribute information and object information in an image using deep learning, and reconstructing the generated caption by predicting the relationship between object information. An automatic generation system and method can be provided.

Those skilled in the art to which the present invention pertains should understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features, and that the embodiments described above are illustrative in all respects and not restrictive. Just do it. The scope of the present invention is indicated by the claims described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

100: Client 200: Caption Generator
210: Caption creation module 220: Relationship creation module
230: Description generation module 212: Attribute extraction model
214: Object recognition model 216: Image caption model
216a: Attribute attention model 216b: Object attention model
216c: Grammar learning model 216d: Language production model
222: Object extraction model 224: Relationship prediction model
226: Relationship graph generation model 232: Sentence restructuring model
234: Visualization model 226: Relationship graph generation model

Claims (13)

In an automatic caption generation system for automatically generating captions for images that describe the image.
a client providing an image for generating the caption; and
A caption generator that analyzes the image provided by the client, generates a caption describing the image, and transmits the generated caption and the basis for generating the caption to the client,
The caption generator is,
a caption generation module that extracts attribute information and object information in the provided image using deep learning and generates the caption using the attribute information and the object information;
a relationship creation module that predicts relationships between objects in the image and generates a tuple set in which the predicted relationships are structured in the form of a tuple; and
Description of generating an extended caption by restructuring the caption using the caption generated by the caption generation module and the tuple set generated by the relationship generation module, and visualizing a graph for the extended caption and the tuple set. Contains a creation module;
The caption creation module is,
an attribute extraction model that extracts words most related to the provided image and converts each word into a tuple form;
an object recognition model that extracts important objects in the image and converts an object area including the extracted objects into a tuple form;
An image caption model that generates a caption for the image using words extracted from the attribute extraction model and an object area extracted from the object recognition model,
The image caption model is,
an attribute attention model that assigns word attention scores to words extracted from the attribute extraction model;
an object attention model that assigns region attention to regions of the object extracted from the object recognition model;
a grammar learning model that learns the grammar of sentences for the image and the caption of the image; and
A language generation model that generates word tags and grammar tags for captions at each time step based on words extracted from the attribute extraction model, object areas extracted from the object recognition model, the word attention level, and the area attention level. Contains ;,
The language generation model includes the word attention level, the area attention level, the average vector of words converted to tuple form in the attribute extraction model, the average vector of object regions converted to tuple form in the object recognition model, and the language generation. An automatic image caption generation system that predicts word tags and grammar tags at the current time by considering both words generated by the model at a previous time and compressed information about all words generated by the language generation model.
delete delete According to paragraph 1,
The image caption model is performed using a deep learning algorithm and is based on a recurrent neural network (RNN), and an automatic image caption generation system that predicts relationships between objects in the image in time series.
delete According to paragraph 1,
The attribute attention model assigns the word attention to the word order based on its relationship with the word tag generated by the language generation model,
The object attention model assigns the region attention to word order based on its relationship with the word tag generated by the language generation model,
The word attention level and the area attention level are values between 0 and 1, and the higher the relevance to the word tag, the closer to 1 is the automatic image caption generation system.
According to paragraph 1,
The relationship creation module is,
an object recognition model that extracts important object areas within the provided image; and
a relationship prediction model that predicts relationships between the extracted regions and generates a tuple set by structuring the relationships between the predicted regions in a tuple form; and
An automatic image caption generation system including a relationship graph generation model that generates a graph for the generated tuple set.
In clause 7,
The description generation module is,
a sentence restructuring model that replaces some words with phrases for tuples according to an algorithm using the caption generated in the caption generation module and the set of tuples generated in the relationship generation module, and expands the generated caption; and
A visualization model that visualizes the caption expanded from the sentence restructuring model by matching it with the tuple set. An automatic image caption generation system comprising a.
In the automatic caption generation method for automatically generating a caption describing an image,
extracting attribute information and object information in an image using deep learning in a caption generation module, and generating the caption using the attribute information and the object information;
Predicting relationships between objects in the image in a relationship creation module and generating a tuple set in which the predicted relationships are structured in the form of a tuple; and
Restructuring the caption in a description generation module using the generated caption and the tuple set to generate an extended caption, and visualizing a graph for the extended caption and the tuple set,
The step of generating the caption is,
extracting words most related to the image from the caption generation module and converting each word into a tuple form;
extracting important objects in the image from an object recognition model and converting an object area including the extracted objects into a tuple form; and
Generating a caption for the image using the extracted words and the extracted object area from an image caption model,
The step of generating a caption for the image is,
assigning word attention scores to words extracted from the image caption model;
assigning region attention to object regions extracted from the image caption model;
predicting word tags and grammar tags for captions at each time step based on the attribute information, the object areas, the word attention level, and the area attention level; and
An image caption automatic generation method comprising: generating a caption by reflecting loss values for word tags and grammar tags.
delete delete The method of claim 9, wherein generating the tuple set comprises:
extracting important object areas within the image from an object recognition model;
predicting relationships between the extracted regions in a relationship prediction model, and structuring the relationships between the predicted regions in a tuple form to generate a tuple set; and
An image caption automatic generation method further comprising: generating a graph for the generated set of tuples in a relationship graph generation model.
The method of claim 9, wherein visualizing a graph for the set of tuples comprises:
In the sentence restructuring model, replacing some words with phrases for tuples according to an algorithm using the tuple set generated in the caption and relationship generation module and expanding the generated caption; and
A method for automatically generating image captions, further comprising matching the expanded caption with the tuple set and visualizing it in a visualization model.

Images