JP6877374B2 - How to train a model that outputs a vector that represents the tag set that corresponds to the image - Google Patents

Description

The present invention relates to a method of training a model that outputs a vector representing a tag set corresponding to an image.

A technique for adding a tag representing an image to an image is known for image analysis and retrieval. For example, Patent Document 1 or Patent Document 2 discloses a system for estimating tags related to images.

In Patent Document 1, "First, during training, a clustering technique is used to reduce the cluster imbalance of data input to a convolutional neural network (CNN) to train feature data. , Clustering techniques can also be used to calculate data point similarity that can be used for tag propagation (for tagging untagged images). During testing, a diversity-based voting framework It is used to overcome user tagging bias. In some embodiments, bigram reweighting can reduce the weighting of keywords that are likely to be part of the bigram based on a predictive tag set. It can be done. ”(Summary).

In Patent Document 2, "systems, methods, and non-transitory computer-readable media may generate a first content item representation of a first content item based on a first content item transformation in the training stage. The first content item can include one or more images and videos. The first user metadata representation of the first user metadata is created based on the first user metadata transformation. The first content item representation and the first user metadata representation can be combined to generate the first combined representation. The first combination representation and the first tag representation of the first tag. Can be embedded in an embedded space within a first threshold distance from each other. "(Summary).

U.S. Patent Application Publication No. 2017/0236055 U.S. Patent Application Publication No. 2016/01885992

By the technique of machine learning, it is possible to form a model that outputs one or a plurality of related tags (tag sets) for an input image. In order to properly train a model by machine learning, a large amount of training data is required, and in general, the frequency distribution of tags associated with image data has a long tail attribute. That is, the frequency of appearance of many tags (the number of images associated with them) is low, and the frequency of appearance of only some tags is high.

It is difficult to collect many training data for infrequently occurring tag sets. On the other hand, a system that estimates a tag set to be associated with an image is required to have the ability to estimate with high accuracy even a tag set that appears infrequently.

One aspect of the disclosure is a method performed on a computer, comprising training a first model that outputs a vector representing a set of tags to be associated with the input image from the input image, wherein the computer stores. The first model comprises from the input image an encoder that outputs a vector representing a tag set to be associated with the input image, and from the encoder, including a device and a processor that operates according to a program stored in the storage device. In the method, the processor inputs a first training image to the encoder, acquires a first output image from the decoder, and includes a first output vector from the encoder. The first from the encoder, which is preset based on the error between the first training image and the first output image and the first tag set preliminarily associated with the first training image. The parameters of the first model are updated based on the constraints on one output vector.

According to one aspect of the present disclosure, it is possible to obtain a model capable of estimating the tag set to be associated with the image with high accuracy.

A configuration example of a computer system including a tag estimation device is shown. Shows an example of an image-text relational database configuration An example of the configuration of an image-tag relational database is shown. An example of a GUI image for a user to acquire a tag set corresponding to an input image is shown. An example of a GUI image for a user to acquire an image corresponding to an input tag set is shown. The flowchart of the process of determining the recommended tag set for the input image by the operation program is shown. The flow chart of the process of making an image related to the input tag set by the operation program is shown. The flowchart of the process by the image-tag-related data generation program is shown. An example of tag distribution table configuration is shown. It is a flowchart which shows the method of training (learning) of a matching model by a training program. It is a flowchart of the processing of the matching model in training. A configuration example is schematically shown in the semantic expression model. The input data and output data in the training of the semantic expression model are shown. It is a flowchart of the training of the semantic expression model by the training program.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the present embodiment is merely an example for realizing the present invention and does not limit the technical scope of the present invention. The same reference numerals are given to common configurations in each figure.

FIG. 1 shows a configuration example of a computer system including a tag estimation device 100. The tag estimation device 100 includes a processor 110, a memory 120, an auxiliary storage device 130, an input / output interface 140, and a network (NW) interface 145. The components are connected to each other by a bus. The memory 120, the auxiliary storage device 130, or a combination thereof is an example of a storage device.

The memory 120 is composed of, for example, a semiconductor memory, and is mainly used for temporarily holding a program or data. The program stored in the memory 120 includes a matching model 121, a semantic expression model 122, a training program 123, an operation program 124, a data crawler 125, and an image-tag related data generation program 126.

The memory 120 further stores a tag distribution table 547 created in the image-tag related data generation program 126. The tag distribution table 547 may also be stored in the auxiliary storage device 130.

The processor 110 executes various processes according to the program stored in the memory 120. When the processor 110 operates according to the program, various functional units are realized. For example, the processor 110 operates as a matching model unit, a semantic expression model unit, a learning unit, an operation unit, a data crawler unit, and a training data generation unit according to each of the above programs.

The auxiliary storage device 130 stores an image-text-related database (DB) 132 and an image-tag-related database 134. The auxiliary storage device 130 is composed of a large-capacity storage device such as a hard disk drive or a solid state drive, and is used for holding programs and data for a long period of time.

The program stored in the auxiliary storage device 130 is loaded into the memory 120 at startup or when necessary, and the processor 110 executes this program to execute various processes of the tag estimation device 100. Therefore, the processing executed by the program in the following is the processing performed by the processor 110 or the tag estimation device 100.

The network interface 145 is an interface for connecting to a network. In the example of FIG. 1, the tag estimation device 100 connects to the Internet via the network interface 145.

The client device 144 is a device used by the user, and accesses the tag estimation device 100 via the network and the Internet in the example of FIG. The client device 144 has, for example, a general computer configuration and includes an input device and a display device. The input device is a hardware device for the user to input instructions, information, and the like to the tag estimation device 100. The display device is a hardware device that displays various images for input and output. The input device and the display device may be connected to the tag estimation device 100 without going through a network.

The matching model 121 and the semantic expression model 122 are models trained (updated) by machine learning. The tag estimation device 100 has a training mode (learning mode) and an operation mode for the matching model 121. It also has a training mode and an operation mode for the semantic representation model 122.

The matching model 121 and the semantic expression model 122 are each trained by the training program 123 in the training mode. The image-tag relational database 134 is used for training the matching model 121 and the semantic representation model 122.

The matching model 121 is used by the image-tag related data generation program 126 in the operation mode. The matching model 121 is used in the operation mode to generate data to be stored in the image-tag relational database 134 from the data stored in the image-text relational database 132.

The semantic representation model 122 is used by the operation program 124 in the operation mode. The semantic representation model 122 is used in the operation mode to estimate a tag set consisting of one or more tags corresponding to an image input to the user. Data from the image-tag relational database 134 is used to estimate the tag set.

In addition, the semantic representation model 122 is used to estimate one or more images corresponding to the tag set input to the user. Data from the image-text relational database 132 is used for image estimation. The image estimation may use the data of the image-tag relational database 134 without using the semantic representation model 122 and the image-text relational database 132.

The data crawler 125 periodically crawls Web pages on the Internet and collects pairs of images and text associated with each other. The data crawler 125 stores the collected image and text pairs in the image-text relational database 132.

FIG. 2 shows a configuration example of the image-text relational database 132. The image-text related database 132 associates an image with the corresponding text. It has an image column 321 and a text column 322. The image column 321 stores the collected images. The text column 322 stores the text associated with each image. Each text is composed of multiple words.

As mentioned above, the image-text relational database 132 is used in the generation of the image-tag relational database 133 and in the estimation of the image corresponding to the user input tag set.

FIG. 3 shows a configuration example of the image-tag relational database 133. The image-tag relational database 133 associates an image with a corresponding tag set. The image-tag relational database 133 has an image column 331 and a tag column 332. The image column 321 stores an image. The tag column 332 stores a tag set associated with each image. Each tag set is composed of one or more tags (words).

As mentioned above, the image-tag relational database 133 is used for the matching model 121 and the semantic representation model 122, and is also used to estimate the appropriate tag set representing the user-entered image.

In the following, the estimation of one or a plurality of images corresponding to the tag set input to the user and the estimation of the tag set corresponding to the image input to the user will be described. The operation program 124 uses the semantic representation model 122 to estimate one or more images corresponding to the input tag set, and also estimate the tag set corresponding to the input image.

FIG. 4A shows an example of a GUI image for the user to acquire a tag set corresponding to the input image. The operation program 124 transmits the image data shown in FIG. 4A to the client device 144. In the client device 144, the user inputs the path of the target image in the field 401 and presses the UPLOAD button 402.

The client device 144 acquires the input image (data) from the path indicated by the field 401 and transmits it to the tag estimation device 100. The operation program 124 receives the target image from the client device 144.

The client device 144 transmits a request for a tag set representing the transmitted image to the tag estimation device 100 in response to the user's selection of the "estimate" button 403 via the input device. The operation program 124 estimates the tag set related to the target image and transmits it to the client device 144 in response to the received request. The client device 144 displays the received tag set in the "Recommended Tags" section 404.

FIG. 4B shows an example of a GUI image for the user to acquire an image corresponding to the input tag set. The operation program 124 transmits the image data shown in FIG. 4B to the client device 144. The user inputs the target tag set in the field 451 in the client device 144. The client device 144 transmits a request for searching an image related to the target tag set to the tag estimation device 100 together with the target tag set in response to the selection of the "search" button 452 by the user via the input device.

The operation program 124 selects one or a plurality of images presumed to be related to the target tag set according to the received request and transmits the images to the client device 144. The client device 144 displays the received one or more images in the "related images" section 454.

FIG. 5 shows a flowchart of the process of determining the recommended tag set for the input image by the operation program 124. The operation program 124 acquires an image input to the user (S101). The operation program 124 generates a semantic expression vector from the acquired image by the semantic expression model 122 (S102). The semantic expression model 122 generates a semantic expression vector of the image from the input image. Details of the configuration and operation of the semantic expression model 122 will be described later.

As will be described later, the semantic expression vector can be regarded as a vector representing a tag set to be associated with the input image, that is, a vector representing a tag set representing an image. In the examples described below, the semantic representation vector corresponds to the vector generated using the word embedding technique.

The operation program 124 sequentially selects tags from the image-tag relational database 134 and generates a vector (tag vector) thereof (S103). In this example, the operation program 124 converts each tag into a tag vector by the word embedding technology. Operational program 124 includes a pre-trained model of word embedding. In other implementations, a word containing a tag may be converted into a vector by a technique different from the word embedding technique.

The operation program 124 compares each of the generated tag vectors with the semantic expression vector, and calculates the degree of similarity with the semantic expression vector (S104). For the calculation of similarity, for example, dot product or cosine similarity can be used.

The operation program 124 determines the tags to be included in the recommended tag set corresponding to the target image based on the similarity (S105). The operation program 124 constitutes a recommended tag set, for example, by tags having a similarity higher than the threshold value. The maximum number of tags that make up the recommended tag set may be preset. When the number of tags whose similarity exceeds the threshold value exceeds the specified number, the operation program 124 may select the specified number of tags from the tags having the highest similarity in the order of similarity. The operation program 124 may select a specified number of tags from the tags having the highest similarity in the order of similarity without referring to the similarity threshold.

As described above, the semantic representation model 122 can be used to estimate the tag set corresponding to the input image with high accuracy. By comparing the vector of each tag with the semantic expression vector, the tag to be associated with the image can be estimated more accurately.

The operation program 124 may compare a vector of a tag set composed of a plurality of tags, for example, a tag set of each record of the image-tag relational database 134 with a semantic expression vector. A single tag is also a set of tags consisting of a single tag. Instead of the image-tag relational database 134, a database that stores only tags is prepared, and the operation program 124 may compare the tag vector in the database with the semantic expression vector.

FIG. 6 shows a flowchart of a process of creating an image related to an input tag set by the operation program 124. The operation program 124 acquires the tag set input to the user (S151).

The operation program 124 generates a vector (tag set vector) from the acquired tag set by using the word embedding model (S152). The operation program 124 generates the tag vector of each tag constituting the tag set by the word embedding model, and generates the tag set vector of the tag set by combining them.

For example, the operation program 124 may determine the simple average value of the tag vector as the tag set vector, or may determine the weighted average of the tag vector as the tag set vector. The tag weight is determined, for example, according to the frequency of occurrence of the tag in the image-tag relational database 134. The larger the number of images associated in the image-tag relational database 134, the greater the weight given to the tag.

Next, the operation program 124 sequentially reads images from the image-text relational database 132, and generates those semantic expression vectors by the semantic expression model 122 (S153). The operation program 124 calculates the similarity between each of the generated semantic expression vectors and the tag set vector generated in step S152 (S154). For the calculation of similarity, for example, dot product or cosine similarity can be used.

The operation program 124 determines an image presumed to be related to the target tag set based on the similarity (S155). The operation program 124 determines, for example, an image having a similarity higher than the threshold value as an image to be presented as an image related to the tag set. The maximum number of images to be presented may be preset. When the number of images whose similarity exceeds the threshold value exceeds the specified number, the operation program 124 may select a specified number of images from the images having the highest similarity in the order of similarity. The operation program 124 may select a specified number of images from the images having the highest similarity in the order of similarity without referring to the similarity threshold.

As described above, the semantic representation model 122 can be used to estimate the image associated with the input tag set with high accuracy. In step S153, the operation program 124 may refer to the image-tag relational database 134 instead of the image-text relational database 132. The operation program 124 generates a tag set vector for each tag set of the records in the image-tag relational database 134, and calculates the degree of similarity with the tag set vector generated in step S152.

Next, a process of generating data to be stored in the image-tag-related database 134 from the data of the image-text-related database 132 by the image-tag-related data generation program 126 will be described. FIG. 7 shows a flowchart of processing by the image-tag related data generation program 126. The image-tag-related data generation program 126 executes the process shown in FIG. 7 periodically, for example, once a week.

The image-tag-related data generation program 126 analyzes the tag appearance frequency distribution in the image-tag-related database 134 and generates a tag distribution table 547 (S201). The frequency of occurrence of a tag is equal to the number of images associated with that tag in the image-tag relational database 134.

FIG. 8 shows a configuration example of the tag distribution table 547. The tag distribution table 547 includes a tag column 471 and an image number column 472. The tag column 471 indicates a tag stored in the image-tag relational database 134. The number of images column 472 indicates the number of images each tag is associated with in the image-tag relational database 134.

Returning to FIG. 7, the image-tag-related data generation program 126 refers to the tag distribution table 547 and selects a tag having a low appearance frequency (S202). For example, the image-tag-related data generation program 126 selects a specified number of tags from the tag having the lowest appearance frequency in the order of appearance frequency. The image-tag-related data generation program 126 may select all tags whose appearance frequency is less than the specified value.

The image-tag relational data generation program 126 selects a pair (record) of an image and a text including any of the selected tags in the text from the image-text relational database 132 (S203).

The image-tag association data generation program 126 uses the matching model 121 to calculate the similarity between each word in the text associated with the image in each selected set of image and text. (S204).

Specifically, the image-tag-related data generation program 126 generates a feature vector of the image and a vector of each word. The feature vector of the image can be generated, for example, by a pre-trained deep learning model (deep neural network). The word vector is generated by a pre-trained word embedding model, as described above.

The image-tag-related data generation program 126 inputs the generated image feature vector and one word vector into the matching model 121. The matching model 121 outputs the degree of similarity between the input image feature vector and the word vector. Details of the processing of the matching model 121 will be described later.

The image-tag-related data generation program 126 selects a word to be registered as a tag among the words in the text based on the similarity with the image in each set of the image and the text (S205). The image-tag-related data generation program 126 selects, for example, words having a similarity higher than the threshold value as image tags. In this way, words are ranked according to their similarity to the image.

The maximum number of tags that can be selected may be preset. The image-tag-related data generation program 126 selects one or more words from the word having the highest similarity in the order of similarity from the word having the similarity higher than the threshold value, up to the specified maximum number. The number of tags that can be selected may be preset. The image-tag-related data generation program 126 selects a specified number of words from the words having the highest similarity in the order of similarity.

One or more words selected as tags constitute a set of tags that represent the image (associate with the image). The image-tag-related data generation program 126 adds the pair of the image and the selected tag set to the image-tag-related database 134 (S206).

As described above, by selecting a tag from the text actually associated with the image, the tag associated with the image can be determined with higher accuracy. As will be described later, the image-tag relational database 134 is used for training the semantic representation model 122, and the semantic representation model 122 can be appropriately trained. Also, by using the matching model 121, which is improved by training, it is possible to generate more appropriate data to be added to the image-tag relational database 134.

By determining the image to be added to the image-tag relational database 134 based on the tag distribution table 547, the appearance frequency distribution of the tags in the image-tag relational database 134 can be made closer to more uniform. By selecting an image that contains a low frequency tag in the associated text, it is possible to increase the possibility of selecting an image to which the low frequency tag is associated.

Thereby, the data of the image (rare image) whose related tag set is generally rare can be included in the image-tag relational database 134, and the semantic expression model 122 can be trained so that the tag set can be estimated accurately even for the rare image. A rare image is an image in which the associated tag set includes tags with a low frequency of appearance, or the combination of tags constituting the tag set has a low frequency of appearance.

FIG. 9 is a flowchart showing a training (learning) method of the matching model 121 by the training program 123. FIG. 10 is a flowchart of processing of the matching model 121 in training. The training program 123 repeats the process described with reference to FIG. 9 for different images in the image-tag relational database 134.

With reference to FIG. 9, the training program 123 selects an image from the image-tag relational database 134 and generates a feature vector of the image (S221). As described above, the feature vector of the image can be generated, for example, by a pre-trained deep learning model (deep neural network).

Next, the training program 123 acquires the tag set associated with the selected image from the image-tag relational database 134 and generates a vector (related tag set vector) of the tag set (S222). The generation of the related tag set vector is the same as the generation of the tag set vector (S152) described with reference to FIG.

Next, the training program 123 samples a plurality of tags not associated with the selected image from the image-tag relational database 134 (S223). The training program 123 further generates a vector (unrelated tag vector) for each sampled tag (S224). The irrelevant tag vector is generated using the word embedding model, for example, as described above.

Instead of or in addition to the vector of each tag, a vector of a tag set consisting of a plurality of tags may be generated. For example, a tag set of tags for each record in the image-tag relational database 134, which includes one or more tags that are not associated with the selected image, or consists of tags that are not associated with the selected image. You may generate a vector. The vector of unrelated tags is also the vector of a tag set consisting of one tag.

The training program 123 inputs the generated image feature vector, the related tag set vector, and the unrelated tag vector into the matching model 121 (S225). Specifically, the training program 123 sequentially inputs a pair of the image feature vector and the related tag set vector, and a pair of each of the image feature vector and the unrelated tag vector into the matching model 121.

The matching model 121 outputs the similarity between the two input vectors. Therefore, the training program 123 acquires the similarity between the image feature vector and the related tag set vector, and the similarity between the image feature vector and the unrelated tag vector, respectively (S226).

The matching model 121 updates the matching model 121 based on the acquired similarity (S227). The matching model 121 updates (trains) the matching model 121 so that the output loss of the matching model 121 is reduced. For example, the hinge loss function is used.

For example, the matching model 121 trains the matching model 121 to minimize max (0, m + sim (x, y')-sim (x, y)). Here, sim indicates the degree of similarity, and m is a preset margin. x is an image feature vector, y is a related tag set vector, and y'is an unrelated tag vector.

FIG. 10 is a flowchart of processing of the matching model 121 in training. The matching model 121 acquires an image feature vector as input data (S251). The matching model 121 acquires an unrelated tag vector or a related tag set vector as input data (S252).

The matching model 121 maps the input image feature vector into a common space (S253). The matching model 121 further maps the input unrelated tag vector or related tag set vector to its common space (S254). The dimensions of the image feature vector and the tag or tag set vector can be different. These are matched in the mapping to the common space. The mapping of the input vector to the common space can be performed by a deep learning model (deep neural network). The parameters of the deep learning model are updated by training by the training program 123.

The matching model 121 calculates the degree of similarity between the image feature vector and the unrelated tag vector or the related tag set vector in the common space (S255). Similarity is defined, for example, by the dot product or cosine similarity. In addition, canonical correlation analysis (CCA: Canonical Correlation Analysis) can be applied to the matching model 121.

FIG. 11 schematically shows a configuration example in the semantic expression model 122. The semantic expression model 122 has an autoencoder configuration. The semantic expression model 122 can be configured by a deep learning model (deep neural network). The semantic representation model 122 includes an encoder 221 and a decoder 222.

The encoder 221 and the decoder 222 have, for example, a symmetrical structure centered on their outputs and inputs (semantic expression vector 225). This makes it possible to efficiently train the semantic expression model 122. The encoder 221 and the decoder 222 include, for example, a plurality of layers of CNN (Convolutional Neural Network) and one or more layers of fully connected layers.

The encoder 221 encodes the input image 227 into the semantic expression vector 225 and outputs it. The decoder 222 decodes the input semantic expression vector 225 into the reconstructed image 228 and outputs it. As described with reference to FIGS. 5 and 6, the semantic expression vector 225 output from the encoder 221 is used in the operation program 124. As will be described later, the semantic representation model 122 is trained so that the difference between the reconstructed image 228 and the input image 227 is small.

The training of the semantic expression model 122 will be described with reference to FIGS. 12A and 12B. FIG. 12A shows the input data and the output data in the training of the semantic expression model 122. The image 227 selected from the image-tag relational database 134 is input to the encoder 221. The decoder 222 receives the semantic expression vector 225 output from the encoder 221 as an input, and outputs the reconstructed image 228.

The training of the semantic representation model 122 constrains the semantic representation vector 225, which is the output of the encoder 221. An example of the constraint will be described later. To constrain the semantic representation vector 225, the tag set associated with the input image 227 is referenced in the image-tag relational database 134.

The error between the semantic representation vector 225 output from the encoder 221 and the associated tag set vector associated with the input image 227 is used in training the semantic representation model 122. In addition, the error between the reconstructed image 228 and the input image 227 is used in training the semantic representation model 122.

In the training of the semantic expression vector 225, the direction of trying to secure a sufficient amount of information to reproduce the input image 227 by considering the self-restorability (error between the reconstructed image 228 and the input image 227). Power works on. As a result, excessive degeneracy (generalization) in the training of the semantic expression model 122 is suppressed, and even a rare tag or tag set that appears infrequently, a tag or tag set that is important as an image feature is generated as noise. It can be prevented from being truncated.

FIG. 12B is a flowchart of training of the semantic expression model 122 by the training program 123. The training program 123 repeats the process described with reference to FIG. 12 for different images in the image-tag relational database 134.

The training program 123 selects an image from the image-tag relational database 134 (S301). The training program 123 acquires a tag set associated with the selected image from the image-tag relational database 134 and generates a vector (related tag set vector) thereof (S302). The generation of the related tag set vector is the same as the generation of the tag set vector (S152) described with reference to FIG.

The training program 123 inputs the selected image (input image 227) to the encoder 221 of the semantic expression model 122 (S303). The training program 123 acquires the output image (reconstructed image) 228 and the semantic expression vector 225 from the semantic expression model 122 (S304).

The training program 123 updates the semantic expression model 122 based on the error between the input image and the output image and the error between the semantic expression vector and the related tag vector. The training program 123 updates the parameters of the semantic expression model 122 so that, for example, || x-x'|| + \ lambda || sw || becomes smaller. The constraint on the semantic expression vector 255 is to reduce the error between the semantic expression vector and the related tag vector.

x and x'are an input image and an output image (reconstructed image), respectively. \ Lambda is a balancing parameter, s is a semantic expression vector, and w is a related tag set vector. The encoder 221 is represented by f, and the decoder is represented by f'. s = f (x), x'= f'(f (x)).

Another example of the constraint on the semantic representation vector 225 in the training of the semantic representation model 122 will be described. The training program 123 selects from the image-tag relational database 134 a tag set consisting of a tag set associated with the selected image and a tag set including or not associated with the selected image. Generate a vector representing a tag set (related tag set vector).

The semantic expression vector of the input image is represented by z, the related tag set vector of the input image is represented by y, and the unrelated tag set vector not related to the input image is represented by y'. The similarity between the related tag set vector y and the unrelated tag set vector y'is smaller than the similarity between the related tag set vector y and the related tag set vector y (same vector).

An example of the constraint on the semantic expression vector 225 in the training of the semantic expression model 122 is L2 (y, z) <L2 (y', z). L2 is the Euclidean distance in the vector space. It should be noted that a plurality of unrelated tag set vectors may be formed to include a plurality of inequalities in the constraint.

Another example of the constraint on the semantic representation vector 225 in the training of the semantic representation model 122 will be described. The training program 123 selects an image of a tag set having a different degree of similarity to the tag set of the input image from the image-tag relational database 134. For example, the training program 123 selects, for example, an image in which tags other than one tag are common to the input image as an image (approximate image) in which the input image and the tag set are similar.

The training program 123 further selects an image (dissimilar image) whose tag is completely different from the input image from the image-tag relational database 134. None of the tags associated with the input image are associated with the dissimilar image.

Let z1 be the semantic expression vector of the input image, z2 be the semantic expression vector of the approximate image, and z3 be the semantic expression vector of the dissimilar image. An example of the constraint on the semantic expression vector 225 in the training of the semantic expression model 122 is L2 (z1, z2) <L2 (z1, z3). It should be noted that a plurality of approximate images and / or a plurality of dissimilar images may be selected to include a plurality of inequalities in the constraint.

The constraints on the semantic representation vector 225 as described above allow the semantic representation model 122 to be properly trained to generate a semantic representation vector representing the tag set to be associated with the input image.

The present invention is not limited to the above-described examples, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the configurations described. Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above-mentioned configurations, functions, processing units and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be placed in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card or an SD card.

In addition, control lines and information lines are shown as necessary for explanation, and not all control lines and information lines are shown in the product. In practice, it can be considered that almost all configurations are interconnected.

100 tag estimator, 110 processor, 120 memory, 121 matching model, 122 semantic representation model, 123 training program, 124 operation program, 125 data crawler program, 126, 130 auxiliary storage, 132 image-text related database, 134 image- Tag-related database, 144 client device, 145 network interface, 221 encoder, 222 decoder, 225 semantic representation vector, 226 constraint, 227 input image, 228 reconstructed image, 547 tag distribution table

Claims (12)

A method performed on a computer that includes training a first model that outputs a vector representing a set of tags to be associated with the input image from the input image.
The calculator
Storage device and
Including a processor that operates according to a program stored in the storage device.
The first model includes an encoder that outputs a vector representing a tag set to be associated with the input image from the input image, and a decoder that inputs the output from the encoder.
In the method, the processor
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set preliminarily associated with the first training image. constraints and, based on, update the parameters of the first model, look including that,
The constraint is to reduce the error between the first output vector from the encoder and the vector of the first tag set . A method performed on a computer that includes training a first model that outputs a vector representing a set of tags to be associated with the input image from the input image.
The calculator
Storage device and
Including a processor that operates according to a program stored in the storage device.
The first model includes an encoder that outputs a vector representing a tag set to be associated with the input image from the input image, and a decoder that inputs the output from the encoder.
In the method, the processor
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
The frequency distribution of tags in the database storing the images used for training the first model and the tag sets associated with each of the images was analyzed.
A method comprising adding new images and new tag sets associated with each other to the database so that the frequency distribution approaches a uniform state. The method according to claim 2.
The processor
Get the new images and texts that are associated with each other
The similarity between the vector of each word in the text and the feature vector of the new image is determined.
A method further comprising selecting the new tag set associated with the new image from the text based on the similarity. The method according to claim 3.
The processor
The similarity between the vector of each word in the text and the feature vector of the image is determined by the second model.
In the training of the second model,
The first similarity between the feature vector of the second training image and the vector of the related tag set previously associated with the second training image is determined by the second model.
The second model determines the second similarity between the feature vector of the second training image and the vector of an unrelated tag set containing words not included in the related tag set.
A method further comprising updating the second model based on the first similarity and the second similarity. The method according to claim 3.
The processor further comprises selecting a first tag in the database whose frequency of occurrence should be increased based on the frequency of occurrence.
The method, wherein the text comprises the first tag. A method performed on a computer that includes training a first model that outputs a vector representing a set of tags to be associated with the input image from the input image.
The calculator
Storage device and
Including a processor that operates according to a program stored in the storage device.
The first model includes an encoder that outputs a vector representing a tag set to be associated with the input image from the input image, and a decoder that inputs the output from the encoder.
In the method, the processor
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
Input the first user image for the first model
The first user output vector for the first user image from the encoder is acquired and
The degree of similarity between the first user output vector and each vector of the tag set prepared in advance is determined.
A method comprising determining a tag set associated with the first user image based on the similarity. The method according to claim 6.
A method in which each of the tag sets prepared in advance is composed of one tag. A method performed on a computer that includes training a first model that outputs a vector representing a set of tags to be associated with the input image from the input image.
The calculator
Storage device and
Including a processor that operates according to a program stored in the storage device.
The first model includes an encoder that outputs a vector representing a tag set to be associated with the input image from the input image, and a decoder that inputs the output from the encoder.
In the method, the processor
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
Get the first user tag set,
Images prepared in advance are sequentially input to the first model, a candidate output vector is acquired from the encoder, and the candidate output vector is acquired.
The degree of similarity between each of the candidate output vectors and the vector of the first user tag set is determined.
A method comprising determining an image associated with the first user tag set based on the similarity. It ’s a computer system,
Storage device and
Including a processor that operates according to a program stored in the storage device.
The processor
A first model is trained that includes an encoder that outputs a vector representing a set of tags to be associated with the input image from the input image and a decoder that inputs the output from the encoder.
In the training of the first model,
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
A computer system in which the constraint is to reduce the error between the first output vector from the encoder and the vector of the first tag set. It ’s a computer system,
Storage device and
Including a processor that operates according to a program stored in the storage device.
The processor
A first model is trained that includes an encoder that outputs a vector representing a set of tags to be associated with the input image from the input image and a decoder that inputs the output from the encoder.
In the training of the first model,
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set preliminarily associated with the first training image. Based on the constraints, the parameters of the first model are updated .
The frequency distribution of tags in the database storing the images used for training the first model and the tag sets associated with each of the images was analyzed.
A computer system that adds new images and new tag sets associated with each other to the database so that the frequency distribution approaches a uniform state. It ’s a computer system,
Storage device and
Including a processor that operates according to a program stored in the storage device.
The processor
A first model is trained that includes an encoder that outputs a vector representing a set of tags to be associated with the input image from the input image and a decoder that inputs the output from the encoder.
In the training of the first model,
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
Input the first user image for the first model
The first user output vector for the first user image from the encoder is acquired and
The degree of similarity between the first user output vector and each vector of the tag set prepared in advance is determined.
A computer system that determines a set of tags associated with the first user image based on the similarity. It ’s a computer system,
Storage device and
Including a processor that operates according to a program stored in the storage device.
The processor
A first model is trained that includes an encoder that outputs a vector representing a set of tags to be associated with the input image from the input image and a decoder that inputs the output from the encoder.
In the training of the first model,
The first training image is input to the encoder,
The first output image from the decoder is acquired and
Obtain the first output vector from the encoder
With respect to the first output vector from the encoder preset based on the error between the first training image and the first output image and the first tag set previously associated with the first training image. Based on the constraints, the parameters of the first model are updated.
Get the first user tag set,
Images prepared in advance are sequentially input to the first model, a candidate output vector is acquired from the encoder, and the candidate output vector is acquired.
The degree of similarity between each of the candidate output vectors and the vector of the first user tag set is determined.
A computer system that determines an image associated with the first user tag set based on the similarity.

Images