JP2021157570A - Image similarity estimation system, learning device, estimation device, and program - Google Patents

Abstract

To provide an image similarity estimation system, a learning device, an estimation device, and a program, which can determine the similarity of images taking not only appearance but also concept into account.SOLUTION: An image similarity estimation system 1 comprises: an appearance information acquisition unit 10 for acquiring appearance information that indicates the appearance of an image; an appearance feature extraction unit 11 for extracting an appearance feature quantity that indicates the appearance feature of an image using the appearance information on the image and an appearance feature extraction model; a classification information acquisition unit 12 for acquiring classification information that indicates the classification of the image; a classification text feature extraction unit 13 for extracting a classification text feature quantity that indicates the feature of words that indicate image classification using the classification information on the image and a classification text feature extraction model; and an overall feature extraction unit 14 for extracting an overall feature quantity which is the feature of the whole of the image using the appearance feature quantity of the image, the classification text feature quantity and a multimodal model.SELECTED DRAWING: Figure 1

Description

The present invention relates to an image similarity estimation system, a learning device, an estimation device, and a program.

In the examination of a trademark application at the JPO, it is determined whether or not the trademark pertaining to the application is similar to the trademark already applied for. As a technique for determining whether or not images such as characters and figures are similar, there is a technique for extracting similar images by performing deep learning based on the characteristics of the images. For example, Patent Document 1 discloses a technique of specifying a plurality of locations of an image, calculating feature amounts of the specified locations, and extracting similar images based on the calculated feature amounts.

Japanese Patent No. 5934653

However, regarding the similarity of trademarks, the impression, memory, association, etc. that the applied trademark and the cited trademark give to the consumer by their appearance, name, or idea are comprehensively observed, and the applied trademark is designated as a designated product or designated service. Judgment is made based on whether or not there is a risk of confusion with the cited trademark when used in (Trademark Law, Article 4, Paragraph 1, Item 11). That is, the similarity of a trademark is comprehensively judged not only from the viewpoint of appearance but also from the viewpoint of each name and idea. Therefore, there is a problem that it is insufficient to judge the analogy of trademarks only by judging the similarity of images, that is, the similarity of appearances by using the technique of Patent Document 1.

The present invention has been made to solve the above problems, and an object of the present invention is to obtain an image similarity estimation system, a learning device, an estimation device, which can determine the similarity of images in consideration of not only appearance but also ideas. And to provide the program.

In order to solve the above problem, one aspect of the present invention uses an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, the appearance information in the image, and an appearance feature extraction model to obtain the appearance of the image. Using the appearance feature extraction unit that extracts the appearance feature amount indicating the characteristics of the image, the classification information acquisition unit that acquires the classification information indicating the classification of the image, the classification information in the image, and the classification text feature extraction model, the image is concerned. Using the classification text feature extraction unit for extracting the classification text feature amount indicating the characteristics of the wording indicating the classification of, the appearance feature amount in the image, the classification text feature amount, and the multimodal model, the entire image in the image is used. The overall feature extraction unit that extracts the overall feature amount that is a feature, the appearance feature extraction model, the model generation unit that generates the multimodal model, the overall feature amount in the target image, and the overall feature amount in the comparative image. Based on the above, the appearance feature extraction model includes an image similarity estimation unit that estimates the degree of similarity between the target image and the comparison image, and the appearance feature extraction model outputs the appearance feature amount in the image from the appearance information in the image. The model generation unit generates the appearance feature extraction model by causing the learning model to learn the correspondence between the appearance information and the classification information in the training image, and the classification text feature extraction model is a model. , The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image, and is a model for extracting the feature amount of the wording indicating the classification. The model generation unit includes the appearance feature amount in the learning image extracted by the appearance feature extraction unit, the classification text feature amount in the learning image extracted by the classification text feature extraction unit, and the learning. This is an image similarity estimation system characterized in that a multimodal model is generated by having a learning model learn a correspondence relationship with the classification information in an image.

Further, in one aspect of the present invention, in the image similarity estimation system described above, the appearance feature extraction model includes an attention mechanism that outputs a weighted value to the internal state of the learning model for deep learning, and the model generation. The unit may make the attention mechanism learn the weight according to the correspondence between the appearance information and the classification information in the learning image.

Further, in one aspect of the present invention, in the image similarity estimation system described above, the classification text feature extraction model weights the word feature amount indicating the feature amount of the word included in the wording by the idf value of the word. It is a model for extracting the feature of the wording based on the assigned value, and the idf value may be a value calculated by performing statistical processing on an image group which is a set of classified images. ..

Further, in one aspect of the present invention, in the image similarity estimation system described above, the idf value is based on the number of images including the classified text feature amount in the number of image groups which are a set of classified images. It may be a value calculated using a ratio.

Further, in one aspect of the present invention, in the image similarity estimation system described above, the model generation unit includes the appearance feature amount and the classification text feature amount in the learning image within the same range. Preprocessing that normalizes the data so that it becomes data is performed, and the learning model is trained to learn the correspondence between the appearance feature amount and the classification text feature amount and the classification information in the preprocessed training image. May generate the multimodal model.

Further, one aspect of the present invention is an appearance feature that shows the appearance feature of the image by using the appearance information acquisition unit that acquires the appearance information indicating the appearance of the image, the appearance information in the image, and the appearance feature extraction model. Using the appearance feature extraction unit that extracts the amount, the classification information acquisition unit that acquires the classification information indicating the classification of the image, the classification information in the image, and the classification text feature extraction model, the wording indicating the classification of the image. Using the classification text feature extraction unit that extracts the classification text feature amount indicating the feature, the appearance feature amount in the image, the classification text feature amount, and the multimodal model, the overall feature amount that is the feature of the entire image in the image. The appearance feature extraction model includes an overall feature extraction unit for extracting the image, an appearance feature extraction model, and a model generation unit for generating the multimodal model. It is a model that outputs a quantity, and the model generation unit generates the appearance feature extraction model by training the learning model on the correspondence between the appearance information and the classification information in the training image, and generates the classification text. The feature extraction model is a model that extracts the feature amount of the wording indicating the classification, and the multimodal model is a model that outputs the total feature amount in the image from the appearance feature amount and the classification text feature amount in the image. The model generation unit includes the appearance feature amount in the learning image extracted by the appearance feature extraction unit and the classification text feature amount in the learning image extracted by the classification text feature extraction unit. , A learning device that generates the multimodal model by having the learning model learn the correspondence with the classification information in the learning image.

Further, one aspect of the present invention is an appearance feature that shows the appearance feature of the image by using the appearance information acquisition unit that acquires the appearance information indicating the appearance of the image, the appearance information in the image, and the appearance feature extraction model. Using the appearance feature extraction unit that extracts the amount, the classification information acquisition unit that acquires the classification information indicating the classification of the image, the classification information in the image, and the classification text feature extraction model, the wording indicating the classification of the image. Using the classification text feature extraction unit that extracts the classification text feature amount indicating the feature, the appearance feature amount in the image, the classification text feature amount, and the multimodal model, the overall feature amount that is the feature of the entire image in the image. An overall feature extraction unit that extracts The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image, and causes the learning model to learn the correspondence between the appearance information and the classification information in the training image. The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification, and the multimodal model is a model obtained from the appearance feature amount and the classification text feature amount in the image. , The model that outputs the overall feature amount in the image, the appearance feature amount in the learning image extracted by the appearance feature extraction unit, and the learning image extracted by the classification text feature extraction unit. This is an estimation device that is a model generated by having a learning model learn the correspondence between the classification text feature amount and the classification information in the training image.

Further, in one aspect of the present invention, a computer is used to show the appearance features of the image by using the appearance information acquisition means for acquiring the appearance information indicating the appearance of the image, the appearance information in the image, and the appearance feature extraction model. Using the appearance feature extraction means for extracting the appearance feature amount, the classification information acquisition means for acquiring the classification information indicating the classification of the image, the classification information in the image, and the classification text feature extraction model, the wording indicating the classification of the image. Using the classification text feature extraction means for extracting the classification text feature amount indicating the feature, the appearance feature amount in the image, the classification text feature amount, and the multimodal model, the overall feature amount which is the feature of the entire image in the image can be obtained. It is a program for functioning as an overall feature extraction means to be extracted, the appearance feature extraction model, and a model generation means for generating the multimodal model, and the appearance feature extraction model is the image from the appearance information in the image. This is a model that outputs the appearance feature amount in the above, and the appearance feature extraction model is generated by having the learning model learn the correspondence between the appearance information and the classification information in the training image in the model generation means. The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification, and the multimodal model is the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image. In the model generation means, the appearance feature amount in the learning image extracted by the appearance feature extraction means and the classification in the learning image extracted by the classification text feature extraction means. This is a program in which the multimodal model is generated by having the learning model learn the correspondence between the text feature amount and the classification information in the learning image.

Further, in one aspect of the present invention, a computer is used to show the appearance features of the image by using the appearance information acquisition means for acquiring the appearance information indicating the appearance of the image, the appearance information in the image, and the appearance feature extraction model. Using the appearance feature extraction means for extracting the appearance feature amount, the classification information acquisition means for acquiring the classification information indicating the classification of the image, the classification information in the image, and the classification text feature extraction model, the wording indicating the classification of the image. Using the classification text feature extraction means for extracting the classification text feature amount indicating the feature, the appearance feature amount in the image, the classification text feature amount, and the multimodal model, the overall feature amount which is the feature of the entire image in the image can be obtained. It functions as an overall feature extraction unit to be extracted, an image similarity estimation means for estimating the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image. The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image, and obtains a correspondence relationship between the appearance information in the learning image and the classification information. It is a model generated by training a training model, the classification text feature extraction model is a model for extracting feature amounts of words indicating classification, and the multimodal model is the appearance feature amount and the appearance feature amount in an image. It is a model that outputs the total feature amount in the image from the classification text feature amount, and is extracted by the appearance feature amount in the learning image extracted by the appearance feature extraction means and the classification text feature extraction means. This is a program generated by having a learning model learn the correspondence between the classification text feature amount in the learning image and the classification information in the learning image.

According to the present invention, it is possible to determine the similarity of an image in consideration of not only the appearance but also the idea.

It is a block diagram which shows the structural example of the image similarity estimation system 1 of embodiment. It is a figure which shows the example of the image G of an embodiment. It is a figure which shows the example of the graphic classification Z of an embodiment. It is a figure which shows the structural example of the appearance information 170 of an embodiment. It is a figure which shows the structural example of the classification information 171 of an embodiment. It is a figure explaining the process performed by the image similarity estimation system 1 of an embodiment. It is a figure explaining the appearance feature extraction model 172 of embodiment. It is a figure explaining the classification text feature extraction model 173 of embodiment. It is a figure explaining the multimodal model 174 of embodiment. It is a flow chart which shows the flow of the process performed by the image similarity estimation system 1 of an embodiment. It is a flow chart which shows the flow of the process performed by the image similarity estimation system 1 of an embodiment. It is a flow chart which shows the flow of the process performed by the image similarity estimation system 1 of an embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

The image similarity estimation system 1 of the embodiment is a system that estimates the degree to which images are similar to each other. The image similarity estimation system 1 is applied to, for example, determining the similarity of a trademark according to an application in the examination of a trademark application at the Japan Patent Office.

In trademark examination, not only similarities in appearance but also similarities in consideration of names and conceptual similarities are judged. For example, in trademark examination, a search logical formula is created using the graphic classification given to the trademark. Then, by executing the search using the created logical expression, the trademarks that may be similar to the trademark according to the application are narrowed down from the trademarks that have already been applied for. From the narrowed down trademarks, those with similar appearances, names, or concepts are extracted.

Generally, in image processing using a deep learning model, appearance features in an image are extracted in multiple dimensions. Then, the degree of similarity is estimated according to the closeness of the distance between the vectors expressing the features of the appearance in the multidimensional space. That is, the degree of similarity is estimated from the appearance characteristics in the image. For this reason, it is almost impossible to presume that images with completely different appearance characteristics are similar. For example, consider a case in which an image representing the same object (for example, a vertical koto), one is a realistic natural image such as a photograph, and the other is a designed illustration image. In this case, if the appearance features of the two images are significantly different, it is difficult to presume that they are similar. That is, it is difficult to presume that the image showing the photograph of the vertical koto and the illustration image showing the vertical koto are similar. However, since the concept of "tatekoto" is the same, it is often judged that the two are conceptually similar in the similarity determination of a trademark. In image processing using a general deep learning model, it has been difficult to accurately estimate images having similar concepts in such trademarks.

As a countermeasure, the image similarity estimation system 1 of the present embodiment estimates using the classification text feature extraction model 173. The classification text feature extraction model 173 is a model in which the feature of the concept in the image is learned. That is, in the image similarity estimation system 1 of the present embodiment, not only the appearance feature in the image but also the conceptual feature in the image can be extracted. This makes it possible to estimate the degree of similarity from the viewpoint of the concept according to the closeness of the distance between the vectors showing the characteristics of the concept extracted from the image. Therefore, images with similar concepts can be extracted.

The concept in the image here is a wording indicating the classification of the image, and is, for example, a wording corresponding to the graphic classification given to the trademark. The feature of the concept in this embodiment is a feature of a word included in a wording, for example, a word vector in which words are expressed in a distributed manner. In the following description, conceptual features in images may be referred to as classification text features.

Further, in the image similarity estimation system 1 of the present embodiment, the appearance feature extraction model 172 and the classification text feature extraction model 173 are generated by using the deep learning model. The appearance feature extraction model 172 is a model in which the appearance features in the image are learned. The classification text feature extraction model 173 is a model that extracts features of the entire image (hereinafter, also referred to as overall features) based on the respective feature amounts of the appearance and the concept. That is, in the image similarity estimation system 1 of the present embodiment, it is possible to extract features (overall features) in which the features of the appearance and the concept in the image are integrated. This makes it possible to estimate the degree of similarity from the viewpoint of integrating both the appearance and the concept according to the closeness of the distance between the vectors that integrally indicate the appearance and the feature of the concept extracted from the image. Therefore, it is possible to extract similar images by looking at the appearance and the concept in an integrated manner.

FIG. 1 is a block diagram showing a configuration example of the image similarity estimation system 1 of the embodiment. The image similarity estimation system 1 includes, for example, an appearance information acquisition unit 10, an appearance feature extraction unit 11, a classification information acquisition unit 12, a classification text feature extraction unit 13, an overall feature extraction unit 14, and a model generation unit 15. , An image similarity estimation unit 16, a storage unit 17, and an estimation result output unit 18.

The appearance information acquisition unit 10 acquires information indicating the appearance in the image. The information indicating the appearance in the image is the information indicating the appearance of the image, for example, the information in which the RGB values are associated with the coordinates of each pixel. The appearance information acquisition unit 10 stores the acquired information as the appearance information 170 of the storage unit 17.

The appearance feature extraction unit 11 extracts the appearance feature amount (appearance feature amount) in the image by using the appearance information 170 in the image and the appearance feature extraction model 172. The appearance feature extraction model 172 is a model that outputs the appearance feature amount in the image from the appearance information in the image. The appearance feature extraction model 172 is generated by the model generation unit 15. The details of the appearance feature extraction model 172 will be described in detail later.

The classification information acquisition unit 12 acquires information indicating classification in the image. The information indicating the classification in the image is the information for classifying the contents shown in the image, for example, the information indicating the graphic classification in the trademark. The classification information acquisition unit 12 stores the acquired information as the classification information 171 of the storage unit 17.

The classification text feature extraction unit 13 uses the classification information 171 in the image and the classification text feature extraction model 173 to extract the feature amount (classification text feature amount) of the wording indicating the classification in the image. The classification text feature extraction model 173 is a model that outputs the classification text feature amount in the image from the classification information in the image. The classification text feature extraction model 173 is generated by the model generation unit 15. The details of the classification text feature extraction model 173 will be described in detail later.

The overall feature extraction unit 14 extracts the feature amount (overall feature amount) of the entire image in the image by using the appearance feature amount, the classification text feature amount, and the multimodal model 174 in the image. The overall feature extraction unit 14 acquires the appearance feature amount in the image from the appearance feature extraction unit 11. The overall feature extraction unit 14 acquires the classification text feature amount in the image from the classification text feature extraction unit 13. The multimodal model 174 is a model that outputs the total feature amount in the image from the appearance feature amount and the classification text feature amount in the image. The details of the multimodal model 174 will be described in detail later.

The model generation unit 15 generates an appearance feature extraction model 172. At this time, the model generation unit 15 causes the deep learning model to learn the correspondence between the appearance information and the classification information in the learning image. As a result, the model generation unit 15 generates a model that outputs the classification information in the image from the input appearance information of the image, and stores the information indicating the generated model as the appearance feature extraction model 172 of the storage unit 17. .. The information indicating the model is, for example, if the deep learning model is a CNN (Convolutional Neural Network) learning model, the number of units in each of the input layer, the intermediate layer, and the output layer of the CNN, the number of hidden layers, and the activation. Information that indicates a function, etc., and information that indicates a connection coefficient or weight that connects nodes in each layer.

In addition, the model generation unit 15 generates a multimodal model 174. At this time, the model generation unit 15 causes the deep learning model to learn the correspondence between the appearance feature amount and the classification text feature amount in the learning image and the classification information. The model generation unit 15 acquires the appearance feature amount in the learning image extracted by the appearance feature extraction unit 11. The model generation unit 15 acquires the classification text feature amount in the learning image extracted by the classification text feature extraction unit 13. As a result, the model generation unit 15 generates a model that outputs the classification information of the image from the input appearance feature amount and the classification text feature amount of the image.

Here, the classification information extracted from the appearance feature amount and the classification text feature amount of the image is a feature based on both the appearance feature amount and the classification text feature amount of the image, and can be said to be an overall feature. That is, the model generation unit 15 generates a model that outputs the overall features in the image by letting the deep learning model learn the correspondence between the appearance features and the classification text features in the learning image and the classification information. do. The model generation unit 15 stores information indicating the created model as a multimodal model 174 of the storage unit 17.

The image similarity estimation unit 16 estimates the degree of similarity of images (image similarity). The image similarity estimation unit 16 acquires the total feature amount for each of the plurality of images. The image similarity estimation unit 16 acquires the total feature amount of the image extracted by the total feature extraction unit 14. The image similarity estimation unit 16 calculates the distance (for example, cosine similarity) in the vector space of each other in the overall features extracted from each image. For example, the image similarity estimation unit 16 estimates the calculated order of distances as the order in which there is a high possibility of similarity. Alternatively, the image similarity estimation unit 16 may estimate that both images are similar when the calculated distance is less than a predetermined threshold value.

The estimation result output unit 18 outputs the estimation result estimated by the image similarity estimation unit 16. The estimation result output unit 18 displays the estimation result by outputting the estimation result to a display (not shown), for example. Alternatively, the estimation result output unit 18 may print the estimation result by outputting the estimation result to a printer (not shown).

Functional units of the image similarity estimation system 1 described above (appearance information acquisition unit 10, appearance feature extraction unit 11, classification information acquisition unit 12, classification text feature extraction unit 13, overall feature extraction unit 14, model generation unit 15, image similarity. The degree estimation unit 16 and the estimation result output unit 18) are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit part; circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), GPU (Graphics Processing Unit). It may be realized by (including), or it may be realized by the cooperation of software and hardware. The program may be stored in advance in a storage device (a storage device including a non-transient storage medium) such as an HDD (Hard Disk Drive) or a flash memory, or a removable storage device such as a DVD or a CD-ROM. It is stored in a medium (non-transient storage medium) and may be installed by mounting the storage medium in a drive device.

The storage unit 17 is configured by arbitrarily combining at least one storage medium. The storage medium is, for example, an HDD (Hard Disk Drive), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access read / write Memory), and a ROM (Read Only Memory). The storage unit 17 stores a program for executing various processes of the image similarity estimation system 1 and temporary data used when performing various processes.

The storage unit 17 stores, for example, the appearance information 170, the classification information 171 and the appearance feature extraction model 172, the classification text feature extraction model 173, and the multimodal model 174.

Here, the appearance information 170 and the classification information 171 will be described with reference to FIGS. 2 to 5.
FIG. 2 is a block diagram showing an example of the image G of the embodiment. FIG. 3 is a diagram showing an example of the graphic classification Z of the embodiment. FIG. 4 is a diagram showing a configuration example of the appearance information 170 of the embodiment. FIG. 5 is a diagram showing a configuration example of the classification information 171 of the embodiment.

As shown in FIG. 2, the image G is, for example, an image showing an illustration of a nurse drawn in a circle. As a feature of the appearance in the image G shown in the example of FIG. 2, for example, the graphic classification Z as shown in FIG. 3 is given. In this example, the graphic classification Z is "2.3.1 head, upper body", "2.3.3 nun, nurse" and the like.

As shown in FIG. 4, the appearance information 170 includes, for example, an image ID and appearance information. The image ID is identification information that uniquely identifies the image. Appearance information is information indicating the appearance in an image. In this example, as appearance information, information indicating coordinates for each pixel and RGB values is shown.

As shown in FIG. 5, the classification information 171 includes, for example, an image ID and classification information. The image ID is identification information that uniquely identifies the image. The classification information is information indicating the classification in the image. In this example, as the classification information, the information in which the number system of the graphic classification in the trademark and the wording of the classification corresponding to the number system are associated with each other is shown.

Here, the flow of the process in which the image similarity estimation system 1 extracts the overall features of the image will be described. FIG. 6 is a diagram illustrating a process performed by the image similarity estimation system 1 of the embodiment.

As shown in FIG. 6, the image similarity estimation system 1 outputs the appearance feature amount of the image G from the appearance feature extraction model 172 by inputting the appearance information in the image G into the appearance feature extraction model 172. Further, the image similarity estimation system 1 outputs the classification text feature amount of the image G from the classification text feature extraction model 173 by inputting the classification information in the image G into the classification text feature extraction model 173. Then, the image similarity estimation system 1 causes the multimodal model 174 to input the appearance feature amount and the classification text feature amount in the image G, so that the multimodal model 174 outputs the overall feature amount in the image G. As described above, in the image similarity estimation system 1, the appearance feature extraction model 172, the classification text feature extraction model 173, and the multimodal model 174 are used, and the overall feature amount of the image G is obtained from the appearance information and the classification information in the image G. Is extracted.

Here, the appearance feature extraction model 172 will be described in detail with reference to FIG. 7. FIG. 7 is a diagram illustrating an appearance feature extraction model 172 of the embodiment. As shown in FIG. 7, the appearance feature extraction model 172 includes, for example, a CNN unit 172A, an attention mechanism 172B, a multiplication unit 172C, and an appearance feature output unit 172D.

CNN part 172A is a model of deep learning by CNN. The attention mechanism 172B is a mechanism that weights and outputs the internal state output from the CNN unit 172A. For example, the attention mechanism 172B weights parts that are not important for estimation (for example, a background area in an image) less than important parts. This makes it possible to focus on the features that are useful for the estimation and have a greater impact on the estimation results. The addition unit 172C multiplies each of the output from the CNN unit 172A and the output from the attention mechanism 172B by a weight and outputs the output. The multiplication unit 172C functions as a switch that outputs either the output from the CNN unit 172A or the output from the attention mechanism 172B, for example. This makes it possible to control the presence / absence of the attention mechanism 172B and verify the influence of the presence / absence of the attention mechanism 172B on the estimation accuracy. The appearance feature output unit 172D is an output layer in which the output from the appearance feature extraction model 172, that is, the appearance feature amount in the image G is stored.

For example, first, the model generation unit 15 performs fine tuning of the CNN unit 172A. Specifically, the model generation unit 15 causes the CNN unit 172A to repeatedly learn the correspondence between the appearance information and the classification information in the learning image until a predetermined end condition is satisfied. The learning image is an image used for learning a model, and is an image in which the classification information is already associated with the image. As the learning image, for example, an applied trademark to which the graphic classification in the trademark is given is used. The predetermined end condition may be an arbitrarily determined condition, but for example, the change in the accuracy of estimation in the learning stage converges. Alternatively, the predetermined end condition may be a condition that the number of learnings reaches a predetermined upper limit or the accuracy of estimation becomes equal to or higher than a predetermined threshold value.

Next, the model generation unit 15 trains the attention mechanism 172B using the fine-tuned CNN unit 172A. By inputting the appearance information in the learning image, the model generation unit 15 classifies the classification information in the learning image with a high probability of being given based on the feature amount output from the attention mechanism 172B via the CNN unit 172A. The attention mechanism 172B is trained by adjusting the parameters in the attention mechanism 172B so as to approach the information.

In this way, the model generation unit 15 generates the appearance feature extraction model 172 by performing the two steps of fine tuning the CNN unit 172A and learning the attention mechanism 172B.

Here, the classification text feature extraction model 173 will be described in detail with reference to FIG. FIG. 8 is a diagram illustrating a classification text feature extraction model 173. As shown in FIG. 8, the classification text feature extraction model 173 includes, for example, an extraction word input layer 173A, a word feature embedding section 173B, a weighted average section 173C, and a classification text feature output section 173D.

The extracted word input layer 173A is an input layer into which words extracted from the wording indicating the classification of the image G are input. In the extracted word input layer 173A, for example, each of the words divided in the wording indicating the classification of the image G is input. For example, when the wording indicating the classification is "head, upper body", "head" and "upper body" are input to the extracted word input layer 173A, respectively. In the example of FIG. 8, for example, in the extracted word input layer 173A, the “head” is input to w1 and the “upper body” is input to w2. As in this example, the extracted word input layer 173A may be set with a number of nodes according to the number of words. In addition, when the word indicating the classification is not divided, the word indicating the classification is separated by part of speech by morphological analysis, and the word indicating the classification (for example, noun) is extracted from the word indicating the classification. You may try to do it.

The word feature embedding unit 173B outputs the feature of the word input to each node of the extracted word input layer 173A. The characteristic of a word is a so-called distributed expression of a word, and is information indicating the characteristic of a word obtained by inputting a word into a natural language processing model such as Word2Vec (hereinafter, W2V) learned using a corpus. be.

Here, in the graphic classification information, particularly in the graphic classification in the trademark, since it is necessary to extract similar trademarks without omission, there are some that are given the graphic classification with a relatively broad concept. The broad concept here is, for example, a classification such as "26.1.1 yen". There are many images in which circles are used, and if the characteristics of the wording indicating classification in such a relatively broad concept are used, many images will be similar, and it may not be possible to narrow down substantially. Highly sexual. In other words, reflecting the characteristics of words that indicate classification in a relatively broad concept may deteriorate the accuracy of estimation.

As a countermeasure for this, in the present embodiment, weighting is performed so that the influence of words for which the effect of narrowing down cannot be expected is reduced. Specifically, the weighted average unit 173C weights the word vector (word feature amount) extracted from the word with the idf value of the word, and outputs the weighted average value for each word vector. The idf value is a value represented by the following equation (1).

idf (X) = log (N_total / N_X) ... (1)

In the equation (1), the idf (X) is the idf value in the word (X). N_total is the total number of images to which the graphic classification is given. N_X is the number of images to which the graphic classification including the word (X) is given. As shown in equation (1), the idf value in the word included in the classification given to many images with respect to the total number of images is a small value, and the classification given to a small number of images with respect to the total number of images The idf value in the included word is a large value. By weighting with such an idf value, the features of words effective for narrowing down can be greatly influenced by the amount of classified text features. On the other hand, it is possible to suppress the influence of word features, which cannot be expected to be effective in narrowing down, on the classified text features.

The classification text feature output unit 173D is an output layer in which the output from the classification text feature extraction model 173, that is, the classification text feature amount in the image G is stored.

Here, the multimodal model 174 will be described in detail with reference to FIG. FIG. 9 is a diagram illustrating a multimodal model 174. As shown in FIG. 9, the multimodal model 174 includes, for example, a feature coupling input layer 174A, a fully coupled layer 174B, and an overall feature output unit 174C.

The feature combination input layer 174A is an input layer of the multimodal model 174 in which the appearance feature amount and the classification text feature amount in the image G are input. The overall feature output unit 174C is an output layer in which the output from the multimodal model 174, that is, the overall feature amount in the image G is stored. The fully connected layer 174B is an FC (Full Connection) layer that fully connects the feature-coupled input layer 174A and the overall feature output unit 174C.

Here, the appearance feature amount in the image G is output from the appearance feature extraction model 172. Further, the classification text feature amount in the image G is output from the classification text feature extraction model 173. Since each feature is output from different models, the range that each feature can take may not be the same range. If such features with different possible ranges are simply integrated and input as they are, the model learns only the correspondence between one feature and the output, and the other feature is reflected. It is more likely that no biased estimates will be made.

As a countermeasure for this, in the present embodiment, preprocessing for normalizing the two features input to the multimodal model 174 is performed. Specifically, the model generation unit 15 determines one feature amount so that the appearance feature amount in the image G and the classification text feature amount in the image G are in the same range (for example, 0 to 1). Multiply a uniform value. The model generation unit 15 multiplies the other feature amount by another uniform value different from the value obtained by multiplying the one feature amount, if necessary. As a result, the model generation unit 15 can train the multimodal model 174 so as to output the total feature amount in consideration of both of the two feature amounts.

Here, the flow of processing performed by the image similarity estimation system 1 will be described with reference to FIGS. 10 to 12. 10 to 12 are flow charts showing a flow of processing performed by the image similarity estimation system 1 of the embodiment.

FIG. 10 shows a flow of processing in which the image similarity estimation system 1 extracts the classification text feature amount from the image using the classification text feature extraction model 173. The image similarity estimation system 1 acquires the classification information of the image G (step S10). The image similarity estimation system 1 uses the classification information to separate (separate) the words indicating the classification of the image G for each word (step S11). The image similarity estimation system 1 extracts a word vector for each word (step S12).

On the other hand, the image similarity estimation system 1 calculates the idf value of each word (step S13). The image similarity estimation system 1 weights (multiplies) the word vector of the word with the idf value of the word (step S14). The image similarity estimation system 1 outputs a weighted average value of the word vectors in each weighted word for each word vector as a classification text feature amount in the image G (step S15).

In FIG. 10, the flow of calculating the idf value of a word in step S13 after extracting the word vector in step S12 has been illustrated and described. However, it suffices if the word vector can be multiplied by the idf value at least in step S14. After calculating the idf value of, the word vector may be extracted by performing the processing shown in steps S10 to S12. Alternatively, a process of calculating the idf value may be performed as a process independent of the process of extracting the word vector in FIG.

FIG. 11 shows a flow of processing in which the image similarity estimation system 1 generates the appearance feature extraction model 172 and the multimodal model 174. The image similarity estimation system 1 acquires appearance information of the learning image (step S20). The image similarity estimation system 1 acquires the classification information of the learning image (step S21). The image similarity estimation system 1 trains the CNN unit 172A of the appearance feature extraction model 172 by having the CNN unit 172A learn the correspondence between the appearance information and the classification information of the learning image (step S22). The image similarity estimation system 1 makes the classification information that is likely to be output based on the internal state of the CNN unit 172A obtained by inputting the appearance information of the learning image closer to the classification information of the learning image. , The attention mechanism 172B of the appearance feature extraction model 172 is learned (step S23). As a result, the image similarity estimation system 1 generates the appearance feature extraction model 172.

The image similarity estimation system 1 extracts the appearance feature amount of the learning image by inputting the appearance information of the learning image into the appearance feature extraction model 172 (step S24). The image similarity estimation system 1 extracts the classification text feature amount of the training image by causing the classification text feature extraction model 173 to input the classification information of the training image (step S25). The image similarity estimation system 1 performs a process of normalizing the appearance feature amount of the learning image and the classification text feature amount (step S26). In the image similarity estimation system 1, the classification information that is likely to be output based on the appearance feature amount and the classification text feature amount of the learning image that has been normalized is the classification information of the learning image (here, the overall feature). A multimodal model 174 is generated by training (adjusting the parameters) the fully connected layer 174B so as to approach (corresponding to the quantity) (step S27). The image similarity estimation system 1 stores the generated appearance feature extraction model 172 and the multimodal model 174 (step S28).

In FIG. 11, the flow of extracting the classification text feature amount in step S25 after extracting the appearance feature amount in step S24 has been illustrated and described, but at least two feature amounts (appearance feature amount and classification) in step S26 have been described. As long as the text feature amount) can be normalized, the appearance feature amount may be extracted after the classification text feature amount is extracted.

FIG. 12 shows a flow of processing in which the image similarity estimation system 1 estimates the similarity between two images (here, a target image and a comparison image). The image similarity estimation system 1 acquires appearance information in the target image (step S30), and uses the acquired information and the appearance feature extraction model 172 to extract the appearance feature amount in the target image (step S31). Further, the image similarity estimation system 1 acquires the classification information in the target image (step S32), and extracts the classification text feature amount in the target image by using the acquired information and the classification text feature extraction model 173 (step). S33). Then, the image similarity estimation system 1 extracts the overall feature amount in the target image by using the appearance feature amount in the target image, the classification text feature amount, and the multimodal model 174 (step S34).

On the other hand, the image similarity estimation system 1 extracts the total feature amount in the comparative image (step S35). The flow of the process in which the image similarity estimation system 1 extracts the total feature amount in the comparative image is the same as the flow of the process of extracting the total feature amount in the target image.

The image similarity estimation system 1 determines whether or not the overall feature amount of all the comparative images for which the similarity with the target image is to be estimated has been calculated (step S36). The image similarity estimation system 1 calculates the distance in the vector space of the overall features of the target image and the comparison image as the cosine similarity (step S37).

In FIG. 12, a flow of calculating the overall feature amount of all the comparative images for which the similarity with the target image is to be estimated in step S36 and then calculating each cosine similarity in step S36 has been illustrated and described. However, it suffices if at least the similarity between the target image and the comparison image can be calculated, and the cosine similarity may be calculated each time the overall feature amount in the comparison image is extracted.

As described above, the image similarity estimation system 1 of the embodiment includes the appearance information acquisition unit 10, the appearance feature extraction unit 11, the classification information acquisition unit 12, the classification text feature extraction unit 13, and the overall feature extraction unit. It includes 14, a model generation unit 15, and an image similarity estimation unit 16. The appearance information acquisition unit 10 acquires appearance information 170 indicating the appearance of the image G. The appearance feature extraction unit 11 extracts the appearance feature amount indicating the appearance feature of the image G by using the appearance information 170 in the image G and the appearance feature extraction model 172. The classification information acquisition unit 12 acquires classification information 171 indicating the classification of the image G. The classification text feature extraction unit 13 uses the classification information 171 in the image G and the classification text feature extraction model 173 to extract the classification text feature amount indicating the feature of the wording indicating the classification of the image G. The overall feature extraction unit 14 extracts the overall feature amount that is a feature of the entire image in the image G by using the appearance feature amount in the image G, the classification text feature amount, and the multimodal model 174. The model generation unit 15 generates an appearance feature extraction model 172 and a multimodal model 174. The image similarity estimation unit 16 estimates the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image. As a result, the image similarity estimation system 1 of the embodiment can extract features in consideration of both the appearance and the concept in the image G, and determines the similarity of the images in consideration of not only the appearance but also the idea. It is possible.

Further, in the image similarity estimation system 1 of the embodiment, the appearance feature extraction model 172 includes an attention mechanism 172B that outputs a weighted value to the internal state of the learning model of deep learning. The model generation unit 15 causes the attention mechanism 172B to learn the weights according to the correspondence between the appearance information in the learning image and the classification information. As a result, in the image similarity estimation system 1 of the embodiment, it is possible to focus on what is effective for extracting the appearance feature in the internal state of the appearance feature extraction model 172, and the appearance feature amount is extracted more accurately. It becomes possible to do.

Further, in the image similarity estimation system 1 of the embodiment, the classification text feature extraction model 173 uses the word feature amount indicating the feature amount of the word included in the wording based on the value weighted by the idf value of the word. It is a model that extracts the features of. The idf value is a value calculated by performing statistical processing on an image group which is a set of classified images. As a result, in the image similarity estimation system 1 of the embodiment, the influence of the graphic classification that cannot be expected to be effective for narrowing down can be weakened, and the influence of the graphic classification effective for narrowing down can be strengthened. Therefore, it is possible to extract effective classification text features by narrowing down.

Further, in the image similarity estimation system 1 of the embodiment, the idf value is the number of images having the same classification as the image from which the classification text feature amount is extracted with respect to the number of image groups which are a set of classified images. It is a value calculated using a ratio. As a result, the image similarity estimation system 1 of the embodiment has the same effect as the above-mentioned effect.

Further, in the image similarity estimation system 1 of the embodiment, the model generation unit 15 performs a process of normalizing the appearance feature amount and the classification text feature amount in the learning image so that the data are included in the same range. conduct. The model generation unit 15 generates a multimodal model 174 by causing the learning model to learn the correspondence between the appearance feature amount and the classification text feature amount and the classification information in the learning image that has been normalized. do. As a result, in the image similarity estimation system 1 of the embodiment, it is possible to extract the total feature amount in which both feature amounts are reflected without being biased to one of the two feature amounts. Therefore, similar images can be estimated in view of both appearance and concept.

Further, the image similarity estimation system 1 of the embodiment may be applied as a learning device. In this case, the learning device includes an appearance information acquisition unit 10, an appearance feature extraction unit 11, a classification information acquisition unit 12, a classification text feature extraction unit 13, an overall feature extraction unit 14, and a model generation unit 15. .. As a result, the learning device can generate a model capable of extracting the overall features in consideration of the appearance and the concept in the image G.

Further, the image similarity estimation system 1 of the embodiment may be applied as an estimation device. In this case, the estimation device includes an appearance information acquisition unit 10, an appearance feature extraction unit 11, a classification information acquisition unit 12, a classification text feature extraction unit 13, an overall feature extraction unit 14, and an image similarity estimation unit 16. To be equipped. As a result, the estimation device can extract the overall features in consideration of the appearance and the concept in the image G. Therefore, it is possible to estimate a similar image in consideration of the appearance and the concept in the image G.

All or part of the image similarity estimation system 1 in the above-described embodiment may be realized by a computer. In that case, the program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read by the computer system and executed. The term "computer system" as used herein includes hardware such as an OS and peripheral devices. Further, the "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built in a computer system. Further, a "computer-readable recording medium" is a communication line for transmitting a program via a network such as the Internet or a communication line such as a telephone line, and dynamically holds the program for a short period of time. It may also include a program that holds a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or a client in that case. Further, the above program may be for realizing a part of the above-mentioned functions, and may be further realized for realizing the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized by using a programmable logic device such as FPGA (Field Programmable Gate Array).

Although the embodiments of the present invention have been described in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs and the like within a range that does not deviate from the gist of the present invention.

1 Image similarity estimation system 10 Appearance information acquisition unit 11 Appearance feature extraction unit 12 Classification information acquisition unit 13 Classification text feature extraction unit 14 Overall feature extraction unit 15 Model generation unit 16 Image similarity estimation unit 17 Storage unit 18 Estimation result output unit 170 Appearance information 171 Classification information 172 Appearance feature extraction model 173 Classification text Feature extraction model 174 Multimodal model

Claims (9)

An appearance information acquisition unit that acquires appearance information indicating the appearance of an image,
Using the appearance information in the image and the appearance feature extraction model, an appearance feature extraction unit that extracts an appearance feature amount indicating the appearance feature of the image, and an appearance feature extraction unit.
A classification information acquisition unit that acquires classification information indicating the classification of images,
Using the classification information in the image and the classification text feature extraction model, the classification text feature extraction unit for extracting the classification text feature amount indicating the feature of the wording indicating the classification of the image, and the classification text feature extraction unit.
Using the appearance feature amount, the classification text feature amount, and the multimodal model in the image, an overall feature extraction unit that extracts the overall feature amount that is a feature of the entire image in the image, and an overall feature extraction unit.
The appearance feature extraction model, the model generation unit that generates the multimodal model, and
An image similarity estimation unit that estimates the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image.
With
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image.
The model generation unit generates the appearance feature extraction model by having the learning model learn the correspondence between the appearance information and the classification information in the learning image.
The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification.
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image.
The model generation unit includes the appearance feature amount in the learning image extracted by the appearance feature extraction unit, the classification text feature amount in the learning image extracted by the classification text feature extraction unit, and the learning. The multimodal model is generated by letting the learning model learn the correspondence with the classification information in the image.
Image similarity estimation system. The appearance feature extraction model includes an attention mechanism that outputs a weighted value to the internal state of the deep learning learning model.
The model generation unit causes the attention mechanism to learn weights according to the correspondence between the appearance information and the classification information in the learning image.
The image similarity estimation system according to claim 1. The classification text feature extraction model is a model that extracts the features of the wording based on the value obtained by weighting the word feature amount indicating the feature amount of the word included in the wording with the idf value of the word.
The idf value is a value calculated by performing statistical processing on an image group which is a set of classified images.
The image similarity estimation system according to claim 1 or 2. The idf value is a value calculated by using the ratio of the number of image groups, which is a set of classified images, to the number of images including the classified text feature amount.
The image similarity estimation system according to claim 3. The model generation unit performs a process of normalizing the appearance feature amount and the classification text feature amount of the training image so that the data is included in the same range, and performs the normalization process. The multimodal model is generated by letting the learning model learn the correspondence between the appearance feature amount and the classification text feature amount and the classification information in the training image.
The image similarity estimation system according to any one of claims 1 to 4. An appearance information acquisition unit that acquires appearance information indicating the appearance of an image,
Using the appearance information in the image and the appearance feature extraction model, an appearance feature extraction unit that extracts an appearance feature amount indicating the appearance feature of the image, and an appearance feature extraction unit.
A classification information acquisition unit that acquires classification information indicating the classification of images,
Using the classification information in the image and the classification text feature extraction model, the classification text feature extraction unit for extracting the classification text feature amount indicating the feature of the wording indicating the classification of the image, and the classification text feature extraction unit.
Using the appearance feature amount, the classification text feature amount, and the multimodal model in the image, an overall feature extraction unit that extracts the overall feature amount that is a feature of the entire image in the image, and an overall feature extraction unit.
The appearance feature extraction model, the model generation unit that generates the multimodal model, and
With
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image.
The model generation unit generates the appearance feature extraction model by having the learning model learn the correspondence between the appearance information and the classification information in the learning image.
The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification.
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image.
The model generation unit includes the appearance feature amount in the learning image extracted by the appearance feature extraction unit, the classification text feature amount in the learning image extracted by the classification text feature extraction unit, and the learning. The multimodal model is generated by letting the learning model learn the correspondence with the classification information in the image.
Learning device. An appearance information acquisition unit that acquires appearance information indicating the appearance of an image,
Using the appearance information in the image and the appearance feature extraction model, an appearance feature extraction unit that extracts an appearance feature amount indicating the appearance feature of the image, and an appearance feature extraction unit.
A classification information acquisition unit that acquires classification information indicating the classification of images,
Using the classification information in the image and the classification text feature extraction model, the classification text feature extraction unit for extracting the classification text feature amount indicating the feature of the wording indicating the classification of the image, and the classification text feature extraction unit.
Using the appearance feature amount, the classification text feature amount, and the multimodal model in the image, an overall feature extraction unit that extracts the overall feature amount that is a feature of the entire image in the image, and an overall feature extraction unit.
An image similarity estimation unit that estimates the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image.
With
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image, and causes the learning model to learn the correspondence between the appearance information and the classification information in the learning image. The generated model,
The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification.
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image, and the appearance in the learning image extracted by the appearance feature extraction unit. A model generated by having a learning model learn the correspondence between the feature amount and the classification text feature amount in the training image extracted by the classification text feature extraction unit and the classification information in the learning image. Is,
Estimator. Computer,
Appearance information acquisition means for acquiring appearance information indicating the appearance of an image,
An appearance feature extraction means for extracting an appearance feature amount indicating the appearance feature of the image by using the appearance information in the image and the appearance feature extraction model.
Classification information acquisition means for acquiring classification information indicating the classification of images,
A classification text feature extraction means for extracting a classification text feature amount indicating a feature of a wording indicating the classification of the image by using the classification information in the image and a classification text feature extraction model.
An overall feature extraction means for extracting an overall feature amount that is a feature of an entire image in the image using the appearance feature amount, the classified text feature amount, and a multimodal model in the image.
The appearance feature extraction model and the model generation means for generating the multimodal model,
It is a program to function as
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image.
In the model generation means, the appearance feature extraction model is generated by letting the learning model learn the correspondence between the appearance information and the classification information in the learning image.
The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification.
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image.
In the model generation means, the appearance feature amount in the learning image extracted by the appearance feature extraction means, the classification text feature amount in the learning image extracted by the classification text feature extraction means, and the learning. The multimodal model is generated by letting the learning model learn the correspondence with the classification information in the image.
program. Computer,
Appearance information acquisition means for acquiring appearance information indicating the appearance of an image,
An appearance feature extraction means for extracting an appearance feature amount indicating the appearance feature of the image by using the appearance information in the image and the appearance feature extraction model.
Classification information acquisition means for acquiring classification information indicating the classification of images,
A classification text feature extraction means for extracting a classification text feature amount indicating a feature of a wording indicating the classification of the image by using the classification information in the image and a classification text feature extraction model.
Using the appearance feature amount, the classification text feature amount, and the multimodal model in the image, an overall feature extraction unit that extracts the overall feature amount that is a feature of the entire image in the image, and an overall feature extraction unit.
An image similarity estimation means that estimates the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image.
It is a program to function as
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image, and causes the learning model to learn the correspondence between the appearance information and the classification information in the learning image. The generated model,
The classification text feature extraction model is a model for extracting the feature amount of the wording indicating the classification.
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image, and the appearance in the learning image extracted by the appearance feature extraction means. A model generated by having a learning model learn the correspondence between the feature amount and the classification text feature amount in the training image extracted by the classification text feature extraction means and the classification information in the learning image. Is,
program.

Images