JP7394680B2 - Image similarity estimation system, learning device, estimation device, and program - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) Publisher name: Institute of Electronics, Information and Communication Engineers Publication name: IEICE Technical Report, vol. 119, no. 386, MVE2019-30, pp. 31-32, January 2020 Publication date January 16, 2020 (2) Publication date January 23, 2020 Meeting name Media Experience Virtual Environment Basic Study Group (MVE) Venue Nara Advanced Research Center Graduate University of Science and Technology Information Science Building Venue A (8916-5 Takayama-cho, Ikoma City, Nara Prefecture (Keihanna Science City))

The present invention relates to an image similarity estimation system, a learning device, an estimation device, and a program.

When examining a trademark application at the Japan Patent Office, it is determined whether the trademark in the application is similar to a trademark that has already been applied for. As a technique for determining whether images of characters, figures, etc. are similar, there is a technique that performs deep learning based on image characteristics to extract similar images. For example, Patent Document 1 discloses a technique for identifying multiple locations in an image, calculating feature amounts for each of the identified locations, and extracting similar images based on the calculated feature amounts.

Patent No. 5934653

However, the similarity of trademarks is determined by comprehensively observing the impression, memory, association, etc. that the applied trademark and cited trademark give to consumers through their appearance, pronunciation, concept, etc., and Judgment will be made based on whether there is a risk of confusion with the source of the cited trademark when used in a trademark (examination criteria under Article 4, Paragraph 1, Item 11 of the Trademark Law). In other words, the similarity of trademarks is judged comprehensively from the viewpoints of not only appearance but also pronunciation and concept. For this reason, there is a problem in that simply determining the similarity of images, that is, the similarity of appearance only using the technique of Patent Document 1, is insufficient for determining the similarity of trademarks.

The present invention has been made to solve the above-mentioned problems, and its purpose is to provide an image similarity estimation system, a learning device, an estimation device, and a learning device capable of determining the similarity of images by considering not only the appearance but also the concept. and programs.

In order to solve the above problem, one aspect of the present invention uses an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, the appearance information in the image, and an appearance feature extraction model to determine the appearance of the image. an appearance feature extraction unit that extracts appearance feature quantities indicating the characteristics of the image; a classification information acquisition unit that acquires classification information indicating the classification of the image; A classified text feature extraction unit that extracts a classified text feature indicating the feature of the wording that indicates the classification of the image, the appearance feature in the image, the classified text feature, and a multimodal model to extract the entire image in the image. an overall feature extraction unit that extracts an overall feature amount that is a feature; a model generation unit that generates the appearance feature extraction model and the multimodal model; the overall feature amount in the target image and the overall feature amount in the comparison image. an image similarity estimating unit that estimates a degree of similarity between the target image and the comparison image based on the above, and the appearance feature extraction model outputs the appearance feature amount in the image from the appearance information in the image. The model generation unit generates the appearance feature extraction model by making the learning model learn the correspondence between the appearance information and the classification information in the learning image, and the classified text feature extraction model , the multimodal model is a model that outputs the overall feature amount of the image from the appearance feature amount and the classified text feature amount of the image; The model generation unit includes the appearance feature amount in the training image extracted by the appearance feature extraction unit, the classified text feature amount in the training image extracted by the classified text feature extraction unit, and the training image. The image similarity estimation system is characterized in that the multimodal model is generated by causing a learning model to learn the correspondence relationship between the images and the classification information.

Further, in one aspect of the present invention, in the image similarity estimation system described above, the appearance feature extraction model includes an attention mechanism that outputs a value weighted to an internal state of a deep learning learning model, and the model generation The unit may cause the attention mechanism to learn a weight according to a correspondence relationship between the appearance information and the classification information in the learning image.

In addition, one aspect of the present invention is that in the image similarity estimation system described above, the classified text feature extraction model weights word features indicating features of words included in a sentence using an idf value of the word. The idf value may be a value calculated by performing statistical processing on an image group that is a set of classified images. .

Further, in one aspect of the present invention, in the image similarity estimation system described above, the idf value is determined based on the number of images including the classified text feature amount in the number of image groups that are a set of classified images. The value may be calculated using a ratio.

Further, in one aspect of the present invention, in the image similarity estimation system described above, the model generation unit is configured such that the appearance feature amount and the classified text feature amount in the learning image are included within the same range. performing preprocessing to normalize the data, and causing a learning model to learn the correspondence between the appearance feature amount, the classified text feature amount, and the classification information in the learning image subjected to the preprocessing. The multimodal model may be generated by:

Further, one aspect of the present invention provides an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, and an appearance feature indicating the appearance feature of the image using the appearance information in the image and an appearance feature extraction model. an appearance feature extraction unit that extracts the quantity; a classification information acquisition unit that acquires classification information indicating the classification of the image; A classified text feature extraction unit that extracts a classified text feature indicating a feature, and a global feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and a multimodal model in the image. and a model generation unit that generates the appearance feature extraction model and the multimodal model, and the appearance feature extraction model extracts the appearance features in the image from the appearance information in the image. The model generation unit generates the appearance feature extraction model by making the learning model learn the correspondence between the appearance information and the classification information in the learning image, and generates the appearance feature extraction model and generates the classification text. The feature extraction model is a model that extracts the feature amount of words indicating classification, and the multimodal model is a model that outputs the overall feature amount of the image from the appearance feature amount and the classification text feature amount of the image. The model generation unit is configured to include the appearance feature amount in the training image extracted by the appearance feature extraction unit, and the classified text feature amount in the training image extracted by the classification text feature extraction unit. , a learning device that generates the multimodal model by causing a learning model to learn a correspondence relationship with the classification information in the learning image.

Further, one aspect of the present invention provides an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, and an appearance feature indicating the appearance feature of the image using the appearance information in the image and an appearance feature extraction model. an appearance feature extraction unit that extracts the quantity; a classification information acquisition unit that acquires classification information indicating the classification of the image; A classified text feature extraction unit that extracts a classified text feature indicating a feature, and a global feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and a multimodal model in the image. an image similarity estimating unit that estimates a degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image. The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image, and causes the learning model to learn the correspondence between the appearance information and the classification information in the learning image. The classified text feature extraction model is a model that extracts the feature amount of the wording indicating the classification, and the multimodal model is a model that extracts the feature amount of the wording that indicates the classification. , is a model that outputs the overall feature amount in the image, the appearance feature amount in the training image extracted by the appearance feature extraction unit, and the training image extracted by the classified text feature extraction unit. The estimation device is a model generated by causing a learning model to learn the correspondence between the classified text feature amount and the classification information in the learning image.

Further, one aspect of the present invention provides a computer that uses an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, the appearance information in the image, and an appearance feature extraction model to indicate the appearance characteristics of the image. Appearance feature extraction means for extracting appearance feature amounts, classification information acquisition means for acquiring classification information indicating the classification of the image, the classification information in the image, and a classification text feature extraction model to extract the text indicating the classification of the image. A classified text feature extraction means for extracting a classified text feature indicating a feature, the appearance feature in the image, the classified text feature, and a multimodal model to extract the overall feature that is the feature of the entire image in the image. A program for functioning as overall feature extraction means for extracting, the appearance feature extraction model, and model generation means for generating the multimodal model, wherein the appearance feature extraction model extracts the appearance information from the image from the appearance information in the image. The model outputs the appearance feature amount in the model generation means, and the appearance feature extraction model is generated by causing the learning model to learn the correspondence between the appearance information and the classification information in the learning image. , the classification text feature extraction model is a model for extracting the feature amount of a wording indicating classification, and the multimodal model extracts the overall feature amount in the image from the appearance feature amount and the classification text feature amount in the image. In the model generating means, the appearance feature quantity in the learning image extracted by the appearance feature extraction means and the classification in the learning image extracted by the classified text feature extraction means are used. This program generates the multimodal model by causing a learning model to learn the correspondence between text features and the classification information in the learning image.

Further, one aspect of the present invention provides a computer that uses an appearance information acquisition unit that acquires appearance information indicating the appearance of an image, the appearance information in the image, and an appearance feature extraction model to indicate the appearance characteristics of the image. Appearance feature extraction means for extracting appearance feature amounts, classification information acquisition means for acquiring classification information indicating the classification of the image, the classification information in the image, and a classification text feature extraction model to extract the text indicating the classification of the image. A classified text feature extraction means for extracting a classified text feature indicating a feature, the appearance feature in the image, the classified text feature, and a multimodal model to extract the overall feature that is the feature of the entire image in the image. It functions as an overall feature extraction unit that extracts, and an image similarity estimation unit that estimates a degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image. The appearance feature extraction model is a model that outputs the appearance feature amount of the image from the appearance information of the image, and calculates the correspondence between the appearance information and the classification information in the learning image. The model is a model generated by training a learning model, the classification text feature extraction model is a model that extracts the feature amount of a wording indicating classification, and the multimodal model is a model that extracts the feature amount of the wording indicating the classification, and the multimodal model is a model that extracts the feature amount of the wording indicating the classification. It is a model that outputs the overall feature amount of the image from the classified text feature amount, and the appearance feature amount in the training image extracted by the appearance feature extraction means and the appearance feature amount extracted by the classified text feature extraction means. The program is a model that is generated by causing a learning model to learn the correspondence between the classified text feature amount in the learning image and the classification information in the learning image.

According to this invention, it is possible to determine the similarity of images by considering not only the appearance but also the concept.

1 is a block diagram showing a configuration example of an image similarity estimation system 1 according to an embodiment. FIG. It is a figure which shows the example of the image G of embodiment. It is a figure which shows the example of figure classification Z of embodiment. It is a figure showing an example of composition of appearance information 170 of an embodiment. It is a figure showing an example of composition of classification information 171 of an embodiment. FIG. 2 is a diagram illustrating processing performed by the image similarity estimation system 1 of the embodiment. FIG. 3 is a diagram illustrating an appearance feature extraction model 172 according to the embodiment. It is a diagram explaining a classified text feature extraction model 173 of the embodiment. It is a figure explaining the multimodal model 174 of an embodiment. It is a flow diagram showing the flow of processing performed by the image similarity estimation system 1 of the embodiment. It is a flow diagram showing the flow of processing performed by the image similarity estimation system 1 of the embodiment. It is a flow diagram showing the flow of processing performed by the image similarity estimation system 1 of the embodiment.

Embodiments of the present invention will be described below with reference to the drawings.

The image similarity estimation system 1 of the embodiment is a system that estimates the degree of similarity between images. The image similarity estimation system 1 is applied, for example, to determining the similarity of trademarks related to applications in the examination of trademark applications at the Japan Patent Office.

When examining trademarks, similarity is determined not only by appearance, but also by appellation and conceptual similarity. For example, in trademark examination, a logical search formula is created using the graphic classification assigned to the trademark. Then, by executing a search using the created logical formula, trademarks that are likely to be similar to the trademark in the application are narrowed down from among the trademarks that have already been applied for. From the narrowed down trademarks, those with similar appearance, name, or concept are extracted.

Generally, in image processing using a deep learning model, external features in an image are extracted in multiple dimensions. Then, the degree of similarity is estimated according to the closeness of the vectors expressing the appearance features in a multidimensional space. That is, the degree of similarity is estimated from the appearance characteristics of the images. For this reason, it is almost impossible to estimate that images with completely different external features are similar. For example, consider a case where two images depict the same object (such as a harp), one of which is a realistic natural image such as a photograph, and the other a designed illustration image. In this case, if the external features of both images are significantly different, it is difficult to estimate that the two images are similar. That is, it is difficult to estimate that an image showing a photograph of a harp and an illustration image in which a harp is designed are similar. However, since the concept of "tatekoto" is the same, when determining similarity of trademarks, it is often determined that the two are conceptually similar. In image processing using general deep learning models, it is difficult to accurately estimate images with similar concepts in trademarks.

As a countermeasure against this, the image similarity estimation system 1 of this embodiment performs estimation using the classified text feature extraction model 173. The classified text feature extraction model 173 is a model that has learned the features of concepts in images. That is, in the image similarity estimation system 1 of this embodiment, it is possible to extract not only external features in images but also conceptual features in images. This makes it possible to estimate the degree of similarity from the concept point of view, depending on the closeness of the vectors representing the characteristics of the concept extracted from the image. Therefore, images with similar concepts can be extracted.

Note that the concept in the image here is a wording that indicates the classification of the image, and is, for example, a wording that corresponds to a graphical classification given to a trademark. The feature of a concept in this embodiment is a feature of a word included in a sentence, and is, for example, a word vector that is a distributed representation of a word. In the following description, conceptual features in images may be referred to as classified text features.

Furthermore, the image similarity estimation system 1 of this embodiment generates an appearance feature extraction model 172 and a classified text feature extraction model 173 using a deep learning model. The appearance feature extraction model 172 is a model that has learned appearance features in an image. The classified text feature extraction model 173 is a model that extracts features of the entire image (hereinafter also referred to as overall features) based on the respective feature amounts of appearance and concept. That is, in the image similarity estimation system 1 of this embodiment, it is possible to extract a feature (overall feature) that integrates the respective feature amounts of appearance and concept in an image. This makes it possible to estimate the degree of similarity from the perspective of integrating both appearance and concept, depending on the closeness of the vectors that integrally represent the features of appearance and concept extracted from the image. Therefore, it is possible to extract similar images based on an integrated view of appearance and concept.

FIG. 1 is a block diagram showing a configuration example of an image similarity estimation system 1 according to an embodiment. The image similarity estimation system 1 includes, for example, an appearance information acquisition section 10, an appearance feature extraction section 11, a classification information acquisition section 12, a classified text feature extraction section 13, an overall feature extraction section 14, and a model generation section 15. , an image similarity estimation section 16 , a storage section 17 , and an estimation result output section 18 .

The appearance information acquisition unit 10 acquires information indicating the appearance in an image. The information indicating the appearance of the image is information indicating the appearance of the image, and is, for example, information in which RGB values are associated with the coordinates of each pixel. The appearance information acquisition unit 10 stores the acquired information as appearance information 170 in the storage unit 17.

The appearance feature extraction unit 11 uses the appearance information 170 in the image and the appearance feature extraction model 172 to extract the appearance feature amount (appearance feature amount) in the image. The appearance feature extraction model 172 is a model that outputs appearance feature amounts in an image from appearance information in the image. The appearance feature extraction model 172 is generated by the model generation unit 15. Details of the appearance feature extraction model 172 will be explained in detail later.

The classification information acquisition unit 12 acquires information indicating the classification of the image. The information indicating the classification in the image is information for classifying the content shown in the image, and is, for example, information indicating the graphic classification in a trademark. The classification information acquisition unit 12 stores the acquired information as classification information 171 in the storage unit 17.

The classified text feature extraction unit 13 uses the classification information 171 in the image and the classified text feature extraction model 173 to extract the feature amount of the wording indicating the classification in the image (classified text feature amount). The classified text feature extraction model 173 is a model that outputs classified text feature amounts in an image from classification information in the image. The classified text feature extraction model 173 is generated by the model generation unit 15. The details of the classified text feature extraction model 173 will be explained in detail later.

The overall feature extraction unit 14 extracts the overall feature amount (overall feature amount) of the image using the appearance feature amount, the classified text feature amount, and the multimodal model 174 in the image. The overall feature extractor 14 acquires the appearance feature amount in the image from the appearance feature extractor 11. The overall feature extraction unit 14 acquires the classified text feature amount in the image from the classified text feature extraction unit 13 . The multimodal model 174 is a model that outputs the overall feature amount of an image from the appearance feature amount and classified text feature amount of the image. Details of the multimodal model 174 will be explained in detail later.

The model generation unit 15 generates an appearance feature extraction model 172. At this time, the model generation unit 15 causes the deep learning model to learn the correspondence between appearance information and classification information in the learning image. Thereby, the model generation unit 15 generates a model that outputs classification information for the image from the appearance information of the input image, and stores information indicating the generated model as the appearance feature extraction model 172 in the storage unit 17. . For example, if the deep learning model is a CNN (Convolutional Neural Network) learning model, the information indicating the model includes the number of units in each layer of the CNN input layer, intermediate layer, and output layer, the number of hidden layers, activation This is information indicating functions, etc., and information indicating coupling coefficients and weights for coupling nodes in each layer.

Furthermore, the model generation unit 15 generates a multimodal model 174. At this time, the model generation unit 15 causes the deep learning model to learn the correspondence between the appearance feature amount and the classified text feature amount in the learning image and the classification information. The model generation unit 15 acquires the appearance feature amount in the learning image extracted by the appearance feature extraction unit 11. The model generation unit 15 acquires the classified text feature amount in the learning image extracted by the classified text feature extraction unit 13. Thereby, the model generation unit 15 generates a model that outputs classification information for the image from the appearance feature amount and the classification text feature amount of the input image.

Here, the classification information extracted from the appearance feature amount and the classified text feature amount of the image is a feature based on both the image appearance feature amount and the classified text feature amount, and can be called an overall feature. That is, the model generation unit 15 generates a model that outputs the overall feature of the image by having a deep learning model learn the correspondence between the appearance feature amount and the classified text feature amount in the training image and the classification information. do. The model generation unit 15 causes the storage unit 17 to store information indicating the created model as a multimodal model 174.

The image similarity estimating unit 16 estimates the degree of image similarity (image similarity). The image similarity estimating unit 16 acquires the overall feature amount for each of the plurality of images. The image similarity estimation unit 16 obtains the overall feature amount of the image extracted by the overall feature extraction unit 14. The image similarity estimating unit 16 calculates the distance (for example, cosine similarity) between the overall features extracted from each image in the vector space. For example, the image similarity estimating unit 16 estimates the order of the calculated distances as the order in which there is a high possibility of similarity. Alternatively, the image similarity estimating unit 16 may estimate that both images are similar if the calculated distance is less than a predetermined threshold.

The estimation result output unit 18 outputs the estimation result estimated by the image similarity estimation unit 16. The estimation result output unit 18 displays the estimation result by outputting the estimation result to a display (not shown), for example. Alternatively, the estimation result output unit 18 may print the estimation result by outputting the estimation result to a printer (not shown).

Functional units of the image similarity estimation system 1 described above (appearance information acquisition unit 10, appearance feature extraction unit 11, classification information acquisition unit 12, classified text feature extraction unit 13, overall feature extraction unit 14, model generation unit 15, image similarity The degree estimation unit 16 and the estimation result output unit 18) are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these components are hardware (circuit parts) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), and GPU (Graphics Processing Unit). ), or may be realized by collaboration of software and hardware. The program may be stored in advance in a storage device (a storage device with a non-transitory storage medium) such as an HDD (Hard Disk Drive) or flash memory, or may be stored in a removable storage device such as a DVD or CD-ROM. It is stored in a medium (non-transitory storage medium), and may be installed by loading the storage medium into a drive device.

The storage unit 17 is configured by arbitrarily combining at least one storage medium. The storage medium is, for example, a HDD (Hard Disk Drive), a flash memory, an EEPROM (Electrically Erasable Programmable Read Only Memory), a RAM (Random Access read/write Memory), or a ROM (Read Only Memory). The storage unit 17 stores programs for executing various processes of the image similarity estimation system 1 and temporary data used when performing the various processes.

The storage unit 17 stores, for example, appearance information 170, classification information 171, appearance feature extraction model 172, classified text feature extraction model 173, and multimodal model 174.

Here, the appearance information 170 and the classification information 171 will be explained using FIGS. 2 to 5.
FIG. 2 is a block diagram showing an example of the image G of the embodiment. FIG. 3 is a diagram showing an example of figure classification Z according to the embodiment. FIG. 4 is a diagram illustrating a configuration example of appearance information 170 according to the embodiment. FIG. 5 is a diagram showing a configuration example of the classification information 171 according to the embodiment.

As shown in FIG. 2, the image G is, for example, an image showing an illustration of a nurse drawn in a circle. As a feature of the appearance of the image G shown in the example of FIG. 2, for example, a figure classification Z as shown in FIG. 3 is assigned. In this example, the figure classifications Z are "2.3.1 Head, upper body" and "2.3.3 Nun, nurse".

As shown in FIG. 4, the appearance information 170 includes, for example, an image ID and appearance information. The image ID is identification information that uniquely identifies an image. Appearance information is information indicating the appearance in an image. In this example, information indicating the coordinates and RGB values of each pixel is shown as appearance information.

As shown in FIG. 5, the classification information 171 includes, for example, an image ID and classification information. The image ID is identification information that uniquely identifies an image. Classification information is information indicating classification in an image. In this example, as classification information, information is shown in which a numbering system for a figure classification in a trademark is associated with a wording of a classification corresponding to the numbering system.

Here, a process flow in which the image similarity estimation system 1 extracts the overall features of an image will be explained. FIG. 6 is a diagram illustrating processing performed by the image similarity estimation system 1 of the embodiment.

As shown in FIG. 6, the image similarity estimation system 1 causes the appearance feature extraction model 172 to output the appearance feature amount of the image G by inputting appearance information on the image G to the appearance feature extraction model 172. Furthermore, the image similarity estimation system 1 causes the classified text feature extraction model 173 to output the classified text feature amount of the image G by inputting the classification information on the image G to the classified text feature extraction model 173. Then, the image similarity estimation system 1 inputs the appearance feature amount and the classified text feature amount in the image G to the multimodal model 174, thereby causing the multimodal model 174 to output the overall feature amount in the image G. In this way, the image similarity estimation system 1 uses the appearance feature extraction model 172, the classified text feature extraction model 173, and the multimodal model 174 to calculate the overall feature amount of the image G from the appearance information and classification information in the image G. Extract.

Here, the appearance feature extraction model 172 will be explained in detail using FIG. 7. FIG. 7 is a diagram illustrating the appearance feature extraction model 172 of the embodiment. As shown in FIG. 7, the appearance feature extraction model 172 includes, for example, a CNN section 172A, an attention mechanism 172B, a multiplication section 172C, and an appearance feature output section 172D.

The CNN unit 172A is a deep learning model using CNN. The attention mechanism 172B is a mechanism that weights and outputs the internal state output from the CNN unit 172A. For example, the attention mechanism 172B weights a portion that is not important for estimation (for example, a background region in an image) smaller than an important portion. This makes it possible to focus on features that are effective for estimation and have a greater influence on the estimation results. The adding unit 172C multiplies the output from the CNN unit 172A and the output from the attention mechanism 172B by a weight, and outputs the result. The multiplication unit 172C functions as a switch that outputs either the output from the CNN unit 172A or the output from the attention mechanism 172B, for example. This makes it possible to control the presence or absence of the attention mechanism 172B and to verify the influence of the presence or absence of the attention mechanism 172B on estimation accuracy. The appearance feature output unit 172D is an output layer in which the output from the appearance feature extraction model 172, that is, the appearance feature amount in the image G is stored.

For example, first, the model generation unit 15 performs fine tuning of the CNN unit 172A. Specifically, the model generation unit 15 causes the CNN unit 172A to repeatedly learn the correspondence between appearance information and classification information in the learning image until a predetermined termination condition is satisfied. The learning image is an image used for model learning, and is an image in which classification information has already been associated with the image. The learning image used is, for example, a trademark that has already been applied for and has been given a graphic classification in the trademark. The predetermined termination condition may be an arbitrarily determined condition, but for example, the predetermined termination condition is that the change in estimation accuracy in the learning stage converges. Alternatively, the predetermined termination condition may be that the number of times of learning reaches a predetermined upper limit, or that the accuracy of estimation exceeds a predetermined threshold.

Next, the model generation unit 15 causes the attention mechanism 172B to learn using the fine-tuned CNN unit 172A. By inputting the appearance information in the learning image, the model generation unit 15 generates classification information that has a high probability of being assigned based on the feature amount output from the attention mechanism 172B via the CNN unit 172A. The attention mechanism 172B is made to learn by adjusting the parameters in the attention mechanism 172B so as to approach the information.

In this way, the model generation unit 15 generates the appearance feature extraction model 172 by performing two procedures: fine tuning of the CNN unit 172A and learning of the attention mechanism 172B.

Here, the classified text feature extraction model 173 will be explained in detail using FIG. 8. FIG. 8 is a diagram illustrating the classified text feature extraction model 173. As shown in FIG. 8, the classified text feature extraction model 173 includes, for example, an extracted word input layer 173A, a word feature embedding section 173B, a weighted average section 173C, and a classified text feature output section 173D.

The extracted word input layer 173A is an input layer into which words extracted from the text indicating the classification of the image G are input. For example, each of the words separated in the text indicating the classification of the image G is input to the extracted word input layer 173A. For example, if the wording indicating the classification is "head, upper body", "head" and "upper body" are respectively input to the extracted word input layer 173A. In the example of FIG. 8, for example, in the extracted word input layer 173A, "head" is input to w1, and "upper body" is input to w2. As in this example, the number of nodes corresponding to the number of words may be set in the extracted word input layer 173A. In addition, when the text indicating the classification is not written separately, by morphologically analyzing the text indicating the classification, it is separated by part of speech and the word indicating the classification (for example, a noun) is extracted from the text indicating the classification. You may also do so.

The word feature embedding unit 173B outputs the features of the words input to each node of the extracted word input layer 173A. Word features are so-called distributed representations of words, and are information indicating the features of words that can be obtained by inputting words into a natural language processing model such as Word2Vec (hereinafter referred to as W2V) trained using a corpus. be.

Here, some graphic classification information, particularly graphic classification of trademarks, is given a relatively broad concept of graphic classification because it is necessary to extract all similar trademarks. The broad concept here is, for example, a classification such as "26.1.1 yen". There are many images that use circles, and if we use the characteristics of the wording that indicates classification based on a relatively broad concept, many images will be similar, and it may not be possible to narrow down the search effectively. Highly sexual. In other words, if the characteristics of the wording indicating a classification based on a relatively broad concept are reflected, the accuracy of estimation may deteriorate.

As a countermeasure for this, in this embodiment, weighting is performed so that the influence of words for which no narrowing effect can be expected is reduced. Specifically, the weighted average unit 173C weights the word vector (feature amount of the word) extracted from the word with the idf value of the word, and outputs the weighted average value for each word vector. The idf value is a value expressed by the following equation (1).

idf(X)=log(N_total/N_X)...(1)

In equation (1), idf(X) is the idf value for word (X). N_total is the total number of images to which graphic classifications have been assigned. N_X is the number of images to which a graphic classification including the word (X) is assigned. As shown in equation (1), the idf value for words included in classifications that are assigned to many images relative to the total number of images is small, and The idf value of the included words becomes a large value. By weighting with such idf values, the word features that are effective for narrowing down can be made to have a greater influence on the classified text feature amounts. On the other hand, it is possible to suppress the influence of word features that are not expected to be effective in narrowing down the classification text features.

The classified text feature output unit 173D is an output layer in which the output from the classified text feature extraction model 173, that is, the classified text feature amount in the image G is stored.

Here, the multimodal model 174 will be explained in detail using FIG. 9. FIG. 9 is a diagram illustrating the multimodal model 174. As shown in FIG. 9, the multimodal model 174 includes, for example, a feature combination input layer 174A, a fully connected layer 174B, and an overall feature output section 174C.

The feature combination input layer 174A is an input layer of the multimodal model 174 to which the appearance feature amount and the classified text feature amount in the image G are input. The overall feature output unit 174C is an output layer in which the output from the multimodal model 174, that is, the overall feature amount in the image G is stored. The fully connected layer 174B is an FC (Full Connection) layer that fully connects between the feature combination input layer 174A and the overall feature output section 174C.

Here, the appearance feature amount in the image G is output from the appearance feature extraction model 172. Further, the classified text feature amount in the image G is output from the classified text feature extraction model 173. Since the respective feature quantities are output from different models, the possible ranges of the respective feature quantities may not be the same. If such features with different possible ranges are simply integrated and input as is, the model will only learn the correspondence between one feature and the output, and the other feature will not be reflected. This increases the possibility that biased estimates will be made.

As a countermeasure for this, in the present embodiment, preprocessing is performed to normalize the two feature quantities input to the multimodal model 174. Specifically, the model generation unit 15 sets a predetermined value to one of the feature amounts so that the appearance feature amount in the image G and the classified text feature amount in the image G are in the same range (for example, from 0 to 1). Multiply by a uniform value. The model generation unit 15 multiplies the other feature amount by a uniform value different from the value multiplied by one feature amount, as necessary. Thereby, the model generation unit 15 can train the multimodal model 174 to take both of the two feature quantities into consideration and output the entire feature quantity.

Here, the flow of processing performed by the image similarity estimation system 1 will be explained using FIGS. 10 to 12. 10 to 12 are flowcharts showing the flow of processing performed by the image similarity estimation system 1 of the embodiment.

FIG. 10 shows a flow of processing in which the image similarity estimation system 1 extracts classified text features from an image using the classified text feature extraction model 173. The image similarity estimation system 1 acquires classification information of the image G (step S10). The image similarity estimation system 1 uses the classification information to separate (separate) the wording indicating the classification of the image G into words (step S11). The image similarity estimation system 1 extracts word vectors for each word (step S12).

On the other hand, the image similarity estimation system 1 calculates an idf value for each word (step S13). The image similarity estimation system 1 weights (multiplies) the word vector of a word by the idf value of the word (step S14). The image similarity estimation system 1 outputs the weighted average of the weighted word vectors for each word vector as the classified text feature amount in the image G (step S15).

In addition, in FIG. 10, the flow of calculating the idf value of the word in step S13 after extracting the word vector in step S12 was explained as an example, but it is only necessary that the word vector can be multiplied by the idf value in step S14, After calculating the idf value of , word vectors may be extracted by performing the processing shown in steps S10 to S12. Alternatively, the process of calculating the idf value may be performed as a process independent of the process of extracting word vectors in FIG. 10.

FIG. 11 shows a process flow in which the image similarity estimation system 1 generates the appearance feature extraction model 172 and the multimodal model 174. The image similarity estimation system 1 acquires appearance information of the learning image (step S20). The image similarity estimation system 1 acquires classification information of the learning image (step S21). The image similarity estimation system 1 causes the CNN unit 172A of the appearance feature extraction model 172 to learn the correspondence between the appearance information of the learning image and the classification information (step S22). The image similarity estimation system 1 is configured so that the classification information that is likely to be output based on the internal state of the CNN unit 172A obtained by inputting the appearance information of the training image approaches the classification information of the training image. , the attention mechanism 172B of the appearance feature extraction model 172 is trained (step S23). Thereby, the image similarity estimation system 1 generates the appearance feature extraction model 172.

The image similarity estimation system 1 extracts the appearance feature amount of the learning image by inputting the appearance information of the learning image to the appearance feature extraction model 172 (step S24). The image similarity estimation system 1 extracts the classified text feature amount of the training image by inputting the classification information of the training image to the classified text feature extraction model 173 (step S25). The image similarity estimation system 1 performs a process of normalizing the appearance feature amount and the classified text feature amount of the learning image (step S26). In the image similarity estimation system 1, classification information that is likely to be output based on the appearance features and classified text features of the training images that have undergone normalization processing is the classification information of the training images (here, the overall The multimodal model 174 is generated by training the fully connected layer 174B (adjusting the parameters) so that it approaches the amount (equivalent to the amount) (step S27). The image similarity estimation system 1 stores the generated appearance feature extraction model 172 and multimodal model 174 (step S28).

In FIG. 11, the flow of extracting the appearance feature amount in step S24 and then extracting the classified text feature amount in step S25 was explained as an example, but at least two feature amounts (appearance feature amount and classification text feature amount) are extracted in step S26. It is only necessary that the text feature amount) can be normalized, and the appearance feature amount may be extracted after the classified text feature amount is extracted.

FIG. 12 shows a process flow in which the image similarity estimation system 1 estimates the similarity between two images (here, a target image and a comparison image). The image similarity estimation system 1 acquires appearance information in the target image (step S30), and uses the acquired information and the appearance feature extraction model 172 to extract appearance features in the target image (step S31). The image similarity estimation system 1 also acquires classification information in the target image (step S32), and uses the acquired information and the classified text feature extraction model 173 to extract classified text features in the target image (step S32). S33). Then, the image similarity estimation system 1 extracts the overall feature amount in the target image using the appearance feature amount, the classified text feature amount, and the multimodal model 174 in the target image (step S34).

On the other hand, the image similarity estimation system 1 extracts the overall feature amount in the comparison image (step S35). The process flow in which the image similarity estimation system 1 extracts the overall feature amount in the comparison image is similar to the process flow in which the overall feature amount is extracted in the target image.

The image similarity estimation system 1 determines whether the overall feature amount has been calculated for all comparison images whose similarity with the target image is to be estimated (step S36). The image similarity estimation system 1 calculates the distance in the vector space in the overall features of each of the target image and the comparison image as a cosine similarity (step S37).

In addition, FIG. 12 illustrates and explains the flow of calculating the cosine similarity of each image in step S36 after calculating the overall feature amount of all comparison images whose similarity with the target image is to be estimated in step S36. However, it is only necessary to be able to calculate at least the similarity between the target image and the comparison image, and the cosine similarity may be calculated each time the overall feature amount in the comparison image is extracted.

As described above, the image similarity estimation system 1 of the embodiment includes the appearance information acquisition section 10, the appearance feature extraction section 11, the classification information acquisition section 12, the classified text feature extraction section 13, and the overall feature extraction section. 14, a model generation section 15, and an image similarity estimation section 16. The appearance information acquisition unit 10 acquires appearance information 170 indicating the appearance of the image G. The appearance feature extraction unit 11 uses the appearance information 170 in the image G and the appearance feature extraction model 172 to extract appearance feature amounts indicating the appearance characteristics of the image G. The classification information acquisition unit 12 acquires classification information 171 indicating the classification of the image G. The classified text feature extraction unit 13 uses the classification information 171 in the image G and the classified text feature extraction model 173 to extract a classified text feature quantity that indicates the feature of the wording that indicates the classification of the image G. The overall feature extraction unit 14 extracts the overall feature amount, which is the feature of the entire image G, using the appearance feature amount, the classified text feature amount, and the multimodal model 174 in the image G. The model generation unit 15 generates an appearance feature extraction model 172 and a multimodal model 174. The image similarity estimating unit 16 estimates the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image. As a result, the image similarity estimation system 1 of the embodiment can extract features that take into account both the appearance and the concept in the image G, and determine the similarity of images by considering not only the appearance but also the concept. Is possible.

Further, in the image similarity estimation system 1 of the embodiment, the appearance feature extraction model 172 includes an attention mechanism 172B that outputs a value weighted to the internal state of the deep learning learning model. The model generation unit 15 causes the attention mechanism 172B to learn weights according to the correspondence between the appearance information in the learning image and the classification information. As a result, the image similarity estimation system 1 of the embodiment can focus on the internal state of the appearance feature extraction model 172 that is effective for extracting appearance features, and extract appearance feature amounts with higher accuracy. It becomes possible to do so.

In addition, in the image similarity estimation system 1 of the embodiment, the classified text feature extraction model 173 extracts the word feature amount representing the feature amount of the word included in the sentence based on the value weighted by the idf value of the word. This is a model that extracts the features of The idf value is a value calculated by performing statistical processing on an image group that is a set of classified images. Thereby, in the image similarity estimation system 1 of the embodiment, it is possible to weaken the influence of figure classification, which is not expected to be effective in narrowing down, and strengthen the influence of figure classification, which is effective in narrowing down. Therefore, it becomes possible to extract effective classified text features by narrowing down.

Furthermore, in the image similarity estimation system 1 of the embodiment, the idf value is the number of images that are classified in the same manner as the image from which the classified text feature is extracted, with respect to the number of image groups that are a set of classified images. This is a value calculated using a ratio. Thereby, the image similarity estimation system 1 of the embodiment achieves the same effects as those described above.

Further, in the image similarity estimation system 1 of the embodiment, the model generation unit 15 performs a process of normalizing the appearance feature amount and the classification text feature amount in the learning image so that they are data included in the same range. conduct. The model generation unit 15 generates a multimodal model 174 by having a learning model learn the correspondence between the appearance feature amount, the classified text feature amount, and the classification information in the learning image that has been subjected to the normalization process. do. Thereby, in the image similarity estimation system 1 of the embodiment, it is possible to extract the entire feature amount in which both feature amounts are reflected, without being biased towards one of the two feature amounts. Therefore, similar images can be estimated based on both appearance and concept.

Moreover, the image similarity estimation system 1 of the embodiment may be applied as a learning device. In this case, the learning device includes an appearance information acquisition section 10, an appearance feature extraction section 11, a classification information acquisition section 12, a classified text feature extraction section 13, an overall feature extraction section 14, and a model generation section 15. . Thereby, the learning device can generate a model that can extract the overall features in consideration of the appearance and concept of the image G.

Moreover, the image similarity estimation system 1 of the embodiment may be applied as an estimation device. In this case, the estimation device includes an appearance information acquisition section 10, an appearance feature extraction section 11, a classification information acquisition section 12, a classified text feature extraction section 13, an overall feature extraction section 14, and an image similarity estimation section 16. Equipped with. Thereby, the estimation device can extract the overall feature in consideration of the appearance and concept of the image G. Therefore, it is possible to estimate similar images by considering the appearance and concept of image G.

All or part of the image similarity estimation system 1 in the embodiment described above may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed. Note that the "computer system" herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, magneto-optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, a "computer-readable recording medium" refers to a storage medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. It may also include a device that retains a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in that case. Further, the above-mentioned program may be one for realizing a part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system. It may be realized using a programmable logic device such as an FPGA (Field Programmable Gate Array).

Although the embodiments of the present invention have been described above in detail with reference to the drawings, the specific configuration is not limited to these embodiments, and includes designs within the scope of the gist of the present invention.

1 Image similarity estimation system 10 Appearance information acquisition unit 11 Appearance feature extraction unit 12 Classification information acquisition unit 13 Classified text feature extraction unit 14 Overall feature extraction unit 15 Model generation unit 16 Image similarity estimation unit 17 Storage unit 18 Estimation result output unit 170 Appearance information 171 Classification information 172 Appearance feature extraction model 173 Classified text feature extraction model 174 Multimodal model

Claims (9)

an appearance information acquisition unit that acquires appearance information indicating the appearance of the image;
an appearance feature extraction unit that uses the appearance information in the image and the appearance feature extraction model to extract appearance feature amounts indicating the appearance characteristics of the image;
a classification information acquisition unit that acquires classification information indicating the classification of the image;
a classified text feature extraction unit that uses the classification information in the image and the classified text feature extraction model to extract a classified text feature quantity that represents a feature of a wording that indicates the classification of the image;
an overall feature extraction unit that extracts an overall feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and the multimodal model in the image;
a model generation unit that generates the appearance feature extraction model and the multimodal model;
an image similarity estimation unit that estimates a degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image;
Equipped with
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image,
The model generation unit generates the appearance feature extraction model by causing a learning model to learn a correspondence relationship between the appearance information and the classification information in the learning image,
The classified text feature extraction model is a model that extracts feature amounts of text indicating classification,
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classified text feature amount in the image,
The model generation unit includes the appearance feature amount in the training image extracted by the appearance feature extraction unit, the classified text feature amount in the training image extracted by the classified text feature extraction unit, and the training image. generating the multimodal model by causing a learning model to learn the correspondence relationship with the classification information in the image for use;
Image similarity estimation system. The appearance feature extraction model includes an attention mechanism that outputs a weighted value for the internal state of the deep learning learning model,
The model generation unit causes the attention mechanism to learn weights according to the correspondence between the appearance information and the classification information in the learning image.
The image similarity estimation system according to claim 1. The classified text feature extraction model is a model that extracts features of a word based on a value obtained by weighting a word feature representing a feature of a word included in the word with an idf value of the word,
The idf value is a value calculated by performing statistical processing on an image group that is a set of classified images.
The image similarity estimation system according to claim 1 or claim 2. The idf value is a value calculated using the ratio of the number of image groups that are a set of classified images to the number of images that include the classified text feature amount,
The image similarity estimation system according to claim 3. The model generation unit performs normalization processing such that the appearance feature amount and the classified text feature amount in the training image are data included in the same range, and performs the normalization processing. generating the multimodal model by causing a learning model to learn the correspondence between the appearance feature amount, the classified text feature amount, and the classification information in the learning image;
The image similarity estimation system according to any one of claims 1 to 4. an appearance information acquisition unit that acquires appearance information indicating the appearance of the image;
an appearance feature extraction unit that uses the appearance information in the image and the appearance feature extraction model to extract appearance feature amounts indicating the appearance characteristics of the image;
a classification information acquisition unit that acquires classification information indicating the classification of the image;
a classified text feature extraction unit that uses the classification information in the image and the classified text feature extraction model to extract a classified text feature quantity that represents a feature of a wording that indicates the classification of the image;
an overall feature extraction unit that extracts an overall feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and the multimodal model in the image;
a model generation unit that generates the appearance feature extraction model and the multimodal model;
Equipped with
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image,
The model generation unit generates the appearance feature extraction model by causing a learning model to learn a correspondence relationship between the appearance information and the classification information in the learning image,
The classified text feature extraction model is a model that extracts feature amounts of text indicating classification,
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classified text feature amount in the image,
The model generation unit includes the appearance feature amount in the training image extracted by the appearance feature extraction unit, the classified text feature amount in the training image extracted by the classified text feature extraction unit, and the training image. generating the multimodal model by causing a learning model to learn the correspondence relationship with the classification information in the image for use;
learning device. an appearance information acquisition unit that acquires appearance information indicating the appearance of the image;
an appearance feature extraction unit that uses the appearance information in the image and the appearance feature extraction model to extract appearance feature amounts indicating the appearance characteristics of the image;
a classification information acquisition unit that acquires classification information indicating the classification of the image;
a classified text feature extraction unit that uses the classification information in the image and the classified text feature extraction model to extract a classified text feature quantity that represents a feature of a wording that indicates the classification of the image;
an overall feature extraction unit that extracts an overall feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and the multimodal model in the image;
an image similarity estimation unit that estimates a degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image;
Equipped with
The appearance feature extraction model is a model that outputs the appearance feature amount of the image from the appearance information in the image, and by making the learning model learn the correspondence between the appearance information and the classification information in the learning image. The generated model is
The classified text feature extraction model is a model that extracts feature amounts of text indicating classification,
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classified text feature amount in the image, and the multimodal model outputs the overall feature amount in the image, and the appearance feature amount in the learning image extracted by the appearance feature extraction unit. A model generated by causing a learning model to learn the correspondence between the feature amount and the classified text feature amount in the training image extracted by the classified text feature extraction unit and the classification information in the training image. is,
Estimation device. computer,
Appearance information acquisition means for acquiring appearance information indicating the appearance of the image;
Appearance feature extraction means for extracting appearance feature amounts indicating the appearance characteristics of the image using the appearance information in the image and the appearance feature extraction model;
classification information acquisition means for acquiring classification information indicating the classification of the image;
Classified text feature extraction means that uses the classification information in the image and a classified text feature extraction model to extract a classified text feature quantity that represents a feature of a wording that indicates the classification of the image;
overall feature extraction means for extracting an overall feature amount that is a feature of the entire image in the image using the appearance feature amount, the classified text feature amount, and the multimodal model in the image;
model generation means for generating the appearance feature extraction model and the multimodal model;
It is a program to function as
The appearance feature extraction model is a model that outputs the appearance feature amount in the image from the appearance information in the image,
In the model generation means, the appearance feature extraction model is generated by causing a learning model to learn the correspondence between the appearance information and the classification information in the learning image,
The classified text feature extraction model is a model that extracts feature amounts of text indicating classification,
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classified text feature amount in the image,
In the model generation means, the appearance feature amount in the learning image extracted by the appearance feature extraction means, the classified text feature amount in the learning image extracted by the classified text feature extraction means, and the learning The multimodal model is generated by causing a learning model to learn the correspondence relationship with the classification information in the image for use.
program. computer,
Appearance information acquisition means for acquiring appearance information indicating the appearance of the image;
Appearance feature extraction means for extracting appearance feature amounts indicating the appearance characteristics of the image using the appearance information in the image and the appearance feature extraction model;
classification information acquisition means for acquiring classification information indicating the classification of the image;
Classified text feature extraction means that uses the classification information in the image and a classified text feature extraction model to extract a classified text feature quantity that represents a feature of a wording that indicates the classification of the image;
an overall feature extraction unit that extracts an overall feature that is a feature of the entire image in the image using the appearance feature, the classified text feature, and the multimodal model in the image;
image similarity estimating means for estimating the degree of similarity between the target image and the comparison image based on the overall feature amount in the target image and the overall feature amount in the comparison image;
It is a program to function as
The appearance feature extraction model is a model that outputs the appearance feature amount of the image from the appearance information in the image, and by making the learning model learn the correspondence between the appearance information and the classification information in the learning image. The generated model is
The classified text feature extraction model is a model that extracts feature amounts of text indicating classification,
The multimodal model is a model that outputs the overall feature amount in the image from the appearance feature amount and the classified text feature amount in the image, and the multimodal model outputs the overall feature amount in the image, and the appearance feature amount in the learning image extracted by the appearance feature extraction means. A model generated by having a learning model learn the correspondence between the feature amount and the classified text feature amount in the training image extracted by the classified text feature extraction means and the classification information in the training image. is,
program.

Images