JP5444106B2 - Tag assignment device, conversion rule generation device, and tag assignment program - Google Patents

Description

  The present invention relates to a tag assignment device, a conversion rule generation device, and a tag assignment program for attaching a tag in a minor language with low usage on the Web to content.

  Conventionally, methods for generating or tagging metadata of content are known. Patent Document 1 provides an automatic tagging device that can assist a viewer in order to organize a document by assigning an appropriate tag to the document, and can assign a tag with high accuracy. Tagged documents are collected from the Internet, and tags that have the same meaning are integrated from tagged documents based on the characteristic words in the document set in which the tags are set. Whether or not the tag indicates a specific topic is determined based on the characteristic words. Further, it describes a method for automatically assigning a tag to a document based on the determination result of the importance of the tag and the integration result.

  Non-Patent Document 1 discloses a method of using an image and word learning data set to which correct tag information is assigned in advance. The method uses a co-occurrence probability model called pLSA (probabilistic Latent Semantic Analysis) and calculates the optimal topic when inputting the unknown input image test from the training data set. And automatically add tags.

  In Non-Patent Document 2, index information is assigned to an image by using an image-feature association model called 2DMHMM (two-dimensional multi-level hidden Markov model), using an image with category information added in advance as learning data. A method is disclosed.

  Non-Patent Document 3 proposes an automatic tagging technique using a pLSA model using image and tag information from Flickr as learning data. Non-Patent Document 4 describes pLSA used in Non-Patent Document 1 and the like. Flickr is a type of external photo sharing site where an unspecified number of people can post and share photos and videos and tag them freely.

  Non-Patent Document 5 describes SIFT feature values used in Non-Patent Documents 1 and 2 and the like. The SIFT feature amount can represent a local image pattern. For example, the feature amount of the learning image data group is k-means clustered to create a C type codebook. Each image is expressed as a set of C types of codebooks.

JP 2008-310626 A

Florent Monay and Daniel Gatica-Perez, “PLSA-based image auto-annotation: constraining the latent space“, Proc. Of ACM Multimedia, pp. 348-351, 2004. Jia Li and James Ze Wang. Automatic linguistic indexing of pictures by a statistical modeling approach.IEEE Trans. Pattern Anal. Mach. Intell, 25 (10): 1075_1088, 2003. Stefan Romberg, Eva Horster, Rainer Lienhart, Multimodal pLSA on visual features and tags, Proc. Of ICME 2009, pp 414-417, 2009 T. Hofmann, “Unsupervised Learning by Probabilistic Latent Semantic Analysis”, Machine Leaning, vol.42, pp.177-196 (2001). David G. Lowe, "Object Recognition from Local Scale-Invariant Features," iccv, vol. 2, pp.1150, Seventh International Conference on Computer Vision (ICCV'99)-Volume 2, 1999

  In Patent Document 1, characteristic and similar tag information can be automatically assigned to an unknown document from a document existing on the Internet. However, when the learning data is not sufficient, the accuracy of grant is not guaranteed. For example, there is a possibility that there is a difference in accuracy between tag information assigned based on a large amount of learning data and tag information assigned based on a small amount of learning data.

  In Non-Patent Documents 1 and 2, learning data to which correct information (tag) is assigned in advance is used. This is based on the premise that there is a lot of learning data to which correct information is assigned, and data that is manually assigned is used in the literature.

  As described above, it is difficult to collect a large amount of learning data to which correct information is manually added. In Non-Patent Document 3, automatic tagging is performed using Flickr image information and tag information as learning data. A system is being built.

  On the other hand, the target language information for these technologies is for major languages such as English (major language system), and when a system (minor language system) is constructed using minor languages such as Japanese. May be less accurate than major language systems.

Actually, as an English tag, "car, dog, fireworks, flower, food, hanami, ski, sumo, tokyotower, sea, bird, bike", as a Japanese tag, "car, dog, fireworks, flowers, food , Hanami, Skiing, Sumo, Tokyo Tower, Sea, Birds, Bicycle "and the search results in Flickr are shown in the table below. In the search, ski and skiing, food and food, ski and ski area were entered as separate queries. We also searched for all English queries using a combination of japan and japanese. Furthermore, the tags attached to the collected images were obtained by deleting the stop words, etc., and obtaining the numerical values shown in the table below. The search period was from October 1, 2005 to October 1, 2009. The number of tags is the total number of tags assigned to all image data. The number of unique tags is obtained by deleting duplicate tags.

  In this way, there is a large difference in the number of tags and the number of image data between major languages and minor languages. If an automatic tagging system is constructed using these as learning data, the accuracy may differ greatly. is there.

  The present invention has been made in view of such circumstances, and it is an object of the present invention to provide a tag assignment device, a conversion rule generation device, and a tag assignment program that can improve the accuracy of automatic tag assignment in a minor language. And

  (1) In order to achieve the above object, a tag assigning device according to the present invention is a tag assigning device that assigns a tag in a minor language with low usage on the Web to the content, A tag selection unit that selects a tag in a major language suitable for the target content in accordance with a feature amount detected from the target content with reference to tag selection data in which a tag in the major language is associated; And a tag conversion unit that converts the tag in the selected major language according to the conversion rule into a tag in a minor language that is easily given at the same time.

  As described above, in the tag assigning device of the present invention, the major language tag assigned according to the feature amount is converted into a minor language tag that is easily given at the same time. As a result, even in a minor language such as Japanese, the accuracy of automatic tagging can be improved as in the case where a major language such as English is used as learning data.

  (2) In the tag assigning device of the present invention, the tag conversion unit uses a rule that is dynamically updated using information on the Web as the predetermined conversion rule. Thereby, since the conversion rule can be dynamically updated based on information on the Web, relatively new words and slang words not included in the dictionary can be appropriately converted. For example, image data and tag information corresponding to a keyword can be acquired by using an image search API provided by Flickr or the like.

  (3) Further, in the tag assigning device of the present invention, the tag conversion unit uses a conversion rule reflecting a co-occurrence frequency in which a major language tag and a minor language tag are assigned to the same content. It is said. Thus, the co-occurrence frequency can be used to convert the tag into a minor language tag strongly related to the major language tag.

  (4) Moreover, the tag provision apparatus of this invention is characterized by the said tag conversion part using a conversion matrix as said conversion rule. As a result, conversion rules can be easily generated, and only the generated conversion matrix can be acquired to convert a tag from a major language to a minor language.

  (5) The conversion rule generation apparatus of the present invention is a conversion rule generation apparatus that generates a conversion rule from a phrase in a major language to a phrase in a minor language. Using a content acquisition unit for acquiring content tagged with a minor language tag with low usage on the Web, a database for storing the acquired content and tag, and the stored content and tag And a conversion rule generation unit that dynamically generates a conversion rule for converting a tag in a major language into a tag in a minor language that is easily given at the same time. Accordingly, it is possible to dynamically generate a conversion rule for converting a tag in a major language into a tag in a minor language that is easily given at the same time. As a result, it is possible to construct a system for automatically attaching tags without requiring manual labor.

  (6) Moreover, the tag provision system of this invention is provided with said tag provision apparatus and said conversion rule production | generation apparatus, It is characterized by the above-mentioned. Accordingly, the tagging device can convert the major language tag into the minor language tag using the conversion rule generated by the conversion rule generation device.

  (7) The tagging program of the present invention is a tagging program that causes a computer to execute a tag in a minor language with low usage on the Web, to the content. A process of selecting a tag in a major language suitable for the target content according to a feature amount detected from the target content with reference to tag selection data in which a tag in the major language is associated, and a predetermined conversion rule And a process of converting a tag in the selected major language into a tag in a minor language that is easily given at the same time.

  According to the present invention, it is possible to improve the accuracy of automatic tagging in a minor language. In addition, since conversion rules are used, tags can be automatically assigned without requiring manual labor.

It is a block diagram which shows the structure of the tag provision system which concerns on this invention. It is a flowchart which shows an example of operation | movement of the conversion rule production | generation apparatus which concerns on this invention. It is a flowchart which shows operation | movement of the tag provision apparatus which concerns on this invention. It is a figure which shows an example of a tag provision process. It is a graph which shows the precision average and standard deviation with respect to each tag provision method. It is the ranking evaluation result average and standard deviation for each tag assignment method. It is a figure which shows the example of each conversion result of a conversion matrix and a dictionary system.

  Embodiments of the present invention will be described below with reference to the drawings. In order to facilitate understanding of the description, the same reference numerals are given to the same components in the respective drawings, and duplicate descriptions are omitted.

(Configuration of tagging system)
FIG. 1 is a block diagram showing the configuration of the tag assignment system 100. As shown in FIG. 1, the tag assignment system 100 includes a conversion rule generation device 110 and a tag assignment device 120. In the tag assignment system 100 described below, the conversion rule generation device 110 and the tag assignment device 120 are connected via a network, but may be a single device.

(Configuration of conversion rule generator)
The conversion rule generation device 110 includes a content acquisition unit 111, a database 112, a tagging model construction unit 113, and a conversion rule generation unit 114, and generates a conversion rule from words in major languages to words in minor languages. Thereby, it is possible to generate a conversion rule for converting a tag in a major language into a tag in a minor language that is easily given at the same time.

  As a result, it is possible to construct a system for automatically attaching tags without requiring manual labor. The conversion rule generation device 110 may be provided as a server separately from the tag assignment device 120, or may be provided as a PC integrally with the tag assignment device 120, for example.

  The content acquisition unit 111 acquires, from the Web, content that has been tagged by a non-specific user with a major language and a minor language tag that is less used on the Web than the major language. For example, content with a tag is acquired from the photo sharing site 180. The content includes not only image data but also music data, moving image data, text data, and the like. For example, image data groups serving as learning data and tag information corresponding to the image data groups are collected from the external API 190 or the like. Further, the collected image data group and tag information are stored in the database 112.

  The database 112 includes a major language learning database (major language learning DB) 112a and a minor language learning database (minor language learning DB) 112b, and stores acquired contents and tags. The database 112 can also be provided outside via a network. Further, the database 112 does not necessarily need to be divided into the major language learning DB 112a and the minor language learning DB 112b.

  The major language learning DB 112a stores an image data group to which a tag is assigned according to the major language and tag information corresponding to the image data group. As a file format of the image data group, a JPEG format, a PNG format, or the like can be used, but is not limited thereto. The tag information represents text information provided by the user, shooting date and time, GPS information, and the like.

  Similar to the major language learning DB 112a, the minor language learning DB 112b stores an image data group to which a tag is assigned by a minor language and tag information corresponding to the image data group. However, regarding the data to be stored, data having a larger amount of data (number of images, number of tags, number of unique tags) or higher quality than the minor language learning DB 112b is defined as the major language learning DB 112a. Therefore, for example, it is possible to automatically select a database having a small number of data as the minor language learning DB 112b and switch between the major language and the minor language.

  The tagging model construction unit 113 constructs a tagging model using content and tags stored in the major language learning database 112a. The tagging model construction unit 113 constructs a tagging model necessary for automatic tagging based on the image data / tag information group stored in the major language learning DB 112a. For example, the correspondence between the feature quantity and the tag is constructed using learning data to which a tag has been assigned in advance. In addition, any model constructed based on learning data can be used as the tagging model construction unit 113.

  The conversion rule generation unit 114 dynamically generates a conversion rule for converting a tag in a major language into a tag in a minor language that is easily given at the same time, using the stored content and tag. Thereby, since the conversion rule can be dynamically updated based on information on the Web, relatively new words and slang words not included in the dictionary can be appropriately converted. For example, image data and tag information corresponding to a keyword can be acquired by using an image search API provided by Flickr or the like.

  The conversion rule generation unit 114 generates a conversion rule for converting a major language into a minor language from the database 112. For example, the conversion matrix is calculated by extracting image data including tags in both languages and calculating the co-occurrence frequency. Thus, the co-occurrence frequency can be used to convert the tag into a minor language tag strongly related to the major language tag. In the example described below, a conversion matrix is used as a conversion rule, but a correspondence relationship of an existing dictionary conversion system may be used. Further, the generation of the conversion matrix and the use of the dictionary system may be performed complementarily.

If the major language is represented by L_ {A} and the minor language is represented by L_ {B}, the vocabulary in the languages L_ {A} and L_ {B} can be represented by the following equations (1) and (2).

Here, M and N represent vocabulary sizes in the respective languages, and w_ {mA} and w_ {nB} represent tags in each vocabulary. When the transformation matrix T_ {L_ {A} → L_ {B}} is expressed on the basis of W_ {A} and W_ {B}, it is expressed as Equation (3) as an M × N matrix.

  Here, the element t _ {(m, n)} represents how much the tags w_ {mA} and w_ {nB} co-occur and are attached as tags, and the tags w_ {mA} and w_ { nB} co-occurrence count cnt (w_ {mA}, w_ {nB}). Moreover, you may express as a co-occurrence probability.

(Configuration of tagging device)
The tag assigning device 120 includes a tag selection unit 121 and a tag conversion unit 122, and assigns a tag in a minor language with low usage on the Web to content. Thereby, even in a minor language such as Japanese, the accuracy of automatic tagging can be improved by converting it into a major language such as English as in the case where the major language is used as learning data. A tag in a major language can be automatically assigned to unknown input image data using the tag assignment model obtained by the tag assignment model construction unit 113.

  The tag assignment device 120 may be provided separately from the conversion rule generation device 110 as a mobile terminal or a PC, for example, or may be provided as a PC integrally with the conversion rule generation device 110. For example, when tagging is performed with a portable terminal, it is more efficient to generate the conversion rule with a separate device such as a server. However, in the case of using a portable terminal, it is possible to limit by parameters such as using only the upper predetermined number depending on the frequency.

  The tag selection unit 121 refers to the tag selection data in which the feature amount of the content and the tag in the major language are associated, and the tag in the major language suitable for the target content according to the feature amount detected from the target content Select. Thereby, a suitable tag according to the feature of image data can be given. The tag selection unit 121 uses the tag addition model obtained by the tag addition model construction unit 113 to automatically add a tag in a major language to unknown input image data.

  The tag conversion unit 122 converts a tag in a major language selected according to a predetermined conversion rule into a tag in a minor language that is easily given at the same time. The tag conversion unit 122 uses a predetermined conversion rule that is dynamically updated using information on the Web. The tag conversion unit 122 uses a conversion rule reflecting the co-occurrence frequency assigned to the same content in the major language tag and the minor language tag. The tag conversion unit 122 uses a conversion matrix as a conversion rule. As a result, conversion rules can be easily generated, and only the generated conversion matrix can be acquired to convert a tag from a major language to a minor language.

  The tag conversion unit 122 converts the major language tag information for the unknown image data provided by the tag selection unit 121 into a minor language based on the conversion matrix obtained by the conversion rule generation unit 114. To convert w_ {mA} to minor language w_ {nB}, cnt (w_ {mA}, w_ {nB}) is maximized using the conversion matrix described in Equation (3) Convert to w_ {m_ {max} A}. Further, when a plurality of the same conversion results are obtained, the tag conversion unit 122 can use the one holding a high value of cnt {w_ {mA}, w_ {nB}} as the conversion result.

  Furthermore, the tag conversion unit 122 can also use the external WEBAPI or the like to integrate the major language tag with the result of simply applying dictionary conversion or the like to obtain the final result. For example, it is possible to use the conversion result by using the site described in the reference information.

  The tag conversion unit 122 can realize the same processing in the automatic tag addition method for other media by applying a tag addition model other than an image such as music or a blog in the tag addition model construction unit 113. it can.

(Operation of conversion rule generator)
Next, the operation of the conversion rule generation device 110 will be described. FIG. 2 is a flowchart illustrating an example of the operation of the conversion rule generation device 110. First, search keywords in major and minor languages are set (step S1). If the major language is English and the minor language is Japanese, `` car, dog, fireworks, flower, food, hanami, ski, sumo, tokyotower, sea, bird, bike '', `` car, dog, fireworks, flowers , Food, cherry blossom viewing, skiing, sumo, Tokyo Tower, sea, birds, bicycles, etc., and corresponding keywords are set. In addition, for example, various languages such as Chinese and Spanish can be used.

  Next, image data and tag information are acquired from an external photo sharing site or the like based on the keyword group set in the major language (step S2). For example, image data and tag information corresponding to a keyword can be acquired by using an image search API provided by Flickr or the like. The acquired image data and tag information are stored in the major language learning DB 112a (step S3).

  Next, image data and tag information are acquired from an external photo sharing site or the like based on the keyword group set in the minor language (step S4). The acquired image data and tag information are stored in the minor language learning DB 112b (step S5). Next, a tagging model is generated based on the data stored in the major language learning DB 112a (step S6). Finally, a conversion rule is created based on the data stored in the database 112 (step S7), and the operation ends.

(Operation of tagging device)
Next, the operation of the tag assignment device 120 will be described. FIG. 3 is a flowchart showing the operation of the tag assigning device 120. First, a tagging model is acquired from the conversion rule generation device 110 (step T1). Similarly, a conversion rule is acquired from the conversion rule generation device 110 (step T2).

  Next, a tag in a major language is selected using the tagging model acquired for the target content to be tagged (step T3). And the tag by the selected major language is converted into the tag of a minor language using a conversion rule (step T4). Then, the target content and the minor language tag are stored in association with each other (step T5), and the operation ends. Each of the above operations can be realized by causing a computer to execute a program.

(Example of tagging process)
FIG. 4 is a diagram illustrating an example of a tag assignment process. For example, the image data group 211 and English and Japanese tags 212 assigned to the image data group 211 are stored in the database 112. In the example shown in Fig. 4, the seaside photo is tagged with the English "sea, beach, japan" and the Japanese "Japan, sea", and the underwater photo where fish swims is English "fish, blue" , japan "and Japanese" Japan, blue "tags. The coastal photos are tagged with English "sea, blue, beach, rocks" and Japanese "blue, sea, rocks", and the rocks protruding above the ground are English "rocks". And the Japanese “rock” tag.

  In such a case, a transformation matrix 223 between the row 221 of “sea, beach, japan, blue, rocks” and the column 222 of “sea, Japan, blue, rocks” can be created using the co-occurrence frequency. For example, the number of elements of the matrix corresponding to the “sea” column of the “sea” row is 2 because it co-occurs in the above-mentioned seaside photograph and coastal photograph. Similarly, the conversion matrix 223 can be created by expressing each element with the co-occurrence frequency. By selecting the most frequently used phrase using this conversion matrix 223, conversion 230 from English tags to Japanese tags becomes possible.

  Using the tag assignment system 100 as described above, the user can take in a large amount of image data taken with a mobile phone into a PC and automatically assign an optimum tag. At this time, it is possible to omit the troublesome work of thinking and tagging by yourself. Then, the search function can be used to filter according to the place and situation where the image was taken. For example, if you want to access that photo taken at the sea, you can use the keyword “sea” to narrow down the candidates for the photo that contains “the sea”. In the above example, English is selected as the major language and Japanese is selected as the minor language. However, it is also possible to select Spanish as the major language or Arabic as the minor language.

(Tagging system used)
Experiments were performed using the tagging system 100 described above. In this experiment, the system of the embodiment (co-occurrence conversion system, dictionary conversion system) using a co-occurrence conversion and a dictionary conversion as the conversion rule generation unit 114 was constructed using English as learning data. For the dictionary conversion system, Google (registered trademark) translation API was used. In addition, as a comparative example system, a tagging model constructed using Japanese learning data as learning data instead of the conversion rule generation unit 114 was used. Therefore, a total of 3 systems were constructed.

  The experimental data uses an image data group (116,273 images) to which both languages are assigned at the same time, and in all collected image data, BoVW feature amount 300 dimensions, hue feature amount 150 dimensions, LBP 250 dimensions, edge direction histogram 73 A total of 773 dimension features were extracted. For BoVW feature values, 10% of image data was randomly selected from the experimental data, and after extracting SIFT feature values, a codebook was created by the k-means method. In each language, 90% of image data was used as learning data, and 10% of data was used as test data.

(experimental method)
As test data, five pieces of image data were selected at random from the items shown in Table 1, and a total of 60 pieces of image data were used in the experiment. The top ten tags of the tags obtained from the system were assigned to all image data. (Total 30 tags) Twelve subjects gave an evaluation as to whether each tag is suitable for image data while browsing all image data and the results of the three methods. The evaluation criteria were three items, correct, incorrect, and unknown, and the evaluation criteria for each item were set as shown in the following table.

  And ranking evaluation was performed by assigning ranks to all systems. First, the accuracy of each tag assigned by the system will be described. A correct evaluation average was calculated for each image from the subject's evaluation of the assigned tag.

  Furthermore, the result of calculating | requiring the average with respect to the whole is shown in FIG. 5, FIG. FIG. 5 is a graph showing the average accuracy and standard deviation for each tagging method. FIG. 6 shows a ranking evaluation result average and standard deviation for each tag assigning method. The average for the Japanese system is 11.1% (standard deviation: 7.98), and the average for the dictionary conversion and co-occurrence conversion systems is 19.8% and 17.5% (standard deviation: 11.8, 9.58), respectively. It became.

  Furthermore, the average ranking evaluation results for each system are 2.3 (standard deviation: 0.36) for the Japanese tagging system, and 1.9 and 1.8 (standard deviation) for the dictionary conversion system and the co-occurrence conversion system, respectively. : 0.20, 0.33). Also, the t-test confirmed that the average difference between the dictionary conversion and co-occurrence conversion system evaluation results was statistically significant for the tag accuracy and ranking evaluation results of the Japanese system (α = 0.05). , P <0.01).

  From these results, it was confirmed that the subjective evaluation value of the translation system using English as learning data was higher than the result of the Japanese tagging system, and English learning data with a large amount of tag information was used as learning data. It was confirmed that the subjective evaluation value increased by converting to the word tag.

  There was no statistical difference between the accuracy of dictionary conversion and co-occurrence conversion system. However, compared to the dictionary conversion system, the co-occurrence conversion system can convert words that are not registered in the dictionary, and there is an example in which conversion closer to a human sense is possible. It can be expected that the accuracy will be further improved if the learning data of the tag increases. FIG. 7 is a diagram illustrating conversion result examples of the conversion matrix and the dictionary system. In FIG. 7, for an English tag 310, a tag 320 converted by a conversion matrix and a tag 330 converted by a dictionary conversion system are displayed. Higher accuracy can be expected by successfully integrating the dictionary transformation and co-occurrence transformation results.

DESCRIPTION OF SYMBOLS 100 Tag assignment system 110 Conversion rule production | generation apparatus 111 Content acquisition part 112 Database 112a Major language learning DB
112b Minor language learning DB
113 Tag addition model construction unit 114 Conversion rule generation unit 120 Tag assignment device 121 Tag selection unit 122 Tag conversion unit 211 Image data group 212 Tag 221 row 222 column 223 conversion matrix

Claims (7)

A tag granting device that grants a tag in a minor language with low usage on the Web to content,
Tag selection for selecting a tag in a major language suitable for the target content according to the feature amount detected from the target content with reference to tag selection data in which the feature amount of the content is associated with a tag in the major language And
A tag assigning device, comprising: a tag conversion unit that converts a tag in the selected major language into a tag in a minor language that is easily attached at the same time according to a predetermined conversion rule.   The tag adding apparatus according to claim 1, wherein the tag conversion unit uses a rule that is dynamically updated using information on the Web as the predetermined conversion rule.   3. The tag assignment according to claim 1, wherein the tag conversion unit uses a conversion rule reflecting a co-occurrence frequency in which a major language tag and a minor language tag are assigned to the same content. apparatus.   4. The tag assigning device according to claim 1, wherein the tag conversion unit uses a conversion matrix as the conversion rule. A conversion rule generation device that generates a conversion rule from a phrase in a major language to a phrase in a minor language,
A content acquisition unit that acquires, from the web, content tagged with a minor language that is less used on the web than the major language and the major language by an unspecified user;
A database for storing the acquired content and tags;
A conversion rule generating unit that dynamically generates a conversion rule for converting a tag in a major language into a tag in a minor language that is easily given at the same time using the stored content and tag; Rule generator. The tag attachment device according to any one of claims 2 to 4,
A tagging system comprising: the conversion rule generation device according to claim 5. A tag granting program that causes a computer to execute a tag in a minor language with low usage on the web to grant content,
Processing for selecting a tag in a major language suitable for the target content according to a feature amount detected from the target content with reference to tag selection data in which the feature amount of the content is associated with a tag in the major language; ,
And a process for converting a tag in the selected major language into a tag in a minor language that is easily added at the same time according to a predetermined conversion rule.

Images