JP5474591B2 - Image selection apparatus, image selection method, and image selection program - Google Patents

Description

  The present invention relates to an image selection device, an image selection method, and an image selection program for selecting an image related to content.

  2. Description of the Related Art Conventionally, an apparatus that reproduces music such as BGM in accordance with image display is known. For example, the information playback apparatus described in Patent Document 1 enhances the video viewing effect by playing back a BGM that matches the shooting time of an image to be played back when playing back a slide show of images taken with a digital camera. Employs a method of searching for music and assigning BGM automatically, focusing on images, and music by matching the date of release of the song or the period when the song's popularity ranking is the highest with the shooting time. Searching for.

  The music distribution server described in Patent Literature 2 selects a song stored in the database when reproducing a song, and specifies items such as lyrics, commentary, and images to be displayed in synchronization with the corresponding song. And the time interval for synchronous reproduction is written in the corresponding music and the header of the corresponding music and transmitted to the terminal, and the music and other information are synchronized and reproduced.

  The portable information terminal described in Patent Literature 3 extracts lyrics by voice recognition during music reproduction, extracts keywords from the extracted lyrics based on a predetermined rule, and uses local storage or the Internet on the basis of the keywords. Images are retrieved from a web page and displayed when music is played.

  The predetermined rule is, for example, a rule that uses the most frequently used words in a lyric word within a certain setting period as an extraction keyword, or a word that is uttered most among lyric words within a certain setting period. Are extracted keywords, and a word randomly selected from a plurality of words within a certain setting period is used as a keyword.

  When an image search is performed using a web page on the Internet as a search destination, an image is searched using a search engine on the Internet using a keyword. In addition, when the web page on the Internet is a search destination, a filtering process for preventing an undesired image from being searched or a filtering process for searching only an image of a desired genre, for example, is performed. It also makes it possible.

  Note that the filtering process may be performed not only when the image search is performed via the Internet but also when the keyword extraction unit extracts a keyword, for example. As a result, in the keyword extraction unit, a filtering process that does not extract a keyword that is likely to cause an undesired image to be searched, and conversely, a keyword that is highly likely to search for an image of a desired genre, for example. Filtering processing that extracts only this is also possible.

  The image presentation device described in Patent Document 4 extracts lyrics from music, specifies the appearance time of the extracted lyrics, associates a plurality of visual effects with time information required to display the visual effects in advance, and extracts them. The visual effect is determined based on the lyrics, and the visual effect with movement is given according to the contents of the lyrics of the music.

  The visual effect database stores words and visual effects in association with each other. The visual effect is searched for whether or not the word in the extracted lyrics exists in the visual effect database. If the word exists, the visual effect corresponding to the word and the time required for the visual effect are displayed in the visual effect database. Is getting from.

  In addition, research and development have been conducted on a system for displaying a Web image in synchronization with the reproduction of music and an automatic music classification technique for searching music based on the impression of lyrics (for example, Non-Patent Documents 1 to 5).

JP 2006-164229 A JP 2002-73049 A JP 2008-8954 A JP 2006-154626 A

Shintaro Funasawa, Hiromi Ishibe, Keiichiro Hoashi, Yasuhiro Takishima, Jiro Katto, “Proposal of Web Image and Music Synchronized Playback System Considering Lyric Features”, 8th Information Technology Forum, 2009, E-034 D. A. Shamma, B. Pardo, and K. J. Hammond, “Music Story: a Personalized Music Video Creator, Proceedings of the 13th Annual ACM International Conference on Multimedia”, 2005, pp.563-566 R. Cai, L. Zhang, F. Jing, W. Lai, and W. -Y. Ma, “Automated Music Video Generation Using Web” “Image Resource, Proceedings of IEEE International Conference on Acoustic, Speech, and Signal Processing”, 2007, 2, pp.737-740. Shintaro Funasawa, Hiromi Ishibe, Keiichiro Hoashi, Yasuhiro Takishima, Jiro Katto, Study on automatic music classification for music search based on the impression of lyrics, 71st National Convention on Information Processing, 2009, 5R-2.

  As described above, there is a technique for using an image on the Web in accordance with a music piece or the like, but the user must basically select an image to be used. If the user himself selects an image from a Web image and tries to add it to the content, the work requires an enormous amount of work. For example, the work of selecting an image when generating a slide show or the like requires an enormous amount of work. Become. In addition, when the conventional technology is used, when an image is selected based on the ranking of the shared site, an image suitable for the content cannot always be selected.

  The present invention has been made in view of such circumstances, and an image selection apparatus, an image selection method, and an image selection apparatus that can automatically select an image suitable for content and eliminate the effort of the user himself to select an image on the Web. The purpose is to provide an image selection program.

 (1) In order to achieve the above object, an image selection device according to the present invention is an image selection device that selects an image related to content, and includes a keyword extraction unit that extracts one or a plurality of keywords from specific content; A tag extraction unit that extracts one or a plurality of tags attached to an image that can be acquired on the network, and a phrase relevance calculation unit that calculates the relevance of the phrase between the extracted keyword and the acquired tag And a selection unit that selects an image having a high degree of matching with the content using the calculated degree of association of the phrases.

  In this way, since the image is selected using the degree of association between the keyword related to the content and the tag attached to the image, an image suitable for the content can be automatically selected, and the user himself has the effort to select an image on the WEB. Can be eliminated.

  (2) Further, the image selection device of the present invention is characterized in that the keyword extraction unit extracts one or a plurality of keywords from a specific music piece. Thereby, an image suitable for music can be selected, and can be used for image display during music reproduction.

  (3) Further, in the image selection device according to the present invention, the phrase relevance calculation unit uses a co-occurrence probability between one phrase of the extracted keywords and one phrase of the acquired tags. The degree of relevance of the word is calculated. In this way, by using the co-occurrence probability, it is possible to calculate the degree of association of words suitable for image selection.

  (4) Further, in the image selection device according to the present invention, the phrase relevance calculation unit is configured so that, for other phrases belonging to the same category as the one phrase among the extracted keywords, The relevance of the phrase is calculated by subtracting the contribution of the co-occurrence probability with the phrase. In this way, by subtracting the contribution of another phrase in the same category, the degree of association between the keyword and the tag can be accurately evaluated, and the degree of association of the phrase suitable for image selection can be calculated.

  (5) The image selection method of the present invention is an image selection method for selecting an image related to content, the step of extracting one or a plurality of keywords from specific content, and an image obtainable on a network. Using the step of acquiring one or more tags attached to the tag, calculating the relevance level of the phrase between the extracted keyword and the acquired tag, and using the calculated relevance level of the phrase And a step of selecting an image highly related to the content. As a result, an image suitable for the content can be automatically selected, and the user can eliminate the labor of selecting an image by WEB.

  (6) The image selection program according to the present invention is an image selection program for selecting an image related to content, and includes a process for extracting one or a plurality of keywords from specific content and an image that can be acquired on a network. Using the processing for acquiring one or more tags assigned to, the processing for calculating the degree of association between the extracted keyword and the acquired tag, and the degree of association between the calculated words It is characterized by having a computer execute processing for selecting an image highly related to content. As a result, an image suitable for the content can be automatically selected, and the user can eliminate the labor of selecting an image by WEB.

  According to the present invention, it is possible to automatically select an image suitable for content, and it is possible to eliminate the effort of the user himself to select an image on the Web.

It is a block diagram which shows the structure of the image selection apparatus which concerns on 1st Embodiment. It is a block diagram which shows the structure of a module. It is a flowchart which shows an example of operation | movement of the image selection apparatus which concerns on 1st Embodiment. It is a flowchart which shows the operation | movement of word relevance calculation. It is a flowchart which shows the operation | movement of image selection. It is a table which shows an example of a phrase related degree dictionary. It is a table which shows an example of the extraction result of a related tag and a phrase relevance degree. It is a block diagram which shows the structure of the image selection apparatus which concerns on 2nd Embodiment. It is a block diagram which shows the structure of a module.

  Embodiments of the present invention will be described below with reference to the drawings. In order to facilitate understanding of the description, the same reference numerals are given to the same components in the respective drawings, and duplicate descriptions are omitted.

[First Embodiment]
(Configuration of image selection device)
FIG. 1 is a block diagram showing the configuration of the image selection apparatus. The image selection device 100 selects an image related to content such as music specified by the user U. In addition to music, the content includes blogs and videos. For the image selection device 100, for example, a PC or a mobile phone is used. As shown in FIG. 1, the image selection apparatus 100 includes a request reception unit 111, a database 112, an image search module 120, a phrase relevance dictionary generation module 130, an image selection module 140, an image management unit 151, a display time management module 160, A playback control unit 171, a playback unit 172, a lyrics display unit 173, and an image display unit 174 are provided.

  The request receiving unit 111 receives an input of a music or image request from the user U. The database 112 stores content and keywords for the content. For example, if the content is a music, the sound source of the music, the lyrics information, the synchronization information of the music and the lyrics, and the like are stored. In addition, a keyword for the corresponding content is stored, for example, a keyword expressing the content is stored as impression information.

  For example, impression information of “spring”, “summer”, “autumn”, and “winter” is stored in the “season” category. Impression information of “morning”, “daytime”, “evening”, and “night” may be stored in the “time zone” category. Further, impression information of “sunny”, “cloudy”, “rain”, “snow”, and “rainbow” may be stored in the “weather” category.

  For example, if the content is music, the image search module 120 searches the Web for images (hereinafter referred to as candidate images) that are candidates for the slide show based on the lyrics information of the music input by the user. Specifically, an optimal query is selected using nouns appearing in the lyrics and words expressing the impression of the entire lyrics as keywords. Then, a Web image search is performed using the query, and a maximum of about 1500 candidate images are acquired for each line of lyrics. It is possible to extract nouns from the lyric information using a Japanese morphological analysis engine published on the Web. Further, as an image database to be searched, an image sharing site 400 such as Flickr (registered trademark) can be used, and an image search can be performed based on a social tag attached to the image data.

  The word / phrase relevance dictionary generation module 130 calculates the word / phrase relevance between the keyword and the tag of the candidate image based on the keyword assigned to the content. In addition, the calculated degree of association is stored as a dictionary.

  The image selection module 140 selects an image suitable for the keyword assigned to the content from the image group obtained by the image search module 120. For each candidate image, a score expressing the degree of matching with the overall impression word of the content is calculated, and an image having the highest value is selected to select an image to be finally used from the candidate images.

  For example, the score of fitness is calculated based on the strength of association between all social tags attached to the image and the overall impression word. This makes it possible to express “likeness of the overall impression word” even in an image in which the overall impression word is not directly assigned as a tag. Then, by selecting images based on that, it is possible to create a unified feeling for the entire slide show. In addition, since the score calculation index changes depending on the combination of the overall impression words assigned to the music, the same image is not selected for different music, and a more diverse slide show can be created.

  Details of the phrase relevance dictionary generation module 130 and the image selection module 140 will be described later. The image management unit 151 manages each music management category in association with the selected image. The image management unit 151 can display an image corresponding to the lyrics display of each section by the correspondence between each section of the music and the image during reproduction.

  The display time management module 160 reconfigures the image switching timing in order to make the time for displaying each image appropriate in the slide show. At that time, the music database 500 and the lyrics database 600 on the network are used.

  If the image to be displayed together with the lyrics is simply switched in units of lyrics, the display time of the image depends on the length of the line of the lyrics, and an appropriate display time does not occur. In order to prevent this, a sufficient display time is taken so that the contents of the image can be sufficiently grasped, and the display time is shortened to such an extent that a feeling of boredom does not occur. That is, a line with a short display time is combined with surrounding lines, and one image is continuously displayed in the combined section, while a line with a long display time is divided, and a plurality of images are displayed in one line. Furthermore, the image switching timing in the interlude section is also set based on the mode value of the music image display time.

  The reproduction control unit 171 communicates with the lyrics database 600 and controls the reproduction of music and the display of lyrics and images synchronized with the reproduction of the music. The playback unit 172 plays back the sound of the music via an output device such as a speaker or headphones. The lyrics display unit 173 displays the lyrics on the screen in accordance with the reproduction of the music. The image display unit 174 displays an image selected over the calculated image display time in accordance with the reproduction of the music. The image display unit 174 may display an image corresponding to the interval when the interval between the lyrics display times corresponding to successive lines exceeds a predetermined time.

  FIG. 2 is a block diagram showing the configuration of the module. As shown in FIG. 2, the phrase relevance level dictionary generation module 130 includes a category extraction unit 131, a keyword extraction unit 132, a tag extraction unit 133, and a phrase relevance level calculation unit 134.

  The category extraction unit 131 extracts related categories from the content specified by the user U (specific content). The keyword extraction unit 132 extracts one or more keywords from the category. That is, when the content is a song, one or more keywords are extracted from the lyrics of the song and the overall impression word. In this way, the keyword of the music is used for calculating the degree of word association with the image tag and used for image selection. The category may be given manually by the user, or may be extracted from the content by some program as described above.

  The tag extraction unit 133 extracts one or a plurality of tags attached to images that can be acquired on the network. By using the image sharing site 400, a large number of images and tags attached to the images can be extracted.

  The phrase relevance calculation unit 134 calculates the relevance of the phrase between the keyword extracted from the content and the tag extracted from the image. In this way, using the category information assigned to the content, an image suitable for the content is selected using the degree of association between the keyword related to the content and the tag attached to the image. As a result, it is possible to automatically select an image suitable for the content, and it is possible to eliminate the labor for the user himself to select an image on the Web. It can also be used to add images to multimedia contents such as slide shows and blogs.

  The phrase relevance calculating unit 134 calculates the relevance of the phrase using the co-occurrence probability of one word of the extracted keywords and one of the acquired tags. By using the co-occurrence probability in this way, it is possible to appropriately and easily calculate the degree of association of words suitable for image selection.

  In that case, the degree of relevance of the phrase is obtained by subtracting the contribution of the co-occurrence probability with the phrase in the acquired tag from another phrase belonging to the same category as the one of the extracted keywords. It may be calculated. Thus, by subtracting the contribution of another phrase in the same category, the phrase relevance of the keyword and the tag can be accurately evaluated, and the relevance of the phrase suitable for image selection can be calculated.

  The image selection module 140 includes a tag acquisition unit 141, an image suitability calculation unit 142, and a selection unit 143. The tag acquisition unit 141 acquires a tag attached to the image from the images extracted as candidates.

  The image suitability calculation unit 142 calculates the suitability between the content designated by the user U (specific content) and the candidate image. At this time, it is preferable to cumulatively add the degree of association between the tag attached to the image and the keyword related to the content, whereby a reasonable degree of association can be calculated. Specific examples will be described later.

  The selection unit 143 selects an image having a high degree of fitness for the calculated word / phrase. An image is selected using the degree of matching between the content keyword and the image tag. Thereby, an image suitable for music can be selected, and can be used for image display during music reproduction.

(Operation of image selection device)
Next, the operation of the image selection device 100 will be described. FIG. 3 is a flowchart illustrating an example of the operation of the image selection device. First, content specification by the user U is accepted (step S1), and keyword specification is accepted (step S2).

  Next, an image search is performed on the network (step S3). The word / phrase relevance between the keyword and the tag is calculated according to the search result (step S4), and an image is selected according to the calculated relevance of the word / phrase (step S5). Then, the display time of the selected image or lyrics is determined (step S6), the music is reproduced, the lyrics and the image are displayed along with the reproduction (step S7), and the process ends. Details of each process will be described below.

(Phrase relevance calculation)
The operation of the word relevance calculation will be described. FIG. 4 is a flowchart showing the operation of calculating the word association degree. As shown in FIG. 4, first, category information is extracted from the designated music (step T1). A keyword related to the category obtained from the music is extracted (step T2).

  Next, a tag group attached to the candidate image searched on the Web is extracted (step T3), and a tag is extracted from the extracted tag group (step T4). And it is determined whether the number of users is more than a threshold value about each extracted tag (step T5). If the number of users is less than the threshold, the process returns to step T4. If the number of users is greater than or equal to the threshold, the degree of association between the tag and the keyword of the music is calculated (step T6).

  After the relevance is calculated, it is determined whether or not the determination has been completed for all tags (step T7). If the determination is not completed for all tags, the process returns to step T4. If the determination is completed for all tags, it is determined whether the processing is completed for all keywords (step T8). If processing has not been completed for all keywords, the process returns to step T2.

  When the process is completed for all keywords, it is determined whether the process is completed for all categories (step T9). If the processing has not been completed for all categories, the process returns to step T1. When the processing is completed for all categories, a phrase relevance dictionary for each category is created (step T10). An example of the phrase relevance dictionary for each category will be described later.

(Image selection operation)
Next, an image selection operation will be described. FIG. 5 is a flowchart showing an image selection operation. First, an input image by the user is acquired (step P1). Then, a tag set in the image is acquired (step P2).

  Next, a score is calculated as the image suitability for the input image (step P3). Then, it is determined whether or not the processing has been completed for all images (step P4). When the processing has not been completed for all the images, the process returns to Step P1. When the processing has been completed for all the images, the ranking of the suitability of the images is calculated (step P5). Then, an image is selected based on the ranking (step P6).

(Specific examples of image selection)
A specific example of the image selection process will be described below. First, definitions using mathematical formulas such as categories and keywords will be described. A set W of keyword categories Cx given to content can be expressed by the following mathematical formula. Nc is the number of categories.

The category Cx can be expressed by the following formula. Nx is the number of keywords belonging to the category.

On the other hand, a tag set given to an image can be expressed by the following mathematical formula. NTwy is the number of tags searched by the keyword wy, and NTi is the total number of tags assigned to the content i.

The user set can be expressed by the following mathematical formula. NUwy is the number of posting users of images searched by the keyword wy.

  Image selection processing is performed under such a definition. First, a certain keyword wy is extracted, and based on the keyword, a tag set Twy and a user set Uwy are calculated from the result of Web search or the like. Next, the number of users UF given the corresponding tag is calculated by adding all the tags existing in the tag group Twy. For UF, for example, noise can be reduced by setting a threshold value TH. For example, TH = 5 can be set.

  Thereafter, the degree of association R of the word with wy is calculated for each element of the tag set Twy. The relevance level R of a phrase can be calculated by the following formula. P is the co-occurrence probability. The co-occurrence probability is a probability that a keyword and a tag are assigned to one and the same image, and can be used as an index representing the strength of association between the keyword and the tag. UF (t | wy) is the number of users who have posted images with the tag t among images searched by wy. Further, UF (wy) is the number of users who have posted images searched by wy. Weight is a weight coefficient and can be set arbitrarily.

  Equation (6) expresses the degree of association R of the phrase indicating the strength of the association between the keyword and the tag, and can be calculated based on the co-occurrence probability such as equation (5). Finally, a tag having a high degree of relevance R of words for each keyword is defined as a related tag set, and based on this, the relevance degree R of words and important words for each keyword can be held as a dictionary. For example, when the keyword is “spring”, since “spring” is an impression word of the concept of “season”, Cx = {“spring”, “summer”, “autumn”, “winter”} can be held.

  FIG. 6 is a table showing an example of the phrase relevance dictionary. In FIG. 6, the degree of association R between the words of each category Cx = {w1, w2, w3, w4} and the tag of each image related to each category corresponds. t_w1_1 is a tag attached to the image 1 and belonging to the category of w1. R (t_w1_1 | w1) is the degree of association R of the phrase between the keyword w1 and the tag t_w1_1 belonging to the category of the image 1 and w1.

  The relevance level R of the words and phrases defined in this way is calculated for the tag t that satisfies UF (t) ≧ TH that co-occurs with each overall impression word, for example, a tag whose relevance level R is 0.024 or more. Can be determined as related tags.

  FIG. 7 is a table showing an example of a result of extracting the degree of association between a related tag and a phrase. FIG. 7 shows related words (tags) and word relevance levels (value) for “spring”, “summer”, “morning”, and “rain”, sorted in descending order of word relevance levels. .

  The co-occurrence probability can be calculated based on UF (User Frequency) in Flickr (registered trademark), and UF (W) is the result of AND search using all the elements of word set W as a query. Indicates the number of unique image contributors. The reason for simply using UF instead of image hits is that in Flickr (registered trademark), if a certain user uploads a large number of images with the same tag, the number of image hits can be unreasonably high. This is to mitigate the effect.

  Then, the degree of association R of the phrase of tag t with keyword wy can be calculated from the co-occurrence probability P based on the difference in co-occurrence probability P with tag t in another keyword belonging to the same category as wy. Considering not only the co-occurrence probability between keywords and tags, but also the co-occurrence probability P of other keywords P belonging to the same category, the tag with a high co-occurrence probability only for one keyword can be regarded as important. it can.

Next, the image Ii is extracted from the image group obtained by the image search module 120. Next, the tag set Ti in Ii is acquired, and the score can be calculated from the keyword assigned to the content and the tag information of the image by the equation (7) or the like. Using the related dictionary obtained by the related dictionary generation module in the equation (7), the score can be calculated as the degree of matching of the image without using the related dictionary.

  For example, by using Equation (7) as the score of the degree of matching of an image, an image having a high degree of word relevance with a keyword and a small number of noise tags can be selected. For example, when the content is music, it can be set to take a larger value for an image to which more tags that are strongly related to the keyword in the music are assigned. Based on this score, images can be displayed in a ranking, and images up to the top N can be selected automatically.

[Second Embodiment]
In the above embodiment, the phrase relevance dictionary is dynamically created when image selection is necessary. However, it may be created in advance and used at the time of image selection. FIG. 8 is a block diagram illustrating a configuration of the image selection device 200 that creates a word-related degree dictionary in advance. FIG. 9 is a block diagram showing the configuration of the module. In the image selection device 200, the word / phrase relevance dictionary generation module 130 functions in advance, and then the image selection module 140 performs image selection. However, since the function of each part is the same as that of Embodiment 1, it abbreviate | omits.

DESCRIPTION OF SYMBOLS 100 Image selection apparatus 111 Request reception part 112 Database 120 Image search module 130 Phrase relevance dictionary generation module 131 Category extraction part 132 Keyword extraction part 133 Tag extraction part 134 Phrase relevance calculation part 140 Image selection module 141 Tag acquisition part 142 Image adaptation Degree calculation unit 143 Selection unit 151 Image management unit 160 Display time management module 171 Playback control unit 172 Playback unit 173 Lyric display unit 174 Image display unit 200 Image selection device 400 Image sharing site 500 Music database 600 Lyrics database U User

Claims (5)

An image selection device for selecting an image related to content,
A keyword extractor that extracts one or more keywords from specific content;
A tag extraction unit that extracts one or more tags attached to an image obtainable on the network;
The extracted keyword and the acquisition by subtracting the contribution of the co-occurrence probability with the phrase in the acquired tag from another phrase belonging to the same category as the one of the extracted keywords A phrase relevance calculator that calculates the relevance of the phrase with the tagged tag,
An image selection apparatus comprising: a selection unit that selects an image having a high degree of matching with the content by using the calculated degree of association of words and phrases.   The image selection apparatus according to claim 1, wherein the keyword extraction unit extracts one or a plurality of keywords from a specific music piece.   The phrase relevance calculation unit calculates the relevance of the phrase using a co-occurrence probability of one of the extracted keywords and one of the acquired tags. The image selection device according to claim 1 or 2. An image selection method for selecting an image related to content,
Extracting one or more keywords from specific content;
Obtaining one or more tags attached to an image obtainable on a network;
The extracted keyword and the acquisition by subtracting the contribution of the co-occurrence probability with the phrase in the acquired tag from another phrase belonging to the same category as the one of the extracted keywords Calculating the relevance of the term to the tagged tag;
Selecting an image highly related to the content using the calculated degree of association of words. An image selection program for selecting images related to content,
Extracting one or more keywords from specific content;
Processing to acquire one or more tags attached to images that can be acquired on the network;
The extracted keyword and the acquisition by subtracting the contribution of the co-occurrence probability with the phrase in the acquired tag from another phrase belonging to the same category as the one of the extracted keywords Calculating the degree of relevance of the word to the tagged tag,
An image selection program that causes a computer to execute processing for selecting an image highly related to the content by using the calculated degree of association of words.

Images