JP2014093058A - Image management device, image management method, program and integrated circuit - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image management device capable of performing hierarchical classification based on the content of images on an image group including various kinds of objects.SOLUTION: The image management device includes: object recognition means that recognizes the similarity with general models in the images to classify the images into semantic concept items; clustering means that classifies the images on the basis of the similarity in the feature amount among the images; detailed classification item selection means that selects an item from the semantic concepts which are classified by the object recognition means while giving the priority to the item in accordance with the diversity of the items in the semantic concepts; and a hierarchical structure generation section that generates a hierarchical classification structure of the image groups by integrating the classification results made by the object recognition means and the clustering means. The object recognition means classifies the image groups into partial image groups of the items in the semantic concept selected by the detailed classification item selection means. The clustering means performs classification on the partial image groups classified by the object recognition means.

Description

  The present invention relates to an image management apparatus for hierarchically classifying images.

  In recent years, digital image capturing devices such as a digital still camera and a mobile phone with a camera function, and a recording medium such as a hard disk for recording a captured image have become widespread.

  Accordingly, a large amount of images are accumulated by the user. Thus, it is difficult to find an image desired by the user from a large number of images.

  In order to make it easier for the user to find the desired image, it is conceivable to classify the content under a predetermined condition. As a predetermined condition for classifying images, there is a technique that uses the similarity of content feature amounts (see, for example, Patent Document 1 and Patent Document 2).

  In Patent Document 1, in image search, images having a similarity with an image designated by the user are classified by clustering, and a narrow search can be performed by selecting a cluster.

  In Patent Document 2, content is hierarchically structured based on the similarity of content, thereby enabling efficient walk-through search.

Japanese Patent No. 3457617 Japanese Patent No. 3614235

  However, image classification based on the similarity of image feature amounts is not always useful for image retrieval by a user. In the image search by the user, for example, the user can easily search by specifying intuitive information as a search condition such as the meaning content of the image (the name of the object appearing in the image, the scene where the image is captured, etc.). It will be possible. However, images having similar image feature amounts are not necessarily images having the same or similar semantic content. As a result, there is a problem that an image search based on a search method that is considered easy for the user, such as designating the semantic content of an image, cannot be performed appropriately only by image classification based on the similarity of image feature amounts.

  Therefore, the present invention has been made in view of such a problem, and an image management apparatus that enables a hierarchical structure more suitable for the semantic content of an image group by appropriately selecting items to be classified by object recognition. The purpose is to provide.

  In order to solve the above-described problems, an image management apparatus according to the present invention includes an object recognition unit that classifies an image for each semantic concept item based on similarity to a general model, and similarity between image feature amounts. Clustering means for classifying images based on the above, detailed classification item selection means for selecting the semantic concept items to be classified by the object recognition means according to the diversity of each semantic concept item, and the object A hierarchical structure generating unit that integrates the classification results of the recognition unit and the clustering unit to generate a hierarchical classification structure of the image group, and the object recognition unit converts the image group into the detailed classification item selection unit. Classification is performed on the partial image group for each item of the selected semantic concept, and the clustering unit classifies each partial image group classified by the object recognition unit.

  With this configuration, in the hierarchical structure of the image group, detailed object recognition is preferentially performed for items for which various objects are photographed, thereby minimizing the number of items for object recognition and matching the contents of the image group with object recognition. Hierarchical classification can be performed on the selected items, and as a result, a hierarchical structure can be made more in line with the semantic content of the image group.

  According to the present invention, when performing hierarchical structure of image groups, it is possible to select items for object recognition according to the diversity of the image groups. As a result, the image group can be classified by object recognition with a granularity of detail suitable for each meaning, and a hierarchical structure can be formed in accordance with the semantic content of the image group. That is, it is possible to prevent the classification performance from deteriorating with an increase in the number of items for object recognition and improve the accuracy of classification by object recognition.

Diagram showing an example of a refined search Diagram showing an example of a hierarchical structure Functional block diagram of the image management apparatus in the first embodiment Functional block diagram of the object recognition unit of FIG. 3 in the first embodiment Functional block diagram of the clustering unit of FIG. 3 in the first embodiment Functional block diagram of the detailed classification item selection unit of FIG. 3 in the first embodiment 6 is a flowchart illustrating an example of the overall operation of the image management apparatus in the first embodiment. The flowchart which shows an example of the process of the detailed classification item selection part in Embodiment 1. The flowchart which shows an example of the process which classifies an image group into an outline classification item in the object recognition part in Embodiment 1. The flowchart which shows an example of the process which classifies an image group into a detailed classification item in the object recognition part in Embodiment 1. The figure which shows an example of image management information A diagram showing an example of detailed classification item candidates of the image management apparatus in a hierarchical manner Diagram showing an example of semantic hierarchy information The figure which shows an example of the summary classification result management information in Embodiment 1 The figure which shows an example of the diversity management information in Embodiment 1. The figure which shows an example of the priority management information in Embodiment 1 The figure which shows an example of the detailed classification item management information in Embodiment 1 The figure which shows an example of detailed classification result management information The figure which shows an example of the image management information according to classification item in Embodiment 1 The figure which shows an example of the clustering result management information in Embodiment 1 FIG. 6 is a diagram hierarchically showing an example of clustering result management information in the first embodiment The figure which shows an example of the hierarchical structure management information in Embodiment 1 FIG. 7 is a diagram hierarchically showing an example of hierarchical structure management information in the first embodiment The figure which shows an example of the input image group in Embodiment 1 Functional block diagram of the image management apparatus in the second embodiment Functional block diagram of the detailed classification item selection unit of FIG. 25 in the second embodiment The flowchart which shows an example of the process of the detailed classification item selection part in Embodiment 2. The figure which shows an example of metadata management information A figure showing an example of interest management information The figure which shows an example of the diversity management information in Embodiment 2. The figure which shows an example of the priority management information in Embodiment 2. The figure which shows an example of the detailed classification item management information in Embodiment 2 Functional block diagram of an image management apparatus in the third embodiment 10 is a flowchart illustrating an example of the overall operation of the image management apparatus according to the third embodiment. The figure which shows an example of the diversity management information in Embodiment 3. The figure which shows an example of the priority management information in Embodiment 3 The figure which shows an example of the image management information according to classification item in Embodiment 3 The figure which shows an example of the clustering result management information in Embodiment 3 The figure which represented hierarchically an example of the clustering result management information in Embodiment 3 Functional block diagram of an image management apparatus provided with a refinement search unit in Modification (1) Functional block diagram of the image management apparatus in Modification (2)

(Background to obtaining one embodiment of the present invention)
The inventor examined in detail the problem in image search by the user. Hereinafter, this point will be described.

  First, an image search method that is easy for the user will be described.

  When there are a large number of search targets, such as images stored by the user, searching for images desired by the user is facilitated by performing a search while gradually narrowing down images similar in meaning to the image that the user wants to search.

  An example of the narrow search screen is shown in FIG. Start with the screen 1a. Each image 1a is a representative image of a partial image group in which an image group accumulated by the user is classified based on an outline meaning. Here, the representative image is selected from images included in each partial image group. In the example of FIG. 1, all image groups are first classified according to the meaning granularity of “meal”, “dog”, and “clothes”, and the representative images in 1a are the meaning items of “meal”, “dog”, and “clothes”. The partial image group is represented. When the user selects any image, the search can be narrowed down to images classified into the respective semantic items. That is, it is possible to search for an image by narrowing the search target to a partial image group including the representative image selected by the user. For example, when the user wants to search for an image of “meal” and selects the image of 1d in 1a, the screen shifts to 1b, which is a screen for classifying “meal” in more detail. In the example of FIG. 1, each image of 1b represents a semantic item of “丼”, “ramen”, and “curry” in the meal image. When the user selects either one, the search can be narrowed down to images classified into more specific semantic items. For example, when the user wants to search for an image of curry, if the user selects the image of 1e, the screen transitions to the screen of 1c that classifies the curry in more detail. In this way, by performing a narrowed search while specifically changing the semantic items to be searched step by step, the user can easily perform a search based on the image of the image to be searched.

  In order to perform the above-described search, it is conceivable that the image group is hierarchically structured while changing the granularity of the semantic items to be classified, and the search is narrowed down by tracing the generated hierarchical structure from above.

  FIG. 2 shows an example of a hierarchical structure for realizing the narrow search as shown in FIG. In the figure, a circle filled with gray represents each image. A white circle represents a partial image group composed of lower-order images. In the state where a plurality of images (or partial image groups) are connected to the lower part of the upper partial image group, the upper partial image group is composed of a plurality of lower images (or partial image groups), which are similar concepts. Represents a thing. In this way, by making the image group into a hierarchical structure while changing the granularity of the semantic items to be classified, it becomes possible to narrow down the search by tracing the generated hierarchical structure from above.

  Here, in order to perform classification based on such semantic items, it is necessary to identify and add the meaning of an object appearing in the image to the image. At this time, since it is troublesome for the user to manually add image information such as comments to each of a large number of images, there is a need for a technique for hierarchically grouping image groups from only image information.

  However, both Patent Document 1 and Patent Document 2 cluster an image group based on the similarity of image feature amounts, and do not perform clustering in consideration of the meaning of an object appearing in the image. In particular, Patent Document 2 manages contents groups in a hierarchical structure, and the hierarchization at this time is also performed based on the similarity of image feature amounts, meaning the object in the image. Is not taken into account.

  Therefore, as a method of adding and classifying the meaning of an object appearing in an image, an object recognition method is used, and the image group is classified into partial image groups based on the meaning by comparison with a classification item model. It is conceivable to classify image groups by classifying the image groups based on their similarities using a clustering technique.

  In this case, in order to make a hierarchical structure in accordance with the semantic content of the image group, it is necessary to classify the semantic items into more detailed granularity. In order to classify into more detailed granularity semantic items, it is necessary to increase the number of classification items for object recognition. However, if the number of items for object recognition is increased, the classification performance deteriorates, so that there is a problem that it is not possible to perform a hierarchical structure in accordance with the semantic content of the image group.

  As a result of studying this problem, the present invention classifies an image group by object recognition with a granularity of detail suitable for each meaning by selecting an object recognition item according to the diversity of the image group to be classified. The present inventors have found that a hierarchical structure can be formed in accordance with the semantic content of an image group, and have reached the present invention.

(Embodiment 1)
If the granularity of the meaning of classification by object recognition is made detailed in the hierarchical structure of images taken by the user, the number of classification items increases, and the accuracy of object recognition deteriorates. It is efficient to change the meaning to be classified in detail according to the contents of the image group.

  The image management apparatus according to the first embodiment performs image recognition processing based on such an idea.

  Hereinafter, the image management apparatus according to Embodiment 1 will be described with reference to the drawings.

  FIG. 3 is a functional block diagram of the image management apparatus according to the first embodiment.

  The image management device 100 is connected to the image storage device 111 and the hierarchical structure storage device 151.

  The image storage device 111 is a device that can store a user image in association with an image ID uniquely assigned to each image.

  The hierarchical structure storage device 151 is a device that can store the hierarchical structure of the image group generated by the image management apparatus 100.

  Next, the internal configuration of the image management apparatus 100 will be described.

  As illustrated in FIG. 3, the image management apparatus 100 includes an image input unit 110, an object recognition unit 120, a clustering unit 130, a detailed classification item selection unit 140, and a hierarchical structure generation unit 150.

  The image input unit 110 acquires images and image IDs stored in the image storage device 111.

  The object recognition unit 120 performs recognition processing of the object selected by the detailed classification item selection unit 140 on the image of the image input unit 110. By the object recognition process, semantic concepts that do not depend on the user can be classified based on a general model. The recognition processing includes, for example, a classification method based on machine learning using SVM (Support Vector Machine), a detection method based on pattern matching with a dictionary prepared in advance, and the like. Examples of semantic items for classification by the object recognition unit include “meal”, “dog”, and “salmon”.

  An example of the functional configuration of the object recognition unit 120 will be described with reference to FIG. FIG. 4 is a functional block diagram of the object recognition unit of FIG.

  The object recognition unit 120 includes a model selection unit 121, an object recognition execution unit 122, a summary classification result storage unit 123, and a detailed classification result storage unit 124.

  The model selection unit 121 selects a model of an item for which semantic classification is performed by object recognition processing based on information in the diversity calculation unit 141 or the detailed classification item storage unit 146.

  The object recognition execution unit 122 performs object recognition processing based on the model selected by the model selection unit 121.

  The summary classification result storage unit 123 stores the result of classifying an image group into semantic items of coarse granularity by the object recognition execution unit 122 when the model selection unit 121 selects a model based on the information of the diversity calculation unit 141. To do.

  When the model selection unit 121 selects a model based on the information in the detailed classification item storage unit 146, the detailed classification result storage unit 124 classifies the image group into detailed granularity semantic items by the object recognition execution unit 122. Remember.

  Here, the description returns to the image management apparatus 100 in FIG.

  The clustering unit 130 classifies the partial image groups classified into the same item by the object recognition unit 120 in more detail by clustering processing. Through clustering, semantic concepts with more detailed granularity depending on the user can be classified based on the similarity between images.

  An example of the functional configuration of the clustering unit 130 will be described with reference to FIG. FIG. 5 is a functional block diagram of the clustering unit of FIG.

  The clustering unit 130 includes a clustering execution unit 131 and a clustering result storage unit 132.

  The clustering execution unit 131 classifies the input image group by clustering processing based on the similarity between images. Clustering processing includes, for example, a method of clustering by a method such as hierarchical clustering, k-means, or affinity propagation based on the similarity of image feature amounts such as color histograms and edge feature amounts. By clustering, images of the same semantic concept can be classified in more detail based on the meaning that is difficult to model in advance. For example, for a group of images classified as “Shiba-inu” by the object recognition unit, “Shiba-inu from my home” and “Shiba-in from the next house” are modeled in advance without user image input or tagging operations. However, it is difficult to classify them using object recognition methods, but by classifying them based on their similarities by clustering, the “Shiba Inu of my house” and “Shiba Inu of the next house” Can be classified in detail.

  The clustering result storage unit 132 stores the result of clustering by the clustering execution unit 131.

  Here, the description returns to the image management apparatus 100 in FIG.

  The detailed classification item selection unit 140 selects a classification item for classifying an image group with detailed semantic granularity by object recognition. Specifically, the detailed classification item selection unit 140 prioritizes items to be classified with detailed semantic granularity among the classification candidate items based on the diversity of image groups acquired by the image input unit 110. select. In the present embodiment, a case will be described in which an image group is first classified into outline classification items, and a detailed classification item is selected based on diversity for each outline classification item.

  An example of a functional configuration of the detailed classification item selection unit 140 will be described with reference to FIG. FIG. 6 is a functional block diagram of the detailed classification item selection unit of FIG.

  The detailed classification item selection unit 140 includes a diversity calculation unit 141, a priority calculation unit 142, a semantic hierarchy storage unit 143, a diversity storage unit 144, a priority storage unit 145, and a detailed classification item storage unit 146.

  The diversity calculation unit 141 calculates a diversity score that is a value obtained by quantifying diversity based on the number of classifications and changes in the subject for each summary classification item.

  The priority calculation unit 142 calculates a priority for selecting an item for object recognition for each item of the classification candidate.

  The semantic hierarchy storage unit 143 stores classification items at a plurality of semantic granularities and their hierarchical relationships.

  The diversity storage unit 144 stores the diversity score calculated by the diversity calculation unit 141 for each outline classification item.

  The priority storage unit 145 stores the priority calculated by the priority calculation unit 142 for each outline classification item.

  The detailed classification item storage unit 146 stores the detailed classification item selected based on the priority calculated by the priority calculation unit 142.

  Here, the description returns to the image management apparatus 100 in FIG.

  The hierarchical structure generation unit 150 combines the classification results in the object recognition unit 120 and the clustering unit 130, and generates a hierarchical structure that can be obtained by classifying the input image group in stages with a plurality of semantic granularities. That is, by combining the result of classifying the input image group by the object recognition unit for each detailed classification item selected by the detailed classification item selection unit and the result of classifying the partial image group for each detailed classification item by the clustering unit, Generate a hierarchical structure. The generated hierarchical structure is stored in the hierarchical structure storage device 151.

  An overview of the overall operation performed by the image management apparatus 100 will be described using the flowchart of FIG. Further, S102 in FIG. 7 will be described as appropriate with reference to FIG. 8, details of S203 in FIG. 8 will be described with reference to FIG. 9, and details of S103 in FIG.

  The image input unit 110 acquires image management information from the image storage device 111 (S101). FIG. 11 shows an example of image management information. In the example of FIG. 11, images are managed in association with image IDs in the JPEG (Joint Photographic Experts Group) format. Note that the image information storage method is not limited to the method described above. Instead of storing an image in the JPEG format, the image may be stored in a PPM (Portable Pixmap Format) format or a PNG (Portable Network Graphics) format, or an image path in the image storage device 111 may be stored.

  Next, the detailed classification item selection unit 140 selects items for performing object recognition in detail (S102).

  Here, before explaining the details of the processing of S102, the semantic hierarchy storage unit will be explained. The semantic hierarchy storage unit stores a plurality of classification item candidates of semantic granularity classified by the image management apparatus 100 and the hierarchical relationship of each item. Here, the candidate for the classification item is a classification item with a common semantic concept regardless of the user, and includes, for example, classification items such as “meal”, “dog”, and “salmon”. FIG. 13 shows an example of semantic memory management information in the case where candidates for classification items of a plurality of semantic granularities classified by the image management apparatus 100 have a hierarchical structure as shown in FIG. In the semantic hierarchy information, each of the hierarchy, the upper item, and the lower item represents an item of the upper concept of the meaning, and an item of the lower concept of the meaning, in which hierarchy the item is located from the top. For example, in FIG. 12, there are “meals”, “dogs”, and “clothes” as candidates for general classification items of “image group”, and subordinate concepts of “meal” as candidates for detailed classification items related to “meals”. This indicates that there is a certain “sashimi”, “ramen” and “curry”. In the second line of FIG. 13, “meal” is in the first layer in the hierarchical structure of classification item candidates, and the subordinate concept items of “meal” are “丼”, “ramen”, and “curry”. The fifth line in FIG. 13 indicates that “丼” is in the second hierarchy in the hierarchical structure of the classification item candidates, and the item of the higher concept of “丼” is “meal”. Yes.

  An example of the process of S102 will be described using the flowchart of FIG.

  The detailed classification item selection unit 140 first acquires a summary classification item from the semantic hierarchy storage unit in order to classify the images included in the image management information into items of rough semantic granularity (S201). Here, the summary classification item is a classification item having a rough semantic granularity among the classification item candidates. For example, in the semantic hierarchy information illustrated in FIG. 13, an item having a hierarchy of 1 is acquired as the summary classification item. In the example of FIG. 13, “meal”, “dog”, and “clothes” are acquired as summary classification items.

  Next, the detailed classification item selection unit 140 classifies the image group into the summary classification items acquired in S201 by the object recognition unit 120 (S202).

  An example of the process of S202 will be described using the flowchart of FIG.

  The object recognition unit 120 first acquires a model corresponding to the summary classification item in the model selection unit 121 (S301). This is realized by selecting a classification model corresponding to each of the summary classification items acquired in S201 from a model group created in advance. For example, a model generated by SVM machine learning using a general image in advance is used.

  Next, in the object recognition execution unit 122, the object recognition unit 120 classifies the image group input from the image input unit 110 by object recognition (S302). For example, when a classification method based on machine learning by SVM is used, a likelihood that is a value representing the likelihood of an object to be classified is calculated by using the image feature amount, the SVM classifier, and the model acquired in S301, and the likelihood is calculated. Each image is classified by comparing with each other.

  The classification result is stored in the summary classification result storage unit 123 as summary classification result management information. An example of the summary classification result management information is shown in FIG. In the example of FIG. 14, the summary classification items of each image are stored in association with the image ID. In the example of FIG. 14, the result of classifying an image with an image ID of 0 with rough semantic granularity by object recognition indicates “meal”. Thus, the process of S202 in FIG. 8 ends.

  Returning to FIG. 8, next, the detailed classification item selection unit 140 selects one of the summary classification items acquired in S201 (S203).

  Next, the detailed classification item selection unit 140 calculates diversity for the summary classification item selected in S203 (S204). Here, diversity is an index representing how many different meaning images are included in a partial image group common to a certain semantic item in the image group. It is determined that the greater the diversity, the more need to be classified into items of detailed semantic granularity by object recognition. The diversity is calculated from, for example, an image number score based on the number of images and a subject change score based on the magnitude of the change in the image feature amount of the subject. This is based on the idea that the greater the number of images and the greater the change in the subject, the more diverse the items. The image number score of each item is, for example, the ratio of the number of partial image groups of each item to the total number of image groups. Further, the subject change score of the partial image group of each item is calculated as, for example, the average of the variances of the color histogram and the edge histogram of the images included in the partial image group. For example, the diversity score is calculated as the sum of the image number score and the subject change score. The calculated diversity score is stored in the diversity storage unit 144 as diversity management information. An example of diversity management information is shown in FIG. The second line in FIG. 15 indicates that the “meal” image count score is 0.5, the subject change score is 0.8, and the diversity score is 1.3, which is the sum of both.

  Note that the calculation of the number of images in the image number score may be calculated with a plurality of images taken at time intervals within a predetermined time as one sheet. For example, assuming that the predetermined time is 5 seconds, if there are images with a difference in shooting time with respect to a certain image of 5 seconds or less, the number of images may be calculated with those images as one. As a result, images taken by continuous shooting can be counted as one, and when the same subject is shot by continuous shooting, the number of images increases, thereby preventing an increase in diversity. it can.

  Note that the calculation of the image feature amount in the subject change score may be limited to the foreground portion of the image, not the entire image. At this time, the foreground part of the image may be performed by determining, for example, for each pixel whether the value of the Saliency by the calculation of the Saliency map of the image is equal to or greater than a predetermined value. May be included by determining whether or not the depth value is greater than or equal to a predetermined value. Further, the diversity score may be calculated as a weighted sum of the image number score and the subject change score, instead of calculating the sum of the image number score and the subject change score. Further, the diversity score may be calculated from only the image number score or only the subject change score. Thereby, it is possible to calculate the diversity of only the subject in the foreground part, excluding the influence due to the change in the background part.

  After the process of S204, it is determined whether or not diversity has been calculated for all summary category items (S205). If there are still items for which diversity has not been calculated (No in S205), the process returns to S203. If diversity is calculated for all items in S205 (Yes in S205), then the detailed classification item selection unit 140 selects one of the items stored in the semantic hierarchy information 1320 in the figure (S206). ).

  Next, the detailed classification item selection unit 140 calculates the priority for the item selected in S206 (S207). The priority is a value to which a high value is assigned to the item to be classified in detail, and is determined based on the diversity for each summary classification item according to the image group. The priority is, for example, the diversity value of the upper item. This is based on the idea that items with higher diversity should be preferentially classified in detail. It should be noted that the priority of an item having a hierarchy of 1 may be a large value (for example, 10,000). This is based on the idea that classification with a general semantic granularity should be given priority for any image group. The calculated priority is stored in the priority storage unit 145 as priority management information. An example of the priority management information is shown in FIG. In this example, for example, it is indicated that the priority of the bowl is 1.3 (the same value as the meal diversity score) and the priority of the meal is 10,000 (because the meal hierarchy is 1). Yes.

  After the process of S207, it is determined whether the priority has been calculated for all items (S208). If there are still items for which priority should be calculated (No in S208), the process returns to S206. If priority is calculated for all items in S208 (Yes in S208), then the detailed classification item selection unit 140 selects a detailed classification item based on the priority calculated in S207 (S209). For the detailed classification item, for example, a predetermined number of items with high priority calculated in S207 are selected. Here, when a certain item and all its lower items are selected as the detailed classification items, the higher items may be excluded from the detailed classification items. This is to prevent the image of the lower item from being classified as overlapping with the upper item. For example, when “meal”, “丼”, “ramen”, and “curry” are selected as the detailed classification items, “meal” may be excluded from the detailed classification items. Thereby, it is possible to prevent the image of the cocoon from being classified into both “両 方” and “meal”. The selection result is stored in the detailed classification item storage unit 146 as detailed classification item management information. An example of detailed classification item management information is shown in FIG. Thus, the process of S102 in FIG. 7 ends.

  Returning to FIG. 7, next, the object recognition unit 120 classifies each image stored in the image storage device 111 into items stored in the detailed classification item storage unit 146 by object recognition (S103). A specific example of S103 will be described using the flowchart of FIG. First, in S401, a model corresponding to the detailed classification item selected by the detailed classification item selection unit 140 is acquired. This is realized by selecting a classification model corresponding to an item stored in the detailed classification item storage unit 146 from a model group created in advance. As the model, for example, a model learned in advance by SVM machine learning using a general image is used.

  Next, in the object recognition execution unit 122, the object recognition unit 120 classifies the image group input from the image input unit 110 by object recognition (S402). For example, when using a classification method based on machine learning based on SVM, a likelihood that is a value representing the probability of an object to be classified is calculated by using the image feature amount, the SVM classifier, and the model acquired in S401, and the likelihood is calculated. Each image is classified into each item by comparing with each other. The classification result is stored in the detailed classification result storage unit 124 as detailed classification result management information. An example of the detailed classification result management information is shown in FIG. In the example of FIG. 18, the result of classifying an image with an image ID of 0 into detailed semantic granularity by object recognition is “丼”. Comparing the contents of FIG. 18 with the summary classification result management information in FIG. 14, the images with the image IDs 0 and 1 classified as “meal” in FIG. 14 and the detailed classification result management information in FIG. It is categorized in more detail as “丼” and “ramen”. Subsequently, based on the detailed classification item management information of FIG. 17 and the detailed classification result management information of FIG. 18, the image ID classified into each detailed classification item is created as image management information by classification item as shown in FIG. The detailed classification result storage unit 124 stores the result. For example, in the example of FIG. 19, the image IDs of the images classified as “丼” are 0, 7, 19, 27, 29, and 32. Above, the process of S103 in FIG. 7 is complete | finished.

  Returning to FIG. 7, next, the image management apparatus 100 selects one of the detailed classification items stored in the detailed classification item storage unit 146 (S104).

  Next, the image management apparatus 100 classifies the image group of each item classified in S103 by clustering (S105). The image group to be clustered is an image group classified into the same item by the detailed classification result management information. That is, the image ID corresponding to the item selected in S104 in the image management information classified by category in the detailed classification result storage unit 124 is set as the clustering target image group. The clustering method is performed by, for example, a hierarchical clustering method based on similarity of image feature amounts. As the image feature amount, for example, a color histogram feature amount or an edge feature amount is used. The clustering method may be a method other than the clustering method based on hierarchical clustering. For example, a k-means method or an Affinity Propagation method may be used. In addition, clusters obtained by clustering once may be classified hierarchically by clustering by clustering again. The clustering result is stored in the clustering result storage unit 132 as clustering result management information. An example of the clustering result management information is shown in FIG. Here, the second line in FIG. 20 names the cluster formed by clustering the parent item of “Dog” as “Dog A” and “Dog B”, and the third line shows the cluster of “Dog A”. , The image ID is 2, 6, 11, 14, 15, 16, 17, 23, 24, 25, 28, 37, 38, 39. The fourth line is the cluster of “Dog B”. It shows that the image has IDs of 34 and 35.

  After the process of S105, it is determined whether or not the clustering process has been completed for all items (S106). If there are still items to be clustered (No in S106), the process returns to S104. When the clustering process is completed for all items in S106 (Yes in S106), the hierarchical structure generation unit 150 combines the hierarchical structures obtained as a result of S103 and S105, and classifies the entire image group hierarchically. A hierarchical structure is generated and stored in the hierarchical storage device 151 as hierarchical structure management information (S107). An example of the hierarchical structure management information is shown in FIG. Here, in FIG. 22, the “image group” refers to the entire input image group, and in the second line of FIG. 22, the child of the “image group” is “meal, dog, cat” As a first stage of the hierarchical classification, it represents that all image groups are roughly classified into “meal”, “dog”, and “cat”. The third line in FIG. 22 indicates that the image of “meal” is classified into items “丼”, “ramen”, and “curry”. The 10th line in FIG. 22 names the clusters obtained by clustering the images of “丼” as “丼 A” and “丼 B”, and the 11th row shows that the cluster of “丼 A” has an image ID of 0, It is composed of 7 and 27 images, and the 12th row indicates that the cluster “丼 B” is composed of images with image IDs 19, 29, and 32.

  The configuration of the image management apparatus according to the first embodiment and the processing performed by the image management apparatus have been described above. A specific example of the operation of the image management apparatus according to the present embodiment will be described below.

  Here, with respect to the image group shown in FIG. 24, the flow from when the image management apparatus 100 selects a detailed classification item to hierarchically structure the image group by object recognition and clustering will be described with reference to the flowchart of FIG. This will be specifically described. In this specific example, a specific example of processing for a total of 40 image groups, part of which is shown in FIG. 24, is shown. Each is assigned an image ID from 0 to 39, and FIG. 24 shows four images from 0 to 3 in the image group. In the image storage device 111, the input image shown in FIG. 24 is managed as shown in FIG. 11, for example.

  In this specific example, it is assumed that the semantic item candidates classified by the image management apparatus 100 have a hierarchical structure as shown in FIG. The meaning storage management information in this case is as shown in FIG. 13, for example. In FIG. 13, items to be classified by the image management apparatus 100 include “meal”, “dog”, and “clothes” as summary category items, and “丼”, “ramen”, and “curry” as candidates for detailed category items for meals. It shows that “Shiba Inu”, “Chihuahua”, “Dalmatian” are candidates for detailed classification items for dogs, and “T-shirts”, “suits”, and “kimono” are candidates for detailed classification items for clothes.

  First, the image input unit 110 acquires an image and an image ID by acquiring image management information from the image storage device 111 (S101).

  Next, the detailed classification item selection unit 140 selects an item for performing object recognition (S102).

  A specific example of the processing of S102 will be described using the flowchart of FIG.

  The detailed classification item selection unit 140 first acquires a summary classification item from the semantic hierarchy storage unit (S201). For example, in the case of acquiring semantic hierarchy information shown in FIG. 13 having a hierarchy of 1, “meal”, “dog”, and “clothes” are selected.

  Next, the detailed classification item selection unit 140 classifies the image group into “meal”, “dog”, and “clothes” (S202).

  An example of the process of S202 will be described using the flowchart of FIG.

  First, the object recognizing unit 120 acquires a classification model of “meal”, “dog”, and “clothes” in the model selection unit 121 (S301). For example, a “meal” classification model can be obtained by machine learning using SVM based on general meal images collected in advance.

  Next, in the object recognition execution unit 122, the object recognition unit 120 classifies each image input from the image input unit 110 by object recognition (S302). For example, the “meal”, “dog”, and “clothing” models selected in S301 and the SVM classifier are used for classification.

  The classification result is stored in the summary classification result storage unit 123 as summary classification result management information. In the example of FIG. 14, the images with image IDs 0, 1, 2, and 3 are roughly classified as “meal”, “meal”, “dog”, and “clothes”, respectively. Here, it is assumed that all 40 images are classified into 20 for “meal”, 16 for “dog”, and 4 for “clothes”. Thus, the process of S202 in FIG. 8 ends.

  Returning to FIG. 8, next, the detailed classification item selection unit 140 selects one of the summary classification items “meal”, “dog”, and “clothes” acquired in S201 (S203). Hereinafter, in S203, the case where “meal” is selected will be described.

  Next, the detailed classification item selection unit 140 calculates the diversity of “meals” (S204). Here, a case where diversity is calculated as the sum of the image number score and the subject change score as described above will be described. First, the image score of “meal” and the ratio of the number of images classified as “meal” to the total number of images, that is, 20 ÷ 40 = 0.5. The subject change score is calculated as the average of the color histogram and edge feature amount variance of 20 images classified as “meal”. For example, if the subject change score is 0.8, the diversity score is the sum of the image number score and the subject change score, that is, 0.5 + 0.8 = 1.3. The values of the image number score, the subject change score, and the diversity score are stored as diversity management information in association with items as shown in the second row of FIG. Diversity scores are similarly calculated for “dogs” and “clothes”, and are calculated as diversity management information as shown in the second and third lines of FIG.

  After calculating the diversity of “meals”, “dogs”, and “clothes” (Yes in S205), the detailed classification item selection unit 140 selects items “meals”, “dogs”, “clothes”, “丼”, “ramen”, “ One of curry, shiba inu, chihuahua, dalmatian, t-shirt, suit, and kimono is selected (S206). Below, the example of each process at the time of selecting "meal" and "ramen" is shown. Further, here, a case will be described in which the priority is the diversity value of the upper item as the priority as described above (however, when the hierarchy is 1, the priority is 10,000).

  When “meal” is selected in S206, the priority of “meal” is calculated in S207. Based on the semantic hierarchy information in FIG. 13, “meal” is identified as having a hierarchy of 1. Therefore, the priority of “meal” is 10,000. This is stored in the first line of the priority management information in FIG.

  When “ramen” is selected in S206, the priority of “ramen” is calculated in S207. Here, as described above, a case will be described in which the priority is set as the diversity value of the upper item as described above. Based on the semantic hierarchy information in FIG. 13, “meal” is specified as the upper item of “ramen”. Based on the diversity management information in FIG. 15, 1.3 is specified as the diversity score of “meal”. This 1.3 is stored as the priority of “ramen” in the sixth row of the priority management information in FIG. For the remaining 10 items, the priority is calculated in the same manner, and is calculated as the priority management information except for the first and sixth lines in FIG.

  After calculating the priority for all items (Yes in S208), the detailed classification item selection unit 140 selects a detailed classification item based on the priority calculated in S207 (S209). Here, as described above, the detailed classification item is a case where a predetermined number of items with high priority are selected, and a case where the predetermined number is 5 will be described. First, in the priority management information of FIG. 16, the items are rearranged in the descending order of priority. Next, among the rearranged items, a predetermined number + 1 = 6 with high priority is selected. Here, the reason why 1 is added to the predetermined number is that when all the lower items are selected as the detailed classification items, the upper items are excluded from the detailed classification items. At this time, “meal”, “dog”, “clothes”, “candy”, “ramen”, and “curry” are selected. Of these, from the second row and the fifth, sixth, seventh rows in FIG. 13, it can be seen that all sub-items “丼”, “ramen” and “curry” of “meal” are selected as detailed classification items. “Meals” are excluded from the detailed classification items. Finally, the detailed classification items are five items of “dog”, “clothes”, “candy”, “ramen”, and “curry”, and are stored as detailed classification item management information as shown in FIG. Thus, the process of S102 in FIG. 7 ends.

  Returning to FIG. 7, next, the object recognition unit 120 classifies the image group into “dog”, “clothes”, “candy”, “ramen”, and “curry” (S103). A specific example of the process of S103 will be described using the flowchart of FIG.

  First, the object recognizing unit 120 obtains a classification model of “dog”, “clothes”, “candy”, “ramen”, and “curry” in the model selection unit 121 (S401).

  Next, in the object recognition execution unit 122, the object recognition unit 120 converts each of the images input from the image input unit 110 into the “dog”, “clothes”, “candy”, “ramen”, “curry” model and SVM selected in S401. Classification is performed by the classifier (S402).

  The classification result is stored in the detailed classification result storage unit 124 as detailed classification result management information as shown in FIGS. In the example of FIG. 18, the images with image IDs 0, 1, 2, and 3 are classified as “丼”, “ramen”, “dog”, and “clothes”, respectively. Above, the process of S103 in FIG. 7 is complete | finished.

  Returning to FIG. 7, next, the image management apparatus 100 selects one of the detailed classification items “dog”, “clothes”, “candy”, “ramen”, and “curry” stored in the detailed classification item storage unit 146 (S104). Hereinafter, a case where “丼” is selected in S104 will be described as an example.

  Next, the image management apparatus 100 classifies the “丼” image by clustering (S105). In the clustering target image group, the images with the image IDs “0, 7, 19, 27, 29, 32” are identified from the fourth row in which the first column in FIG. 19 is “丼”. These images are subjected to hierarchical clustering based on, for example, the similarity of the color histogram feature quantity, and the “丼” images are clustered in more detail.

  The clustering results shown in the hierarchical structure of FIG. 21 are stored in the clustering result storage unit 132 as clustering result management information as shown in FIG. Here, in FIG. 21, circles represent clusters, and the numbers in the squares represent the image IDs of images included in the upper cluster. In this example, the hierarchical relationship is expressed by describing the parent-child relationship in the hierarchical structure with respect to each cluster and the image ID. For example, in the tree structure 21a representing the clustering result management information, the “丼” cluster is classified into two clusters by clustering (named “丼 A” and “丼 B”, respectively), and the images included in each of them. The image IDs of “0, 7, 72” and “19, 29, 32” respectively. The clustering result management information corresponding to this can be expressed as, for example, rows “丼”, “丼 A”, and “丼 B” in FIG. There are two clusters “丼 A” and “丼 B” below “丼”, and “、 A” includes images with image IDs “0, 7, 27”. "Indicates that an image having an image ID" 19, 29, 32 "is included. The remaining “dogs”, “clothes”, “ramen”, and “curry” are clustered in the same manner and stored as clustering result management information as shown in FIG.

  When the clustering process is completed for all items (Yes in S106), the hierarchical structure generation unit 150 combines the hierarchical structures obtained as a result of S103 and S105, and generates the hierarchical structure of the entire image group illustrated in FIG. . Then, it is stored in the hierarchical structure storage device 151 as the hierarchical structure management information of FIG. 22 (S107). Here, in FIG. 23, a circle represents a cluster, and a number in a square represents an image ID of an image included in a higher cluster. In FIG. 22, the hierarchical relationship is represented by describing the parent-child relationship in the hierarchical structure with respect to each cluster and the image ID.

  As described above, the image management apparatus 100 according to the first embodiment can select an object recognition item according to the diversity of the image group when performing the hierarchical structure of the image group. This classifies the image group by object recognition with semantic granularity that matches the diversity of each item of the image group while preventing the deterioration of the classification performance due to the increase in the number of items of object recognition, and the semantic content of the image group It is possible to make a hierarchical structure according to the above.

  Note that, based on the diversity of the detailed classification items selected by the detailed classification item selection unit 140, the semantic item of the lower concept may be used as the detailed classification item. Specifically, for example, the detailed classification item selection unit 140 calculates diversity for a detailed classification item with high priority, and classifies items with high priority including semantic items of subordinate concepts of items with high diversity. It is good as an item. For example, when the priority of “ramen” is high and the diversity of “ramen” is high, the meaning items of the subordinate concepts of “ramen” are “soy sauce ramen”, “miso ramen”, and “pork bone ramen” High priority may be set, and “soy sauce ramen”, “miso ramen”, and “pork bone ramen” may be set as detailed classification items.

  As a result, it is possible to select the detailed classification items to be classified by object recognition from the candidates for the detailed classification items having three or more levels of semantic granularity, giving priority to them while taking diversity into consideration.

  Further, the type of feature amount used in the clustering execution unit 131 may be selected according to the semantic item of the clustering target image group. Specifically, a type suitable for classifying the semantic items of the clustering target partial image group given by the object recognition unit 120 may be selected as the type of feature amount. For example, when the partial image group classified as “meal” by the object recognition unit 120 is classified by clustering, it is easy to understand if the partial image group is classified based on the similarity of the overall shape. When the partial image group classified as “clothes” by the object recognition unit 120 is classified by clustering, it is easy to understand if the partial image group is classified based on the similarity of colors. Therefore, clustering may be performed based on the similarity of color feature amounts.

  Thereby, easy-to-understand classification according to the semantic item contents is possible.

  Further, the clustering unit 130 may classify the partial image group obtained by one clustering again into a plurality of hierarchies, thereby classifying the hierarchies into a plurality of hierarchies. Furthermore, the type of feature amount in each clustering may be changed. For example, when classifying the partial image group classified as “clothes” by the object recognition unit 120 by clustering, first, the classification is performed based on the overall color similarity, and then the classification is performed based on the difference in detailed pattern. The partial image group classified as “clothes” is classified by clustering based on the similarity of the color feature values, and each classified image is clustered based on the similarity of the shape feature values, so that it can be classified in two levels. good.

  As a result, even if the number of input images is large, the number of hierarchical levels is increased, so that the number of selection options is reduced in the narrow search, and easy-to-understand classification according to the semantic items is possible.

(Embodiment 2)
In the first embodiment, the method for calculating the priority only from the diversity has been described. However, the priority may be calculated by adding the interest information of the user to the diversity. If the item of the object in which the user is interested is known, it is considered that the hierarchical classification according to the contents of the image group can be realized by classifying the item in detail by the object recognition unit.

  The image management apparatus according to the second embodiment performs a hierarchical structure of image groups based on such an idea.

  Hereinafter, the image management apparatus according to the second embodiment will be described.

  Here, the priority of the item is set by adding the interest information of the user that can be understood from the metadata of the comment information posted by the user to the social networking service to the diversity calculated by the detailed classification item selection unit 140 An embodiment for selecting an item for object recognition will be described.

  FIG. 25 is a functional block diagram of the image management apparatus 100a according to the second embodiment. The configuration of the image management apparatus 100a is different from that of the image management apparatus 100 in the internal configuration of the detailed classification item selection unit. In addition, a metadata storage device 161 is connected to the detailed classification item selection unit. In the following, the detailed classification item selection unit 140a and the metadata storage device 161 having different configurations from those of the first embodiment will be described.

  In addition to the diversity calculated by the detailed classification item selection unit in the first embodiment, the detailed classification item selection unit 140a in the present embodiment, from the degree of interest calculated from the metadata acquired from the metadata storage device 161, The priority of each item is calculated and a classification item is selected.

  An example of the functional configuration of the detailed classification item selection unit 140a will be described with reference to FIG. FIG. 26 is a functional block diagram of the detailed classification item selection unit 140a in the image management apparatus of FIG.

  The detailed classification item selection unit 140a includes a diversity calculation unit 141, a priority calculation unit 142a, a semantic hierarchy storage unit 143, a diversity storage unit 144, a priority storage unit 145, a detailed classification item storage unit 146, and an interest information extraction unit 147. And an interest degree storage unit 148.

  Since the operations of the diversity calculation unit 141, the semantic hierarchy storage unit 143, the diversity storage unit 144, the priority storage unit 145, and the detailed classification item storage unit 146 are the same as the operations in the first embodiment, the description thereof is omitted. .

  The priority calculation unit 142a calculates the priority of each classification item from the interest information of the user in addition to the diversity.

  The interest information extraction unit 147 extracts information on an object that is of interest to the user from the metadata information.

  The interest degree storage unit 148 stores user interest information appearing in the metadata information for each classification item candidate.

  Here, the description returns to the image management apparatus 100a in FIG.

  The metadata storage device 161 stores metadata management information including information related to the user's interest. The metadata includes, for example, comment information posted by the user to the social networking service, comment information given by the user to the photo, location information of the place where the photo was taken, information such as the user's age and sex.

  Next, the operation of the image management apparatus in this embodiment will be described. The overall operation of the image management apparatus in the present embodiment is the same as in FIG. However, as described above, the operation of the detailed classification item selection unit 140 is different from that of the first embodiment, and the specific processing of S102 in FIG. Hereinafter, this point will be described.

  FIG. 27 is a flowchart showing an example of processing of the detailed classification item selection unit 140 in the present embodiment.

  In the flowchart of FIG. 27, the process of S210 is added compared to the flowchart of FIG. The operations up to S205 are the same as those in the first embodiment. In the following, the processing after S210 in FIG. 27, which is different from the first embodiment, will be described.

  After the processing up to S205, the priority calculation unit 142a calculates the interest level of the classification item candidate from the metadata in the interest information extraction unit 147 (S210). The degree of interest is assigned a high value to an item that is determined to be more interested by the user based on information obtained from the metadata. For example, a noun included in metadata is extracted as the degree of interest, and a high value is assigned to a large number of items in which the extracted noun matches the name of the classification item candidate. In this case, for example, the ratio of the number of nouns that match the name of the classification item candidate included in the metadata is set as the interest level of each item. The metadata is stored in the metadata storage device 161 as metadata management information. An example of metadata management information is shown in FIG. In the example of FIG. 28, the content posted to the social networking service of the user “I ate ramen!” Is stored as metadata with a metadata ID of 0. For example, nouns are extracted by performing morphological analysis on metadata. For example, the number of occurrences of the name of the classification item candidate included in the metadata is stored in the interest degree storage unit 148 as appearance number management information (second column in FIG. 29). Calculated as a ratio to the number of occurrences of all items. The calculated interest level is stored in the interest level storage unit 148 as shown in the third column of FIG. In the example of the interest level management information in FIG. 29, the number of occurrences of “ramen” is 7 and the interest level is 0.7.

  Next, the priority calculation part 142a selects one from the items memorize | stored in the semantic hierarchy information of FIG. 13 (S206).

  Next, the priority calculation part 142a calculates the priority which considered the user's interest level in diversity (S207). In the present embodiment, the priority calculated from diversity as in the first embodiment is referred to as diversity priority, and the priority in which the user's interest is added to diversity is referred to as the overall priority. . First, diversity priority is calculated by the same method as the priority calculation in the first embodiment. Specifically, the priority calculated by the processing up to S207 in FIG. 8 is stored in the priority storage unit 145 as the diversity priority of the priority management information as shown in the second column of FIG. Next, the priority calculation unit 142a stores the interest calculated in S210 as an interest priority for each corresponding item in the priority management information. For example, the priority management information is stored in the third column in FIG. 31 in association with each item. Next, the priority calculation part 142a calculates the total priority which considered the diversity priority and the interest priority. The total priority is calculated as, for example, the sum of diversity priority and interest priority. The calculated total priority is stored as the total priority of the priority management information as shown in the fourth column of FIG. Note that the total priority calculated by the priority calculation unit 142a may be calculated not by the sum of the diversity priority and the interest priority but by the sum of the diversity priority and the interest priority. .

  After the process of S207, it is determined whether or not the priority has been calculated for all items (S208). If there are still items for which priority should be calculated (No in S208), the process returns to S206. If priority is calculated for all items in S208 (Yes in S208), then the detailed classification item selection unit 140 selects a detailed classification item based on the priority calculated in S207 (S209). Here, if the upper item is a common item and both the item selected as the detailed category item and the item not selected exist, add “Other item” to the detail category item, and the upper item will be the detail category If the item is selected, the upper item may be excluded from the detailed classification item. At this time, in the detailed object recognition in S402, the item selected as the detailed classification item and the model of the upper item are selected in S401, and the likelihood of each of the lower item and the upper item is calculated by SVM. If the likelihood is high, the likelihood of all the lower items is low, but if the likelihood of the upper items is high, the items may be classified as “higher items and others”. This is because any lower-level item image is classified as either “higher level item or the like” or lower level item. For example, if “meal” and “ramen” are selected as detailed category items, and “丼” and “curry” are not selected as detailed category items, “meal and other” is added to the detailed category item and “meal” is selected. By excluding, the detailed classification items related to meals are set as “ramen” and “meal and others”. At this time, the model of “ramen” or “meal” is selected in S401. As a result, images with ramen are classified as “ramen” because the likelihood of “ramen” and “meal” are both high, and the likelihood of “ramen” with respect to images with wrinkles. Although it is low, the likelihood of “meal” is high, so it can be classified as “meal and others”.

  Although the configuration of the image management apparatus according to the second embodiment and the processing performed by the image management apparatus have been described above, a specific example of the operation of the image management apparatus according to the present embodiment will be described below.

  Here, the flow until the image management apparatus 100a selects the detailed classification item for the image group partially shown in FIG. 24 will be specifically described with reference to the flowcharts of FIGS. This specific example shows a specific example of processing for a total of 40 image groups whose contents are different from the specific example in the first embodiment. Each image ID is assigned from 0 to 39, and FIG. 24 shows four images with image IDs 0 to 3 in the image group. In the image storage device 111, the input image shown in FIG. 24 is managed as shown in FIG. 11, for example.

  In this specific example, it is assumed that the classification item candidates of the image management apparatus 100a have a hierarchical structure as shown in FIG.

  First, the image input unit 110 acquires an image and an image ID by acquiring image management information from the image storage device 111 (S101).

  Next, the detailed classification item selection unit 140a selects an item for performing object recognition (S102).

  A specific example of the process of S102 will be described using the flowchart of FIG.

  The detailed classification item selection unit 140a first acquires a summary classification item from the semantic hierarchy storage unit (S201). For example, in the case of acquiring semantic hierarchy information shown in FIG. 13 having a hierarchy of 1, “meal”, “dog”, and “clothes” are selected.

  Next, the detailed classification item selection unit 140a classifies the image group into “meal”, “dog”, and “clothes” (S202).

  An example of the process of S202 will be described using the flowchart of FIG.

  First, the object recognizing unit 120 acquires a classification model of “meal”, “dog”, and “clothes” in the model selection unit 121 (S301). For example, the “meal” classification model is obtained by collecting a large number of images of general meals in advance and performing machine learning using SVM.

  Next, in the object recognition execution unit 122, the object recognition unit 120 classifies each image input from the image input unit 110 by object recognition (S302). For example, the “meal”, “dog”, and “clothing” models selected in S301 and the SVM classifier are used for classification.

  The classification result is stored in the summary classification result storage unit 123 as summary classification result management information. In the example of FIG. 14, the images with image IDs 0, 1, 2, and 3 are roughly classified as “meal”, “meal”, “dog”, and “clothes”, respectively. Here, it is assumed that all 40 images are classified into 12 items for “meal”, 24 items for “dog”, and 4 items for “clothes”. This completes the process of S202 in FIG.

  Returning to FIG. 27, next, the detailed classification item selection unit 140a selects one of the summary classification items “meal”, “dog”, and “clothes” acquired in S201 (S203). Hereinafter, in S203, the case where “meal” is selected will be described.

  Next, the detailed classification item selection unit 140a calculates the diversity of “meals” (S204). Here, a case where diversity is calculated as the sum of the image number score and the subject change score as described above will be described. First, the image score of “meal” and the ratio of the number of images classified as “meal” to the total number of images, that is, 12 ÷ 40 = 0.3. The subject change score is calculated as an average of color histograms and edge feature amount variances of 12 images classified as “meals”. For example, if the subject change score is 0.2, the diversity score is the sum of the image number score and the subject change score, that is, 0.3 + 0.2 = 0.5. Image number score, subject change score, and diversity score values are stored as diversity management information as shown in the second row of FIG. Diversity scores are similarly calculated for “dogs” and “clothes”, and are calculated as diversity management information as shown in lines 3 and 4 of FIG.

  After calculating the diversity of “meal”, “dog”, and “clothing” (Yes in S205), the interest information extraction unit 147 calculates the degree of interest for each item of the classification candidate (S210). Assume that the metadata shown in FIG. 28 is given as comment information posted by the user to the social networking service. The detailed classification item selection unit 140a first performs morphological analysis on the metadata with metadata ID = 0 stored in the metadata management information, and extracts “ramen” as a noun that matches the classification item name. Therefore, the number of occurrences of “ramen” in the interest management information in FIG. 29 is increased by one. The above item name extraction operation for metadata is performed on all metadata, and the number of occurrences of the word of each item is stored in the interest level management information as shown in the second column of FIG. At this time, the total number of appearances in the item name metadata is 7 + 1 + 2 = 10, and the degree of interest is calculated by dividing the number of appearances of each item by 10 in the interest degree management information. Stored as in the row. For example, the degree of interest of “ramen” is 7 ÷ 10 = 0.7, and this value is stored as interest degree management information in the third column of the sixth row in FIG.

  Next, the detailed classification item selection unit 140a selects the classification item candidates “meal”, “dog”, “clothes”, “ramen”, “curry”, “shiba inu”, “chihuahua”, “Dalmatian”, “T-shirt”, “suit”, “kimono”. "Is selected from" (S206). Below, the example of the process of each S207 at the time of selecting "meal" and "ramen" is shown. Further, here, as described above, a case will be described in which the priority is set to the diversity value of the upper item (however, when the hierarchy is 1, the priority is set to 10,000).

  When “meal” is selected in S206, the priority calculation unit 142a first calculates the diversity priority of “meal”. Based on the semantic hierarchy information in FIG. 13, “meal” is identified as having a hierarchy of 1. Therefore, the diversity priority of “meal” is 10,000. This is stored in the second column of the second row of the priority management information in FIG. Next, the priority calculation unit 142a calculates the interest priority of “meal”. The interest priority of “meal” becomes 0 by referring to the “interest level” column in the “meal” row of the interest level management information in FIG. This is stored in the third column of the “meal” row of the priority management information of FIG. Next, the priority calculation unit 142a calculates an overall priority by adding the diversity priority and the interest priority. The overall priority is 10000 + 0 = 10000. This is stored in the fourth column of the second row of the priority management information in FIG.

  When “ramen” is selected in S206, the priority of “ramen” is calculated. Here, as described above, a case will be described in which the priority is set as the diversity value of the upper item as described above. The priority calculation unit 142a first calculates the diversity priority of “meal”. Based on the semantic hierarchy information in FIG. 13, “meal” is specified as the upper item of “ramen”. Based on the diversity management information of FIG. 30, 0.5 is specified as the diversity score of “meal”. This 0.5 is stored in the second column of the sixth row of the priority management information of FIG. 31 as the diversity priority of “ramen”. Next, the priority calculation unit 142a calculates the interest priority of “ramen”. The interest priority of “ramen” becomes 0.7 by referring to “interest” in the row of “ramen” in the interest level management information in FIG. This is stored in the third column of the sixth row of the priority management information in FIG. Next, the priority calculation unit 142a calculates an overall priority by adding the diversity priority and the interest priority. The overall priority is 0.5 + 0.7 = 1.2. This is stored in the fourth column of the sixth row of the priority management information in FIG.

  After calculating the priority for all items (Yes in S208), the detailed classification item selection unit 140a selects a detailed classification item based on the priority calculated in S207 (S209). Here, as described above, a case where a predetermined number of items with high priority is selected as the detailed classification item and the predetermined number = 6 will be described. First, in the priority management information of FIG. 16, the items are rearranged in the descending order of priority. Next, among the rearranged items, a predetermined number + 1 = 7 with high priority is selected. Here, the reason why 1 is added to the predetermined number is that when all the lower items are selected as the detailed classification items, the upper items are excluded from the detailed classification items. At this time, “meal” “dog” “clothes” “ramen” “shiba inu” “chihuahua” “Dalmatian” is selected. Of these, from the second row and the fifth, sixth, seventh rows in FIG. 13, it can be seen that all sub-items “丼”, “ramen” and “curry” of “meal” are selected as detailed classification items. “Meals” are excluded from the detailed classification items. In addition, from the second and sixth lines of FIG. 13, there are “ramen” selected as the detailed classification item among the lower items of “meal” and “と” and “curry” not selected. "Is excluded from the detailed classification items, and" meal and other "is added to the detailed classification items. Finally, the detailed classification items are six items of “clothes”, “ramen”, “meal and other”, “Shiba inu”, “Chihuahua”, and “Dalmatian”, and are stored as detailed classification item management information as shown in FIG. Thus, the process of S102 in FIG. 7 ends.

  As described above, when the image management apparatus 100a according to the present embodiment performs hierarchical structuring of an image group, the user's interest that cannot be understood from only image information is added to the diversity of the image group. The recognition items can be selected with priority. This classifies the image group by object recognition with the semantic granularity of the detail that matches the semantic content of the image group and the user's interest while preventing the deterioration of the classification performance due to the increase in the number of items of object recognition. It is possible to make a hierarchical structure of the image group in accordance with the contents and the user's interest.

  Note that the metadata acquired by the interest information extraction unit 147 may be acquired not for all the metadata but for a limited period. For example, only recent metadata may be acquired, such as “Recent 1 month”. Thereby, in selection of a detailed classification item, priority can be given according to a user's recent interest.

  Further, the metadata acquired by the interest information extraction unit 147 may be position information. At this time, the interest information extraction unit 147 may set the degree of interest of the item corresponding to the frequently appearing position information so as to increase. For example, if there are more than a certain number of images taken at a zoo, the user is considered to like animals, so the priority of items related to animals such as “dog”, “shiba inu”, “chihuahua”, “Dalmatian”, etc. May be increased. Here, the position information may be acquired from Exif (Exchangeable image file format) information attached to the image. Thereby, in selection of a detailed classification item, priority can be given according to the interest information understood from a user's action information.

  The metadata acquired by the interest information extraction unit 147 may be a user attribute. At this time, the interest information extraction unit 147 may set the degree of interest of the item corresponding to the user's specific attribute to be large. At this time, the user's attribute may be the user's gender or age. For example, if the user's gender is female and the age is in his twenties, the degree of interest in “sweets”, an item that women in their twenties are likely to shoot, may be set large. Thereby, in the selection of the detailed classification item, even if the user does not give metadata to the image, priority can be given according to the interest information that can be understood from the user's attributes.

(Embodiment 3)
In the first embodiment, the case where the information calculated by the detailed classification item selection unit 140 is used only for selection of the detailed classification item has been described. However, the clustering operation is performed based on the diversity calculated by the detailed classification item selection unit 140. These items may be switched. If the diversity for each item of the image group is known, switching the operation of the clustering unit in accordance with the diversity makes it possible to realize a hierarchical structure that is more suitable for the contents of the image group.

  The image management apparatus according to the third embodiment performs a hierarchical structure of image groups based on such an idea.

  Hereinafter, the image management apparatus according to the third embodiment will be described.

  Here, an embodiment in which the clustering operation for each item is selected based on the diversity calculated by the detailed classification item selection unit 140 will be described.

  FIG. 33 is a functional block diagram of the image management apparatus 100b according to the third embodiment. The configuration of the image management apparatus 100b differs from the image management apparatus 100 in that the internal operation of the clustering unit 130b and the content of the detailed classification item selection unit affect the internal operation of the clustering unit 130b.

  The clustering unit 130b in the present embodiment uses the diversity calculated by the detailed classification item selection unit 140 to determine whether or not to perform clustering.

  Next, the operation of the image management apparatus in this embodiment will be described. FIG. 34 shows a flowchart of the overall operation of the image management apparatus in the present embodiment. In the flowchart of FIG. 34, the processes of S108 and S109 are added as compared with the flowchart of FIG. The operations up to S104 are the same as those in the first embodiment. Hereinafter, the processing after S108 in FIG. 34, which is different from the first embodiment, will be described.

  After the processing up to S104, the clustering unit 130b determines whether to perform clustering on the summary classification item (S108). Specifically, if the item selected in S104 is the summary classification item acquired in S201 and the item whose diversity is a predetermined value or less is determined not to be clustered, otherwise clustering is performed. judge. Thereby, when diversity is small, it can prevent that an image group is classified too much by clustering. When it is determined in S108 that clustering is to be performed (Yes in S109), clustering is performed in the same manner as in the first embodiment (S105). If it is determined in S108 that clustering is not performed (No in S109), S105 is skipped and the process proceeds to S106.

  The configuration of the image management apparatus according to the third embodiment and the processing performed by the image management apparatus have been described above. A specific example of the operation of the image management apparatus according to the present embodiment will be described below.

  Here, the flow until the image management apparatus 100b performs clustering on the image group partially shown in FIG. 24 will be specifically described with reference to the flowcharts of FIGS. This specific example shows a specific example of processing for a group of 40 images whose contents are different from the specific example in the first embodiment. Each is assigned an image ID from 0 to 39, and FIG. 24 shows four images from 0 to 3 in the image group. In the image storage device 111, the input image shown in FIG. 24 is managed as shown in FIG. 11, for example.

  In this specific example, it is assumed that the semantic item candidates classified by the image management apparatus 100b have a hierarchical structure as shown in FIG.

  First, the image input unit 110 acquires an image and an image ID by acquiring image management information from the image storage device 111 (S101).

  Next, the detailed classification item selection unit 140 selects an item for performing object recognition (S102).

  A specific example of the processing of S102 will be described using the flowchart of FIG.

  The detailed classification item selection unit 140 first acquires a summary classification item from the semantic hierarchy storage unit (S201). For example, in the case of acquiring semantic hierarchy information shown in FIG. 13 having a hierarchy of 1, “meal”, “dog”, and “clothes” are selected.

  Next, the detailed classification item selection unit 140 classifies the image group into “meal”, “dog”, and “clothes” (S202).

  An example of the process of S202 will be described using the flowchart of FIG.

  First, the object recognizing unit 120 acquires a classification model of “meal”, “dog”, and “clothes” in the model selection unit 121 (S301).

  Next, in the object recognition execution unit 122, the object recognition unit 120 classifies each image input from the image input unit 110 by object recognition (S302). For example, the “meal”, “dog”, and “clothing” models selected in S301 and the SVM classifier are used for classification.

  The classification result is stored in the summary classification result storage unit 123 as summary classification result management information. In the example of FIG. 14, the images with image IDs 0, 1, 2, and 3 are classified into “meal”, “meal”, “dog”, and “clothes”, respectively, and summary category items. Here, it is assumed that all 40 images are classified into “meals”, 32 “dogs”, and “clothes”. Thus, the process of S202 in FIG. 8 ends.

  Returning to FIG. 8, next, the detailed classification item selection unit 140 selects one of the summary classification items “meal”, “dog”, and “clothes” acquired in S201 (S203). Hereinafter, in S203, the case where “meal” is selected will be described.

  Next, the detailed classification item selection unit 140 calculates the diversity of “meals” (S204). Here, a case where diversity is calculated as the sum of the image number score and the subject change score as described above will be described. First, the image score of “meal” and the ratio of the number of images classified as “meal” to the total number of images, that is, 32 ÷ 40 = 0.8. The subject change score is calculated as the average of the color histogram and edge feature amount variance of 20 images classified as “meal”. For example, if the subject change score is 0.8, the diversity score is the sum of the image number score and the subject change score, that is, 0.8 + 0.8 = 1.6. The values of the image number score, the subject change score, and the diversity score are stored as diversity management information as shown in the second row of FIG. Diversity scores are similarly calculated for “dogs” and “clothes”, and are calculated as diversity management information as shown in the third and fourth lines of FIG.

  After calculating the diversity of “meals”, “dogs”, and “clothes” (Yes in S205), the detailed classification item selection unit 140 selects items “meals”, “dogs”, “clothes”, “丼”, “ramen”, “ One of curry, shiba inu, chihuahua, dalmatian, t-shirt, suit, and kimono is selected (S206). Below, the example of each process at the time of selecting "meal" and "ramen" is shown. Further, here, as described above, a case will be described in which the priority is set to the diversity value of the upper item (however, when the hierarchy is 1, the priority is set to 10,000).

  When “meal” is selected in S206, the priority of “meal” is calculated in S207. Based on the semantic hierarchy information in FIG. 13, “meal” is identified as having a hierarchy of 1. Therefore, the priority of “meal” is 10,000. This is stored in the second row of the priority management information in FIG.

  When “ramen” is selected in S206, the priority of “ramen” is calculated in S207. Here, as described above, a case will be described in which the priority is set as the diversity value of the upper item as described above. Based on the semantic hierarchy information in FIG. 13, “meal” is specified as the upper item of “ramen”. Based on the diversity management information of FIG. 35, 1.6 is specified as the diversity score of “meal”. This 1.6 is stored as the priority of “ramen” in the sixth line of the priority management information in FIG. For the remaining 10 items, the priority is calculated in the same manner, and is calculated as the priority management information except for the second and sixth lines in FIG.

  After calculating the priority for all items (Yes in S208), the detailed classification item selection unit 140 selects a detailed classification item based on the priority calculated in S207 (S209). Here, as described above, the detailed classification items will be described in the case where a predetermined number of items with high priority are selected and the predetermined number = 5. First, in the priority management information shown in FIG. 36, the items are rearranged in descending order of priority. Next, among the rearranged items, a predetermined number + 1 = 6 with high priority is selected. Here, the reason why 1 is added to the predetermined number is that when all the lower items are selected as the detailed classification items, the upper items are excluded from the detailed classification items. At this time, “meal”, “dog”, “clothes”, “candy”, “ramen”, and “curry” are selected. Of these, from the second row and the fifth, sixth, seventh rows in FIG. 13, it can be seen that all sub-items “丼”, “ramen” and “curry” of “meal” are selected as detailed classification items. “Meals” are excluded from the detailed classification items. Finally, the detailed classification items are five items of “dog”, “clothes”, “candy”, “ramen”, and “curry”, and are stored as detailed classification item management information as shown in FIG. Above, the process of S102 in FIG. 34 is complete | finished.

  Returning to FIG. 34, next, the object recognition unit 120 classifies the image group into “dog”, “clothes”, “candy”, “ramen”, and “curry” (S103). A specific example of the process of S103 will be described using the flowchart of FIG.

  First, the object recognizing unit 120 obtains a classification model of “dog”, “clothes”, “candy”, “ramen”, and “curry” in the model selection unit 121 (S401).

  Next, in the object recognition execution unit 122, the object recognition unit 120 converts each of the images input from the image input unit 110 into the “dog”, “clothes”, “candy”, “ramen”, “curry” model and SVM selected in S401. Classification is performed by the classifier (S402).

  The classification result is stored in the detailed classification result storage unit 124 as detailed classification result management information as shown in FIG. Thus, the process of S103 in FIG. 34 ends.

  Returning to FIG. 34, the image management apparatus 100b next selects one of the detailed classification items “dog”, “clothes”, “candy”, “ramen”, and “curry” stored in the detailed classification item storage unit 146 (S104). Hereinafter, a case where the predetermined value for determining whether or not to perform clustering is set to 0.5, and “dog”, “clothes”, and “丼” are selected in S104 will be described as an example.

  When “dog” is selected in S104, the image management apparatus 100 determines whether or not to classify the “dog” image by clustering (S108). Specifically, “Dog” is one of the summary classification items acquired in S201, and the diversity score is 0. 0 from the fourth column of the “Dog” row in the diversity management information of FIG. 2. Since this is smaller than the predetermined value 0.5, it is determined that the “dog” is not clustered. In S109, No is determined, and the process proceeds to S106.

  When “clothes” is selected in S104, the image management apparatus 100 determines whether or not to classify the “clothes” image by clustering (S108). Specifically, “clothes” is one of the summary classification items acquired in S201, and the diversity score is 0.8 from the fourth column of the “clothes” row in the diversity management information in FIG. Since this is larger than the predetermined value, it is determined that “clothes” is to be clustered. In S109, Yes is determined, and the process proceeds to S105. For the image group to be clustered, the image having the image ID “2, 6, 11, 26” is specified from the third row in which the first column is “clothes” in the image management information classified by category in FIG. These are subjected to, for example, hierarchical clustering based on the similarity of the color histogram feature quantity, and the “clothes” image is clustered in more detail.

  When “丼” is selected in S104, the image management apparatus 100 determines whether or not to classify the image of “丼” by clustering (S108). Specifically, since “丼” is not the summary classification item acquired in S201, it is determined to cluster “丼”. In S109, Yes is determined, and the process proceeds to S105. The image group to be clustered has the image IDs “0, 7, 19, 27, 29, 32, 35, 37 from the fourth row in which the first column is“ 丼 ”in the image management information classified by category in FIG. , 38, 39 ". These are subjected to, for example, hierarchical clustering based on the similarity of the color histogram feature quantity, and the “丼” image is clustered in more detail.

  The remaining “ramen” and “curry” are similarly clustered, and the clustering result is stored in the clustering result storage unit 132 as clustering result management information as shown in FIG. FIG. 39 shows a hierarchical illustration of the parent-child relationship in FIG.

  As described above, the image management apparatus 100b according to the present embodiment selects whether to perform clustering based on the diversity calculated by the detailed classification item selection unit 140 when clustering image groups.

  Note that the diversity calculated by the detailed classification item selection unit 140 may be calculated from the number of images, not the ratio of the entire image. Thereby, even if the total number of images is large, the clustering operation can be selected so as not to cluster items whose classification number itself is small.

  As described above, the image management apparatus according to the present invention is not limited to those shown in the above-described embodiments.

  (1) It has been described that the image management apparatus according to the first, second, or third embodiment classifies the image group hierarchically based on the semantic content and generates a hierarchical structure as shown in FIG. 22 or FIG. At this time, the hierarchical structure generated by the hierarchical structure generation unit may be used to implement the narrowing search as shown in FIG.

  Thus, by letting the user specify a representative image of a partial image group having a similar meaning to the image that the user wants to search among the image group, the user can change the image to a partial image group having a similar meaning to the image specified by the user You can search while narrowing down the search range step by step. Therefore, since the search can be performed while narrowing down the granularity of the meaning of the object shown in the image, it is easy to search for an image desired by the user from a large number of input images.

  At this time, when classifying images with semantic items of a certain granularity, items with high diversity are classified in detail for each type item with lower semantic granularity, and items with low diversity are similarities between images. Based on the detailed classification.

  The overall configuration of the image management apparatus 100c in this modification is shown in FIG. Compared with the image management apparatus 100, the image management apparatus 100 c has a refinement search unit 170 added as an internal configuration, and the image management apparatus 100 is coupled to the user operation input unit 180 and the display unit 190. Is different.

  The image management apparatus 100c in the present modification example searches and searches for an image group based on the hierarchical structure management information stored in the hierarchical structure storage device 151. That is, the input image group is narrowed down by following the hierarchical structure shown in FIGS. 22 and 23 according to the selection of the partial image group corresponding to the representative image of the user. Specifically, after the hierarchical structure management information is stored in the hierarchical structure storage device 151, the display unit 190 displays a representative image of each child item for the “image group”. When the user operation input unit 180 receives the representative image selection information of the user, the display unit 190 displays the representative image of the partial image group of each item as a child of the item of the representative image selected by the user. This is repeated until the child of the image selected by the user is not the item name but the image ID.

  Thus, the user can search for a desired image from the input image group based on the semantic content.

  Here, for a representative image of a certain item, for example, a predetermined number of images are selected from images positioned below the item. As a representative image selection method, for example, a predetermined number of images may be selected in the order in which the feature amount of the image is close to the average of all the images positioned at the lower level of each item.

  In the example of FIG. 40, the image management apparatus 100c is configured to be coupled to the display unit 190 for displaying the representative image and the image group. However, instead of the display unit 190, the representative image or It is good also as a structure couple | bonded with the display control part (not shown) for displaying an image group.

  (2) It has been described that the image management apparatus according to the second embodiment selects the detailed classification item based on the metadata. At this time, clustering may be performed using metadata provided to the image by the user. For example, when a keyword indicating an item name is included in metadata provided by a user of a plurality of images, the plurality of images may be clustered as items of the keyword.

  FIG. 41 shows the overall configuration of the image management apparatus 100d in this modification. The image management apparatus 100d differs from the image management apparatus 100a in that the internal operation of the clustering unit 130d and the content of the detailed classification item selection unit affect the internal operation of the clustering unit 130d.

  The clustering unit 130d in this modification example determines whether or not to perform clustering using the metadata acquired by the detailed classification item selection unit 140a. When metadata is given to an image, the interest information extraction unit 147 determines whether a keyword representing an item name is included in the metadata. If there is a noun that matches the classification item name, the item name included in the metadata is managed in association with the image. In S105, the clustering unit 130d clusters images having a common lower item name when there are a plurality of images including the keyword representing the same lower item name for the summary classification item.

  For example, when the clustering unit 130d clusters an image group classified as “meal”, the comments “Ramen restaurant ABC dinner”, “Osaka ramen”, “Tokyo ramen” are added to the three images in the image group, respectively. Is given as metadata. “Ramen” is extracted from these comments as a common keyword. From the hierarchical semantic information in FIG. 13, it can be determined that “ramen” is a subordinate item of “meal”, so that the image group of “meal” is composed of a cluster composed of images with comments including ramen, Classify into clusters.

  According to this modification, a group of images can be hierarchically structured in accordance with the semantic content by supplementing the classification that is impossible only with image information with user metadata.

  The image management apparatus described in each of the above embodiments includes, for example, an AV device such as a BD recorder, a personal computer, a stationary terminal such as a server terminal, a digital camera, or the like having the function of the image management apparatus. It can be realized as a mobile terminal such as a mobile phone.

  Furthermore, it is possible to use a server apparatus that provides the method described in the above embodiment as a network service. In this case, when an image is received from a AV device, personal computer, digital camera, or the like in which the image is stored via a network, the image recognition process is performed on the received image by the method described in the above embodiment, The processing result may be transmitted to an AV device, personal computer, digital camera, or the like via a network.

  In this case, the image management apparatus includes an image storage unit (not shown), and the hierarchy according to the method described in the above embodiment is used for both the image received from the external device and the image stored in the image management apparatus. You may make it structure.

  In addition, a program describing the procedure of the method described in the above embodiment is stored in a memory, a CPU (Central Processing Unit) or the like reads the program from the memory, and executes the read program, thereby This method may be realized.

  Further, a program describing the procedure of the method may be stored in a recording medium such as a DVD and distributed. A program describing the procedure of the method may be widely distributed via a transmission medium such as the Internet.

  Each configuration according to each of the above embodiments may be realized as an LSI (Large Scale Integration) which is an integrated circuit. These configurations may be made into one chip, or may be made into one chip so as to include a part or all of them. Here, it is expressed as LSI, but depending on the degree of integration of the circuit, it may be referred to as IC (Integrated Circuit), system LSI, super LSI, or ultra LSI. Further, the method of circuit integration is not limited to LSI, and circuit integration may be performed with a dedicated circuit or a general-purpose processor. Alternatively, a field programmable gate array (FPGA) that can be programmed after the LSI is manufactured, or a reconfigurable processor (reconfigurable processor) that can reconfigure the connection and setting of circuit cells inside the LSI may be used. Alternatively, these functional blocks can be calculated using, for example, a DSP (Digital Signal Processor) or a CPU (Central Processing Unit). Further, these processing steps can be processed by being recorded on a recording medium as a program and executed.

(Supplement)
Hereinafter, the configuration of the image management apparatus according to the embodiment of the present invention, its modified examples, and each effect will be described.

  An image management apparatus according to the present invention includes an object recognition unit that classifies an image for each item of a semantic concept based on similarity to a general model, and a clustering unit that classifies an image based on similarity between image feature amounts According to the diversity of each semantic concept item, detailed classification item selection means for selecting the semantic concept items classified by the object recognition means with priority, classification in the object recognition means and the clustering means A hierarchical structure generating unit that integrates the results and generates a hierarchical classification structure of the image group, wherein the object recognition unit selects the image group as the item of the semantic concept selected by the detailed classification item selection unit. The clustering unit classifies each partial image group classified by the object recognition unit as a target.

  With this configuration, in the hierarchical structure of the image group, detailed object recognition is preferentially performed for items for which various objects are photographed, thereby minimizing the number of items for object recognition and matching the contents of the image group with object recognition. Thus, it is possible to make a hierarchical structure in accordance with the semantic content of the image group.

  In the image management apparatus according to the present invention, the diversity may be calculated from the number of images classified into semantic concept items and subject changes.

  As a result, an item with more classified images and a larger subject change can be classified based on more detailed meaning content of granularity.

  In the image management apparatus according to the present invention, the subject change may be a sum of an image color histogram and an edge feature amount variance.

  As a result, items with various colors and shapes can be classified based on the detailed contents of the granularity.

  In the image management apparatus according to the present invention, the clustering unit may determine whether to perform clustering based on diversity.

  Thereby, when diversity is small, it can prevent that an image group is classified too much by clustering.

  In the image management apparatus according to the present invention, the detailed classification item selection unit may select a predetermined number of items having high priority as items to be classified by the object recognition means.

  Thereby, the number of items classified by object recognition can be limited to a predetermined number. As a result, it is possible to prevent deterioration in accuracy of object recognition accompanying an increase in the number of items for object recognition.

  The image management apparatus according to the present invention further includes an interest information extraction unit that extracts user interest information from metadata provided by the user, and the priority is calculated in consideration of the user interest information. It is good as well.

  Thus, when the image group is hierarchically structured, the object recognition items can be selected with priority given to the diversity of the image group and the user's interest that cannot be extracted only from the image information. As a result, it is possible to perform a hierarchical structure in accordance with the semantic content of the image group and the user's interest.

  In the image management apparatus according to the present invention, the metadata includes at least one of a word included in the text data provided by the user, position information where the image is captured, the age of the user, and the gender of the user. May be included.

  Thus, the object recognition item can be selected with priority given to the text data provided by the user, the location information frequently visited by the user, and the degree of interest according to the user's attributes. As a result, it is possible to perform a hierarchical structure in accordance with the semantic content of the image group and the user's interest.

  Further, in the image management apparatus according to the present invention, the clustering unit may cluster the common metadata assigned to the image by the user.

  As a result, classification that is impossible with only image information can be supplemented with user metadata. As a result, the image group can be hierarchically structured more in accordance with the semantic content.

  The image management apparatus according to the present invention includes user operation input means for acquiring a user input operation, and display control means for displaying an image group on a display device, wherein the user operation input means is configured to display an image group. The user's selection operation for any one of the representative images of the partial image group obtained by classification is acquired, and the display control means causes the display device to display the representative image of the partial image group obtained by classifying the image group. In response to the user selection operation, the image group is repeatedly narrowed down to the partial image group, and the classification is performed when the image group is classified into semantic concept items having a predetermined granularity. Items with high diversity for each semantic concept item are classified in detail for each semantic concept item with a granularity lower than the granularity, and items with low diversity are classified in detail based on the similarity between images.

  As a result, it is possible to search while narrowing down images having similar meanings to the image that the user wants to search. As a result, an image desired by the user can be searched from a large number of images.

  The image management device according to the present invention can be applied to a digital still camera, an imaging device such as a camera-equipped mobile phone or a movie camera, and a PC (Personal Computer).

100, 100a, 100b, 100c, 100d Image management device 110 Image input unit 111 Image storage device 120 Object recognition unit 121 Model selection unit 122 Object recognition execution unit 123 Summary classification result storage unit 124 Detailed classification result storage unit 130, 130b, 130d Clustering unit 131 Clustering execution unit 132 Clustering result storage unit 140, 140a Detailed classification item selection unit 141 Diversity calculation unit 142, 142a Priority calculation unit 143 Semantic hierarchy storage unit 144 Diversity storage unit 145 Priority storage unit 146 Detailed classification item Storage unit 147 Interest information extraction unit 148 Interest level storage unit 150 Hierarchical structure generation unit 151 Hierarchical structure storage device 161 Metadata storage device 170 Narrowing search unit 180 User operation input unit 190 Display unit

Claims (12)

An image management device that hierarchically classifies an image group according to image content,
The image management device includes:
Object recognition means for classifying images into semantic concept items based on similarity to general models;
Clustering means for classifying images based on the similarity between image feature quantities;
Detailed classification item selection means for prioritizing and selecting the semantic concept items to be classified by the object recognition means according to the diversity of semantic concept items;
A hierarchical structure generating unit that generates a hierarchical classification structure of an image group by integrating the classification results in the object recognition unit and the clustering unit;
With
The object recognition means includes
Classifying the image group into partial image groups for each item of the semantic concept selected by the detailed classification item selection means;
The image management apparatus, wherein the clustering unit classifies each partial image group classified by the object recognition unit. The diversity is calculated from the number of images classified into the semantic concept items and subject changes. The image management apparatus according to claim 2, wherein the subject change is a sum of an image color histogram and an edge feature amount variance. The image management apparatus according to claim 2, wherein the clustering unit determines whether to perform the clustering based on the diversity. The image management apparatus according to claim 1, wherein the detailed classification item selection unit selects a predetermined number of items having high priority as items to be classified by the object recognition unit. The image management device further includes:
An interest information extraction unit that extracts user's interest information from the metadata given by the user,
The image management apparatus according to claim 1, wherein the priority is calculated in consideration of interest information of the user. The metadata includes at least one of a word included in text data provided by the user, position information obtained by capturing the image, an age of the user, and a gender of the user. The image management apparatus described in 1. The image management apparatus according to claim 6, wherein the clustering unit clusters items having the same metadata assigned to the image by the user. The image management device includes:
User operation input means for acquiring a user input operation;
Display control means for displaying an image group on a display device;
With
The user operation input means includes
Obtaining a user's selection operation for one of the representative images of the partial image group obtained by classifying the image group;
The display control means displays a representative image of a partial image group obtained by classifying the image group on the display device, and narrows down the image group to the partial image group in accordance with the selection operation of the user. Repeatedly,
In the classification, when the image group is classified into semantic concept items having a predetermined granularity, items having high diversity for each semantic concept item included in the image group are semantic concept items having a granularity lower than the granularity. Each item is classified in detail, and the low diversity items are classified in detail based on the similarity between images.
An image management apparatus characterized by the above. An image management method for hierarchically classifying image groups according to image contents,
The image management method includes:
An object recognition step for classifying images into items of semantic concepts based on similarity to a general model;
A clustering step of classifying the images based on the similarity between the feature quantities of the images;
A detailed classification item selection step for prioritizing and selecting the semantic concept items to be classified by the object recognition means according to the diversity for each semantic concept item;
Run
The object recognition step includes
Classifying the image group into partial image groups for each item of the semantic concept selected by the detailed classification item selection means;
The image management method, wherein the clustering means classifies each partial image group classified by the object recognition means. A program for causing a computer to execute an image management method for hierarchically classifying image groups according to image contents,
The program is
An object recognition step for classifying images into items of semantic concepts based on similarity to a general semantic model;
A clustering step of classifying the images based on the similarity between the feature quantities of the images;
A detailed classification item selection step for prioritizing and selecting the semantic concept items to be classified by the object recognition means according to the diversity for each semantic concept item;
To the computer,
The object recognition step includes
Classifying the image group into partial image groups for each item of the semantic concept selected by the detailed classification item selection means;
The clustering means performs classification on each partial image group classified by the object recognition means. An integrated circuit for an image management apparatus that hierarchically classifies an image group according to image contents,
The integrated circuit comprises:
Object recognition means for classifying images into items of semantic concepts based on similarity to general semantic models;
Clustering means for classifying images based on the similarity between image feature quantities;
Detailed classification item selection means for prioritizing and selecting the semantic concept items to be classified by the object recognition means according to the diversity of semantic concept items;
With
The object recognition means includes
Classifying the image group into partial image groups for each item of the semantic concept selected by the detailed classification item selection means;
The integrated circuit according to claim 1, wherein the clustering means classifies each partial image group classified by the object recognition means.

Images