JP6283308B2 - Image dictionary construction method, image representation method, apparatus, and program - Google Patents

Description

  The present invention relates to an image dictionary construction method, an image expression method, an apparatus, and a program, and more particularly to an image dictionary composition method, apparatus, and program for constructing an image dictionary for obtaining an image representation of an image, and an image image. The present invention relates to an image expression method, apparatus, and program for obtaining expression.

  Due to the advancement and quality of communication environments, computers, and distributed processing infrastructure technologies, the number of images and video content distributed on the network has become enormous. One site reports that 350 million images are uploaded every day, and another site reports that 64 hours of video are newly released per minute.

  Such an enormous amount of content is a rich source of information for users, but it also raises the problem that it becomes increasingly difficult to quickly access the content that the user wants to browse. In such a trend, there is an increasing demand for media analysis technology for efficiently searching for contents to be browsed and viewed.

  Although the following description is limited to the image, it goes without saying that the video can be applied as it is to the video within the scope described in the present specification because it is composed of a series of continuous images.

  The first step in image analysis is to obtain an image representation, ie describe the images as numerically comparable vectors. In this way, for example, when image recognition is performed, it is possible to classify the space in a specific area within the space generated by the image expression into the same category. Alternatively, in the case of an image search, when a certain image is given as a query, the similarity between the images can be evaluated by image representation, and a similar image can be searched. In addition, even in the case of image recommendation, when a user finds an image similar to the image browsed / viewed so far, recommends it, and collects many images into a smaller number of representative images. Then, a process for finding and omitting a similar image is executed. In either case, image representation is required.

  In view of the above utilization form, it is preferable that the “semantic content” of an image is captured as a requirement for image representation. “Semantic content” here means subjects (“dog”, “house”, “computer”, etc.) and scenes (“coast”, “office”, “forest”, etc.) contained in the image. This is a part or object that characterizes the character and its characteristic, and is a partial area in the image that can be designated as a language. For example, if it is a “dog”, it should be “coast” such as “ear shape” (“pointed ear”, “drooping ear”, etc.) or “foot” (“short round leg”, “elongate leg”, etc.) "Beach", "Sea", "Ship", etc. If the presence or absence of such parts and objects is obtained in the image, it is possible to a priorily infer the subject and scene that are photographed from the set. As a result, classification and search based on semantic content are possible, and the industrial application value is high.

  In the past, various image expression methods have been devised. The technique described in Patent Document 1 discloses a method in which statistics over the entire image are obtained as a histogram or the like for image brightness, color, texture (pattern), edge, and the like, and this is used as an image representation.

  Also, a method has been disclosed in which an image dictionary serving as a model is configured in advance for obtaining an image expression, and the image expression is obtained based on this dictionary.

  Non-Patent Document 1 discloses a technique generally known as Bag-of-Words or Bag-of-Key-Points. In this technique, an image is regarded as a set of minute regions on several pixels, and the presence or absence of the region is counted over the entire image, thereby expressing the image as a histogram of the minute region. First, a set of minute regions having a particularly high contrast in an image is obtained, the minute regions are described by a luminance gradient, and then quantized to obtain an image dictionary (codebook). When an image is expressed, a luminance gradient of a minute region having a strong contrast over the entire image is obtained, and these are encoded based on a dictionary. Thereafter, the appearance frequency of each code is obtained and converted into a histogram, which is used as an image expression.

  Non-Patent Document 2 discloses a method for finding an area that characteristically represents a subject or a scene and expressing an image by this. In this method, an image is divided into random partial areas, and similar partial areas are grouped by clustering them. Subsequently, the feature amount of the partial region included in each cluster is modeled as a function by Support Vector Machine (SVM) or the like, and the partial region suitable for this model (that is, when the feature amount of the partial region is input to the model, A dictionary is constructed as a set of those whose output values are high. When a new image is input using the obtained dictionary, the frequency of appearance of a partial region similar to the partial region registered in the dictionary is obtained, and the image is expressed by forming a histogram.

JP 2014-67174 A

J. Sivic and A. Zisserman, “Video Google: A Text Retrieval Approach to Object Matching in Videos”, In Proc. Of IEEE International Conference on Computer Vision, 2003, pp. 1470-1477 S. Singh, Saurabh Singh, A. Gupta, and Alexei A. Efros, `` Unsupervised Discovery of Mid-Level Discriminative Patches '', In Proceedings of the European Conference on Computer Vision, 2005, pp.239-248

  As described above, from the viewpoint of many industrial applications such as image recognition and image search, it is preferable that the image expression is an expression that well represents the semantic content of the subject or scene. In this respect, the prior art has the following problems.

  The techniques described in Patent Document 1 and Non-Patent Document 1 express an image with very low-order physical quantities (color, texture, luminance gradient, etc.) extracted from the entire image. However, there is a problem in that low-order physical quantities extracted from the entire image cannot discriminate subjects and scenes that are similar to each other. In particular, even the same “bird” (similar to “hawk” and “falcon”), or similar to “dog” (“Siberian Husky” and “Alaskan Malamute”) Etc.), although there are some differences, the whole picture is very similar, and practical image recognition accuracy cannot be obtained with such image representation.

  On the other hand, the technique described in Non-Patent Document 2 is an image dictionary that can identify minute differences between subjects in image representation by extracting partial areas that are different from characteristic partial areas and constructing an image dictionary. May be earned. However, since we are still trying to build an image dictionary based only on image features, we need to identify and extract partial areas based on semantic content (such as “pointed ears” and “elongate legs”) as described above. There is a problem that a valid image dictionary cannot be obtained.

  From the above, the inventions that have been disclosed in the past are all image dictionary construction techniques that can acquire an image representation that represents the semantic content of the subject or scene in the image, which is a requirement for the image representation, and It was not an image expression technology.

  The present invention has been made to solve the above-described problems, and can constitute an image dictionary for obtaining an image expression capable of finding a meaningful characteristic region in an image. It is an object to provide an image dictionary construction method, apparatus, and program.

  It is another object of the present invention to provide an image expression method, apparatus, and program capable of obtaining an image expression capable of finding a meaningful characteristic area in an image.

  To achieve the above object, an image dictionary construction method according to the first invention includes a partial region dividing unit, a feature amount extracting unit, a classifying unit, a candidate region determining unit, and a discriminator learning unit, An image dictionary construction method in an image dictionary construction device that constructs an image dictionary from each of one or more input images and document data corresponding to each of the images, wherein the partial region dividing unit is the input Each of the one or more images is divided into one or more partial areas, and the feature amount extraction unit is included in a set of partial areas formed by the partial areas divided by the partial area division unit. Extracting a feature amount for each of the partial regions, and the classification unit based on a similarity degree of each of the partial regions extracted by the feature amount extraction unit, the set of the partial regions Part Classifying each of the regions into one of one or more clusters, and the candidate region determining unit, for each of the clusters, for each of the partial regions classified into the clusters by the classifying unit Determining a candidate area that is a partial area representing the cluster based on the feature quantity of the partial area and the input document data corresponding to the image including the partial area; and the classifier For each of the clusters, the learning unit uses the feature amount of the candidate region determined by the candidate region determination unit as a positive example, and sets the feature amount of the partial region not classified into the cluster as a negative example. Learning and obtaining a classifier for identifying whether a region belongs to the cluster, the classifier obtained for each of the clusters, And executes includes a step of outputting as an image dictionary, a.

  An image dictionary construction device according to a first aspect of the present invention is an image dictionary construction device that constructs an image dictionary from each of one or more images received as input and document data corresponding to each of the images. The partial region included in a set of partial regions formed by the partial region dividing unit that divides each of the one or more images that have been divided into one or more partial regions, and the partial region divided by the partial region dividing unit Each of the partial regions of the set of partial regions based on a similarity with respect to the feature amounts of the partial regions extracted by the feature amount extraction unit. A classifying unit that classifies one of the one or more clusters, and for each of the clusters, for each of the partial areas classified into clusters by the classifying unit, A candidate area determination unit that determines a candidate area that is a partial area representing the cluster based on the feature amount of the partial area and the input document data corresponding to the image including the partial area; and the cluster For each of the above, the partial region belongs to the cluster, with the feature amount of the candidate region determined by the candidate region determination unit as a positive example and the feature amount of the partial region not classified into the cluster as a negative example A classifier learning unit that learns and acquires a classifier for identifying whether or not the cluster is obtained and outputs the classifier acquired for each of the clusters as an image dictionary.

  An image representation method according to a second aspect of the present invention is an image representation method in an image representation device including a partial region dividing unit, a feature amount extracting unit, and a representation unit, wherein the partial region dividing unit includes an input Dividing the image into one or more partial areas, the step of extracting the feature quantity for each of the partial areas, and the expression section extracting by the feature quantity extraction section On the basis of the feature amount of each of the partial areas and the image dictionary output by the image dictionary construction method according to the first invention, the partial area belongs to each of the clusters for each of the partial areas. To determine whether the partial region belongs to any of the clusters or does not belong to any of the clusters based on the calculated accuracy, and based on the result of the determination , A histogram representing the frequencies which are determined to belong to the cluster for each of the clusters, and executes contain, and outputting an image representation of the input image.

  An image expression device according to a second aspect of the present invention includes a partial region dividing unit that divides an input image into one or more partial regions, a feature amount extracting unit that extracts a feature amount for each of the partial regions, Based on the feature amount of each of the partial regions extracted by the feature amount extraction unit and the image dictionary output by the image dictionary construction device according to claim 2, the partial region is the Calculating the accuracy belonging to each of the clusters, and determining whether the partial region belongs to any of the clusters or does not belong to any of the clusters based on the calculated accuracy; A representation unit that outputs, as an image representation of the input image, a histogram representing the frequency determined to belong to the cluster for each of the clusters based on a result. It is configured.

  A program according to a first invention is a program for causing a computer to execute each step constituting the image dictionary construction method or the image expression method according to the first invention.

  According to the image dictionary configuration method, apparatus, and program of the present invention, an input image is divided into partial areas, each of the partial areas is classified into clusters, and for each of the clusters, the feature amount of the partial area and the partial Based on the document data corresponding to the image including the region, a candidate region that is a partial region that is a representative of the cluster is determined, and the classifier is learned using the candidate region as a positive example. An effect is obtained that an image dictionary for obtaining an image expression capable of finding a certain characteristic region can be constructed.

  In addition, according to the image expression method, apparatus, and program, an input image is divided into partial areas, and each of the partial areas is determined based on the feature amount of each partial area and the image dictionary. A histogram representing the frequency of each cluster determined to belong to the cluster is output as an image representation of the input image. By doing so, the effect that the image expression which can discover the meaningful characteristic area in an image can be calculated | required is acquired.

It is a block diagram which shows the structure of the image dictionary structure apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the image representation apparatus which concerns on embodiment of this invention. It is a flowchart which shows the image dictionary structure process routine in the image dictionary structure apparatus which concerns on embodiment of this invention. It is a flowchart which shows the image expression process routine in the image expression apparatus which concerns on embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of Image Dictionary Configuration Device According to First Embodiment of the Present Invention>

  First, the configuration of the image dictionary configuration device according to the first embodiment of the present invention will be described. As shown in FIG. 1, the image dictionary construction device 100 according to the first embodiment of the present invention stores a CPU, a RAM, a program for executing an image dictionary construction processing routine described later, and various data. It can be composed of a computer including a ROM. Functionally, the image dictionary construction device 100 includes an image database 10, a calculation unit 20, and an image dictionary 50 as shown in FIG.

  It is assumed that the image database 10 stores an image itself or an address that uniquely indicates the location of the image file. Further, it is assumed that document data corresponding to one or more images among the stored images is stored. This document data represents the semantic content related to the entire image. The document data representing the semantic content here is a document describing a part or object that characterizes a subject or scene photographed in an image. The format may be given in the form of a keyword, for example, or may be given as a sentence. In the former case, it is preferable that the word is given as a word that describes the subject or scene of the image taken in whole or in part. For example, if the subject is a “dog”, “ear shape” (“pointed ears”, “drooping ears”, etc.), “foot” (“short round legs”, “elongate legs”, etc.), etc. “shore” If so, it will be given as “Beach”, “Sea”, “Ship” etc. In the latter case, the document is preferably given as a document describing the subject or scene of the image. The specific nature is arbitrary. For example, if there is a dog with a sharp ear on the coast where the ship travels, it may be described as “the dog is on the coast” or “the coast where the ship travels” It may be written that there is a dog with a sharp ear.

  The method for preparing the above document data does not matter. For example, when an image on a web page on the Internet is used, there is usually a document associated with the image around the image, but this may be used as document data. In this case, there is an advantage that document data can be obtained without human intervention. Alternatively, the document data may be manually input for each image. In this case, there is an advantage that highly reliable document data can be configured in accordance with a person's accurate judgment.

  Further, the image database 10 only needs to be able to store each image or address and the corresponding document data in association with each other, and may be configured by a so-called RDBMS (Relational Database Management System) or the like. The image database 10 may be inside or outside the image dictionary construction apparatus 100, and any known communication means can be used. Furthermore, as long as the image dictionary construction apparatus 100 can receive one or more images as input, it does not necessarily have to be a database. In this embodiment, it is assumed that the image database 10 is external, and the communication means is connected to communicate via the Internet and TCP / IP.

  Further, each unit and the image database 10 included in the image dictionary configuration apparatus 100 may be configured by a computer or a server including an arithmetic processing device, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image dictionary construction apparatus 100, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Of course, any other component does not have to be realized by a single computer or server, and may be realized by being distributed to a plurality of computers connected by a network.

  The computing unit 20 includes a partial region dividing unit 30, a feature amount extracting unit 32, a classification unit 34, a candidate region determining unit 36, and a classifier learning unit 38.

  The partial area dividing unit 30 reads one or more images input from the image database 10, divides and selects each image into one or more partial areas, and outputs these to the feature amount extracting unit 32.

  Hereinafter, the partial area extraction processing in the partial area dividing unit 30 will be described in detail. This process is performed for all the images stored in the image database 10, but since the same process is performed for all the images, only the process for one image will be described here.

  In the partial area extraction process, only the partial area is cut out and extracted from the entire image. Specifically, the number of partial areas and the partial area size are designated, and partial areas are extracted at regular intervals.

  For example, an example in which the original image size is 360 × vertical × 240 horizontal, the number of partial areas is 16 × 16 = 256, and the partial area size is 32 × 32 pixels will be described. In this case, the vertical is (360-32) / 16 = 20 pixels (every decimal point cut) and the horizontal is (240-32) / 16 = 13 pixel shift, 32 pixels × 32 pixels. Extract a partial area.

  Any positive integer may be set for the number of partial areas and the partial area size. The partial areas may overlap each other, or a combination of several settings may be used.

  By performing the above processing on the entire image, a set of partial areas can be obtained. The partial area set obtained in this way is output to the feature quantity extraction unit 32, and the process is terminated.

  The feature amount extraction unit 32 analyzes each partial region included in the set of partial regions composed of the partial regions of the image divided by the partial region dividing unit 30, and extracts a predetermined feature amount. The feature amount is output to the classification unit 34. In the present embodiment, an image feature vector is extracted as a feature amount.

  Hereinafter, extraction of feature amounts in the feature amount extraction unit 32 will be described. In this embodiment, all the feature quantities listed below are extracted, but what kind of feature quantity is extracted is not important as a requirement of the embodiment of the present invention, and is a publicly known publicly known feature extraction. Processing may be used. Specifically, any numerical data (scalar or vector) having a dimension extracted from an image is effective for all feature quantities and combinations thereof. For example, brightness feature, color feature, texture feature, landscape What is necessary is just to extract a feature, a shape feature, etc.

  The brightness feature can be obtained as a histogram of V values in the HSV color space for the pixels in the partial region.

  The color feature can be obtained as a histogram of the values of the respective axes (L *, a *, b *) in the L * a * b * color space.

  As the texture feature, a local feature amount may be extracted for each key point extracted from the partial region at regular intervals. As the local feature, for example, SIFT (Scale Invariant Feature Transform) described in Reference Document 1 below, SURF (Speeded Up Features) described in Reference Document 2 below, and the like can be used.

  [Reference 1] D.G. Lowe, “Distinctive Image Features from Scale-Invariant Keypoints”, International Journal of Computer Vision, pp.91-110, 2004

  [Reference 2] H. Bay, T. Tuytelaars, and L.V. Gool, “SURF: Speeded Up Robust Features”, Lecture Notes in Computer Science, vol. 3951, pp.404-417, 2006

  The local feature extracted by these is, for example, a 128-dimensional real value vector per key point, and therefore has 128 dimensions × the number of key points. Alternatively, refer to the code length generated by learning this vector in advance, convert it to a code, and generate a histogram by counting the number of codes present in a block of an appropriate size in the partial area. Can do. In this case, the number of bins in the histogram matches the code number of the code length. Alternatively, the sparse expression described in Reference 3 or the feature expression based on the Fisher kernel described in References 4 and 5 may be used.

  [Reference 3] Jinjun Wang, Jianchao Yang, Kai Yu, Fengjun Lv, Thomas Huang, and Yihong Gong, “Locality-constrained Linear Coding for Image Classification”, IEEE Conference on Computer Vision and Pattern Recognition, pp. 3360-3367, 2010.

  [Reference 4] Florent Perronnin, Jorge Sanchez, Thomas Mensink, “Improving the Fisher Kernel for Large-Scale Image Classification”, European Conference on Computer Vision, pp. 143-156, 2010.

  [Reference 5] Herve Jegou, Florent Perronnin, Matthijs Douze, Jorge Sanchez, Patrick Perez, Cordelia Schmid, “Aggregating Local Image Descriptors into Compact Codes”, IEEE Trans. Pattern Recognition and Machine Intelligence, Vol. 34, No. 9, pp. 1704-1716, 2012.

  In any case, the resulting feature quantity is a real value vector having a length that depends on the number of codes of the code length.

  A landscape feature is a feature amount that represents a landscape or scene of an image. For example, the GIST descriptor described in Reference 6 can be used. The GIST descriptor is expressed by a coefficient when a filter having a certain orientation is applied to each block in the partial area. In this case, the generated feature amount depends on the type of filter (number of blocks × number of orientations). It becomes a vector of the length.

  [Reference 6] A. Oliva and A. Torralba, “Building the gist of a scene: the role of global image features in recognition”, Progress in Brain Research, 155, pp. 23-36, 2006

  The shape feature is a feature amount representing the shape of an object shown in an image. For example, a Histogram of Oriented Gradient (HOG) feature amount described in Reference Document 7 or an edge histogram can be used.

  [Reference 7] N. Dalal and B. Triggs, “Histograms of Oriented Gradients for Human Detection”, IEEE Conference on Computer Vision and Pattern Recognition, pp.886-893, 2005

  One or a plurality of these feature quantities may be used, or other known feature quantities may be used.

  The feature quantity extraction unit 32 outputs the feature quantity of each block for each partial region obtained by the above process to the classification unit 34, and ends the process.

  The classification unit 34 classifies each partial region of the partial region set into one of one or more clusters based on the similarity of the partial region extracted by the feature amount extraction unit 32 with respect to each image feature vector. Then, the classification result is output to the candidate area determination unit 36.

  In the classification unit 34, in the feature amount extraction by the feature amount extraction unit 32, each partial region is expressed as the same image feature vector, and therefore any clustering method can be used for classification. For example, it may be classified into an arbitrary number of clusters by using a K-means method or the like. The number of clusters may be set to 1/4 of the total number of partial areas, for example.

  Here, depending on the case, there may be a case where a very large number of partial regions belong to some clusters, or a case where only a small number of partial regions belong. The partial areas belonging to such clusters are often extremely general partial areas or extremely rare partial areas, and may be less effective for image recognition and search. Therefore, a cluster in which the number of partial areas in the cluster is equal to or greater than a certain value (for example, 1000 or more) and a cluster that is equal to or less than a certain value (for example, 3 or less) may be deleted.

  Then, the classification unit 34 outputs the obtained classification result (each of the partial areas and the cluster to which each of the partial areas belong) to the candidate area determination unit 36, and ends the process.

  The candidate area determination unit 36 receives the classification result output from the classification unit 34, and for each of the clusters, the image feature vector of the partial area and the image for each of the partial areas classified into the cluster by the classification unit 34 Based on the document data corresponding to the image including the partial area stored in the database 10, a candidate area that is a partial area representing the cluster is determined. The candidate area is used as a positive example in learning of the classifier of the classifier learning unit 38.

  The process of the candidate area determination unit 36 is a process that is important in constructing an image dictionary that holds the semantic content that is the object of the embodiment of the present invention. Specifically, the object is to construct an image dictionary storing partial areas that satisfy the following two requirements.

  Requirement 1 is a partial area having a typical appearance that frequently appears in an image in the image database 10. Requirement 2 is a partial area that captures the semantic content of the subject and scene that appear in the image in the image database 10.

  Such a partial area not only can efficiently describe subjects and scenes included in the image database (requirement 1), but can also capture its semantic content (requirement 2) at the same time. It is suitable as an image dictionary for realizing image recognition, image search, and the like.

  Hereinafter, the first to fourth processes in the candidate area determination unit 36 will be described in detail. In the process of the candidate area determination unit 36, exactly the same process is performed for each cluster, so only the process in a single cluster is described here.

  The candidate area determination unit 36 first takes the correspondence between the partial area and the document data as the first process. As described above, the image database 10 stores an image (or an address that uniquely indicates the location) and document data representing the semantic content of the entire image in association with each other. Therefore, by looking at which image the partial area is extracted from, it is possible to associate the partial area with the document data associated with the extracted image. With this first processing, document data is associated with all or some of the partial areas.

  Subsequently, as a second process, the candidate area determination unit 36 constructs a document vector obtained by digitizing the document data for each partial area associated with the document data in the first process. Any known technique may be used for this, and a Bag-of-words histogram in which a vector is configured based on the normal frequency of words can be simply applied. Alternatively, the tf-idf method weighted by the word occurrence frequency may be applied. Such a method is convenient in that it can be uniformly converted into the same document vector and expressed regardless of whether the document data registered in the image database 10 is a word or a sentence. Is expensive.

  Subsequently, the candidate area determination unit 36 performs the third and fourth processes to find a candidate area from the image feature vector of the partial area and the document vector of the document data associated with the partial area.

  The purpose of the third and fourth processes is to narrow down a smaller number of candidate areas from the partial areas belonging to each cluster and the corresponding document vectors. More precisely, when there are M feature image vectors and corresponding document vectors in a certain cluster, T <M candidate regions are selected and determined from these.

  The basic policy of the third and fourth processes of the candidate area determining unit 36 is to calculate a degree representing a cluster as a representativeness score among the partial areas belonging to the cluster to be processed, T pieces are selected as candidate areas. There are various methods for obtaining such a representative score. For example, in the K-means method or the like, the center of the cluster, that is, the point having the smallest sum of the distances from other partial regions belonging to the cluster is representative. From this point of view, it can be said that the representativeness score is given by the proximity of other data. Based on the same idea, in the present embodiment, it is determined that the distance is determined based on the proximity of other partial areas, and this is determined using clustering. Hereinafter, clustering in the third process will be described in detail.

  In the third process of the candidate area determination unit 36, clustering is separately applied to each of the image feature vector and document vector of the partial area for the partial area set belonging to the cluster to be processed. Find the degree score. Note that this processing applies only to the case of an image feature vector because the same processing is applied to both an image feature vector and a document vector.

  Any known clustering method may be used. However, depending on the fact that there are often differences in the variation in the feature values of the partial areas in the cluster, and depending on the type of the image feature vector (or document vector), an appropriate distance (Euclidean distance, cosine similarity, or In view of the fact that histogram intersections and the like are different, it is preferable to specify Affinity Propagation, as compared to the case where it is necessary to specify the number of clusters in advance or the distance is limited to a specific type as in the K-means method. It is better that the number of clusters can be automatically estimated and clustering can be performed based on an arbitrary distance.

  As a result of the clustering in the third process of the candidate area determination unit 36, the cluster centers of K image feature vectors can be found. The candidate area determination unit 36 obtains a representative score based on these cluster centers.

  In the third process of the candidate area determination unit 36, for example, the one having the smallest average distance (or any statistic such as a median value) between the cluster center and the partial area belonging to the cluster is represented. A representativeness score of 1.0 is given as the cluster center, and the representativeness score may be determined so that the score increases in order from the closest distance from the representative cluster center. In this case, the representative score is calculated by setting the distance from the center of the representative cluster as dist, for example:

  Etc.

  Alternatively, the distance from the nearest cluster center among the K cluster centers may be obtained as dist, and similarly, the representative score may be obtained using the above equation (1).

  Although the representativeness score based on the image feature vector is obtained here, the representativeness score based on the document vector is obtained similarly. If there is a partial area for which there is no associated document vector, the representative score based on the document vector may be determined as a predetermined value. For example, it may be set to 0 uniformly, or an average value or median value of the obtained document vectors.

  Next, in the fourth process, the candidate area determination unit 36 selects a candidate area based on the image feature vector and the representative score based on the document vector previously obtained for the cluster to be processed. Finally, a final representativeness score is obtained based on both the representativeness score obtained independently from the image feature vector and the document vector, and a candidate region is selected based on this.

  In the simplest case, the partial areas are ranked in descending order of the sum of the representative scores of the image feature vector and the document vector, and T elements having the highest sum are selected as candidate areas. Instead of using the sum of the representative scores of the image feature vector and the document vector, either the larger or the smaller representative score of the image feature vector and the document vector is used as the representative degree of the partial area. You may adopt as a score.

  Or, by ranking each image feature vector and document vector in descending order of representative score, creating two different ranking lists, and then integrating these two ranking lists into one ranking list May be. When integrating the two rankings, for example, the Boulder scoring method can be used. In this case, M points to 1 point may be given to the ranking lists 1 to M, respectively, and T may be selected as a candidate region in descending order of the sum. Such a method is robust and highly accurate because, even when the representativeness score includes noise, a robustly effective candidate region can be selected only by its rank.

  Then, the candidate area determination unit 36 outputs the candidate area for each cluster selected by the fourth process to the classifier learning unit 38, and ends the process.

  As a first process, the classifier learning unit 38 uses, as a positive example, the image feature vector of the candidate area selected by the candidate area determination unit 36 for each of the clusters, and selects any partial area not classified into the cluster. Using the image feature vectors of the partial areas included in the set as a negative example, the discriminator is learned and output as an image dictionary 50.

  In general, there are various known methods for learning a discriminator under given positive examples and negative examples. An arbitrary one may be used, but in this embodiment, SVM or Support vector regi- sion (SVR) described in [Reference 8] can be used.

[Reference 8] A.J. Smola, B. Scholkopf “A Tutorial on Support Vector Regression”, Statistics and Computing, Vol. 14, Issue 3, pp.199-222, 2004

  In any case, with respect to a certain cluster, the candidate region is a positive example, and an arbitrary partial region that does not belong to the cluster is a negative example. Can be obtained. Then, the classifier obtained for each cluster is output as the image dictionary 50, and the processing is terminated.

  As the second process, the classifier learning unit 38 applies the classifier once learned to a new image data set prepared in advance, and which cluster the partial area included in the image data set belongs to. And classifying each cluster, the classification result may be output to the candidate area determination unit 36. Thereafter, the candidate area determination unit 36 determines the candidate area of each cluster for the new image data set classified into each cluster by the process of the candidate area determination unit 36 described above, and then performs the classifier learning again. To the unit 38. In this way, when a new image data set is added, it is obtained by repeatedly executing candidate region determination and classifier learning using the image data set until a predetermined condition is satisfied. By outputting the discriminator as the image dictionary 50, the influence of the bias of the image data set is reduced, and it reacts to a partial area having a specific semantic content (outputs a high positive value). A discriminator can be obtained.

  In the present embodiment, the classifier learning unit 38 executes the first and second processes, but the classifier obtained by executing only the first process may be output as the image dictionary 50. Good.

  The above is an example of the processing details of each processing unit of the image dictionary construction device.

<Configuration of Image Representation Device According to First Embodiment of the Present Invention>

  Next, the configuration of the image expression device according to the first embodiment of the present invention will be described. As shown in FIG. 2, the image expression device 200 according to the first exemplary embodiment of the present invention includes a CPU, a RAM, a ROM that stores programs and various data for executing an image expression processing routine described later, and , Can be configured with a computer including. The image expression device 200 functionally includes an image database 210 and a calculation unit 220 as shown in FIG.

  The image database 210 stores at least an image itself or an address that uniquely indicates the location of the image file. Other configurations are the same as those of the image database 10 of the image dictionary configuration apparatus 100.

  The computing unit 220 includes a partial region dividing unit 230, a feature amount extracting unit 232, an expression unit 234, and an image dictionary 236.

  The partial area dividing unit 230 reads an image input from the image database 210, divides and selects each image into partial areas, and outputs this to the feature amount extracting unit 232. Other configurations are the same as those of the partial region dividing unit 30 of the image dictionary configuration apparatus 100.

  The feature quantity extraction unit 232 analyzes each partial area included in the set of partial areas composed of the partial areas of the image divided by the partial area division unit 230, and extracts a predetermined feature quantity. The feature amount is output to the expression unit 234. Other configurations are the same as those of the feature amount extraction unit 32 of the image dictionary configuration apparatus 100.

  Based on the feature amounts of the partial regions extracted by the feature amount extraction unit 232 and the image dictionary 236 output by the image dictionary construction device 100, the expression unit 234 determines that the partial region is The accuracy belonging to each cluster is calculated, and based on the calculated accuracy, it is determined whether the partial region belongs to any of the clusters or does not belong to any of the clusters. Based on the determination result, the expression unit 234 outputs, as an image representation of the input image, a histogram representing the frequency determined for each cluster as belonging to the cluster.

  Hereinafter, the processing of the expression unit 234 will be described in detail. Here, with respect to the input image, partial areas and feature amounts for the partial areas are extracted. Based on this, the image dictionary 236 learned in advance by the image dictionary construction device 100 is used to obtain an image representation for this image.

  First, the expression unit 234 performs a process similar to that performed by the classification unit 34 in the image dictionary construction apparatus 100, so that each partial area belongs to any one of “clusters” or “does not belong to any cluster”. Determine whether it corresponds.

Most simply, for each of the clusters, the frequency determined to belong to each cluster K _i (i = 1,..., V) is obtained, and this is used as an image representation by constructing a V-dimensional histogram. be able to.

  Alternatively, the frequency may be calculated by a process called Spatial Pyramid Matching (or Spatial Pooling) described in [Reference 9].

  [Reference 9] S. Lazebnik, C. Schmid, J. Ponce, “Beyond Bags of Features: Spatial Pyramid Matching for Recognizing Natural Scene Categories” In Proc. IEEE Conference on Computer Vision and Pattern Recognition. Pp.2169-2178, 2006.

  In addition to this, an image representation can be obtained by an arbitrary statistic that can be obtained from one or more clusters and a set of elements belonging to the clusters.

  Then, the expression unit 234 outputs a histogram representing the frequency determined to belong to the cluster for each cluster as an image expression, and ends the process.

<Operation of Image Dictionary Constructing Device According to First Embodiment of the Present Invention>

  Next, the operation of the image dictionary construction device 100 according to the first embodiment of the present invention will be described. When receiving one or more images and input of document data corresponding to each of the images from the image database 10, the image dictionary construction device 100 executes an image dictionary construction processing routine shown in FIG.

  First, in step S100, one or more images received from the image database 10 and document data corresponding to each of the images are read.

  In step S102, each of the images read in step S100 is divided into one or more partial areas.

  In step S104, each partial area of the partial area set composed of the partial areas divided in step S102 is analyzed, and an image feature vector is extracted as a feature amount for each partial area.

  In step S106, the partial region set including all the partial regions divided in step S102 is classified into one or more clusters based on the image feature vector extracted in step S104.

  In step S108, a cluster to be processed is selected from the cluster set obtained in step S106.

  In step S110, for the cluster selected in step S108, the partial area included in the set of partial areas classified into the cluster in step S106, and the document data corresponding to the image including the partial area read in step S100, and Associate.

  In step S112, a document vector is constructed for each partial area based on the document data associated with each partial area in step S110.

  In step S114, for the cluster selected in step S108, the cluster center is obtained by clustering the image feature vector based on the image feature vector of the partial region classified into the cluster in step S106, and based on the cluster center, A representative score for each of the partial areas is obtained. In addition, the cluster center is obtained by clustering the document vector based on the document vector configured in step S112 for each partial region classified into the cluster, and the representative score of each partial region is determined based on the cluster center. Ask for. Then, for each partial region classified into the cluster, the representative score of the partial region is obtained based on the representative score obtained by both.

  In step S116, for the cluster selected in step S108, a candidate area that is a partial area representing the cluster is determined based on the representative score of each partial area obtained in step S114.

  In step S118, for the cluster selected in step S108, the discriminator is learned using the candidate region determined in step S116 as a positive example and the partial region not belonging to the cluster as a negative example.

  In step S120, it is determined whether or not the processing of step S108 to step S118 has been completed for all the clusters. If not, the process returns to step S108 to select the cluster and repeat the processing. The process proceeds to step S122.

  In step S122, the classifiers learned for all clusters in step S118 are output as an image dictionary, and the process ends.

<Operation of Image Representation Device According to First Embodiment of the Present Invention>

  Next, the operation of the image expression device 200 according to the first embodiment of the present invention will be described. When an image input is received from the image database 210, the image expression device 200 executes an image expression processing routine shown in FIG.

  First, in step S200, an image received from the image database 210 is read.

  In step S202, the image read in step S200 is divided into one or more partial areas.

  In step S204, a feature amount is extracted for each partial area of the partial area set including the partial areas divided in step S202.

  In step S206, based on the feature amount of each partial area extracted in step S204 and the image dictionary 236 output by the above image dictionary construction processing routine, the partial area is a cluster of each partial area. The accuracy attributed to each is calculated.

  In step S208, based on the probability that each of the partial areas calculated in step S206 belongs to each of the clusters, it is determined whether each partial area belongs to any of the clusters or does not belong to any of the clusters. To do.

  In step S210, a histogram representing the frequency determined to belong to the cluster for each cluster is constructed based on the result of the determination in step S208.

  In step S212, the histogram formed in step S212 is output as an image expression, and the process ends.

  As described above, according to the image dictionary construction device according to the first exemplary embodiment of the present invention, the input image is divided into partial areas, each of the partial areas is classified into clusters, and each of the clusters is classified. A candidate area that is a representative partial area of the cluster is determined based on the image feature vector of the partial area and the document data corresponding to the entire image including the partial area, and the classifier is used as a positive example By learning the above, it is possible to construct an image dictionary for obtaining an image expression capable of finding a meaningful characteristic region in the image.

  In addition, according to the image representation device according to the first exemplary embodiment of the present invention, the input image is divided into partial areas, and the partial area is determined based on the feature amounts of the partial areas and the image dictionary. For each, it was determined whether it belongs to any of the clusters or not to any of the clusters, and a histogram representing the frequency determined to belong to the cluster was input for each of the clusters based on the determination result. By outputting the image as an image representation of the image, an image representation capable of finding a meaningful characteristic area in the image can be obtained.

<Configuration of Image Dictionary Configuration Device According to Second Embodiment of the Present Invention>

  Next, the configuration of the image dictionary configuration device according to the second embodiment of the present invention will be described. In addition, about the part which becomes the structure similar to the image dictionary structure apparatus 100 of 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  As shown in FIG. 1, the image dictionary construction apparatus 100 according to the second embodiment of the present invention stores a CPU, a RAM, a program for executing an image dictionary construction processing routine described later, and various data. And a computer including a ROM. Functionally, the image dictionary construction device 100 includes an image database 10, a calculation unit 20, and an image dictionary 50 as shown in FIG.

  It is assumed that the image database 10 according to the second embodiment stores an image itself or an address that uniquely indicates the location of the image file. Further, it is assumed that document data corresponding to one or more images among the stored images is stored. This document data represents the semantic content related to a partial area of the image.

  Further, it is assumed that the contents of the partial area of the image described in the document data are described along with the position information about which area of the image is described. For example, for a section surrounded by a rectangle in a partial area of the image, it is sufficient if the vertical and horizontal pixel positions and the positions of four points of width and height are given as position information. In this case, since the specificity is increased as to what semantic content is included in the image, it is possible to obtain an image dictionary and an image expression that represent the semantic content more precisely.

  The computing unit 20 according to the second embodiment includes a partial region dividing unit 30, a feature amount extracting unit 32, a classifying unit 34, a candidate region determining unit 36, and a classifier learning unit 38. ing.

  The candidate area determination unit 36 according to the second embodiment receives the classification result output from the classification unit 34, and for each of the clusters, the part for each of the partial areas classified into the cluster by the classification unit 34 Based on the feature amount of the area and the document data corresponding to the partial area of the image including the partial area stored in the image database 10, a candidate area that is a partial area representing the cluster is determined.

  The candidate area determination unit 36 according to the second embodiment first takes the correspondence between the partial area and the document data as the first process. As described above, the image database 10 stores an image (or an address that uniquely indicates the location) and document data that represents semantic content regarding a partial region of the image in association with each other. Therefore, by viewing from which image the partial area is extracted, the partial area can be associated with the document data associated with the image from which the partial area has been extracted. In the second embodiment, the document data describes a partial area of the image, and the positional information of the area is also stored. Based on this, the direct association between the partial area and the document data is performed. I do. For example, association can be performed using the overlapping ratio of regions. In other words, the ratio of the size of the area obtained by the sum to the size of the area obtained by the product of the ratio of overlapping areas (area to which document data is allocated) and (partial area) is equal to or greater than a threshold (for example, 0.5). In such a case, the document data may be associated with the partial area. For example, it is assumed that certain document data is allocated to an image position (36 pixels wide and 56 pixels high) with a size (18 pixels wide and 24 pixels high). For example, suppose that an image area is taken at a position (width 20 pixels, height 20 pixels) at a position (width 40 pixels, height 60 pixels). At this time, the overlapping ratio is (36 + 18-40) × 20 / (18 × 24 + 20 × 20− (36 + 18-40) × 20) = 0.51. If the threshold is set to 0.5, the document data is associated with this partial area. The second to fourth processes after the first process in the candidate area determination unit 36 according to the second embodiment are the second to fourth processes in the candidate area determination unit 36 according to the first embodiment. This is the same as the process.

  The other configuration and operation of the image dictionary configuration apparatus according to the second embodiment are the same as those of the image dictionary configuration apparatus 100 according to the first embodiment, and thus detailed description thereof is omitted.

<Configuration of Image Representation Device According to Second Embodiment of the Present Invention>

  Next, the configuration of the image expression device according to the second embodiment of the present invention will be described. In addition, about the part which becomes the structure similar to the image representation apparatus 200 of 1st Embodiment, the same code | symbol is attached | subjected and description is abbreviate | omitted.

  As shown in FIG. 2, the image expression device 200 according to the second embodiment of the present invention includes a CPU, a RAM, and a ROM that stores programs and various data for executing an image expression processing routine to be described later. And a computer including The image expression device 200 functionally includes an image database 210 and a calculation unit 220 as shown in FIG.

  Note that other configurations and operations of the image expression device according to the second embodiment are the same as those of the image expression device 200 according to the first embodiment, and thus detailed description thereof is omitted.

  As described above, according to the image dictionary construction device according to the second exemplary embodiment of the present invention, an input image is divided into partial areas, each of the partial areas is classified into clusters, and each of the clusters is classified. Based on the image feature vector of the partial area and the document data corresponding to the partial area of the image including the partial area, a candidate area that is a partial area representing the cluster is determined, and the candidate area is used as a positive example. By learning the discriminator, it is possible to construct an image dictionary for obtaining an image expression capable of finding a meaningful characteristic area in the image.

  In addition, according to the image expression device according to the second exemplary embodiment of the present invention, the input image is divided into partial areas, and based on the feature amount of each partial area and the image dictionary, the partial area For each, it was determined whether it belongs to any of the clusters or not to any of the clusters, and a histogram representing the frequency determined to belong to the cluster was input for each of the clusters based on the determination result. By outputting the image as an image representation of the image, an image representation capable of finding a meaningful characteristic area in the image can be obtained.

  Further, according to the image dictionary construction device in the above-described embodiment, the semantic content of the subject or the scene is obtained by using, as reference information, document data indicating not only the image feature but also the semantic content accompanying the image feature. Can be found, and an image dictionary can be constructed based on this.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made without departing from the gist of the present invention.

  For example, in the first embodiment described above, the document data corresponding to the entire image is used, and in the second embodiment, the document data corresponding to the partial area of the image is used. However, the present invention is not limited to this, and document data corresponding to the entire image and a partial area may be used. Here, when both document data corresponding to the entire image and document data corresponding to the partial area exist as document data corresponding to the partial area, the document data corresponding to the partial area is given priority. A document vector may be configured, or a document vector may be configured for document data that is a combination of document data corresponding to the entire image and document data corresponding to a partial area.

10, 210 Image database 20, 220 Arithmetic unit 30, 230 Partial region division unit 32, 232 Feature amount extraction unit 34 Classification unit 36 Candidate region determination unit 38 Discriminator learning unit 50, 236 Image dictionary 100 Image dictionary construction device 200 Image representation Device 234 Expression unit 250 Output unit

Claims (5)

Each of the input one or more images and a document corresponding to each of the images, including a partial region dividing unit, a feature amount extracting unit, a classifying unit, a candidate region determining unit, and a classifier learning unit An image dictionary construction method in an image dictionary construction device for constructing an image dictionary from data,
The partial region dividing unit dividing each of the inputted one or more images into one or more partial regions;
The feature amount extracting unit extracting a feature amount for each of the partial regions included in a set of partial regions formed by the partial regions divided by the partial region dividing unit;
The classification unit selects each of the partial regions of the set of partial regions from one or more clusters based on the similarity regarding the feature amount of each of the partial regions extracted by the feature amount extraction unit. Categorizing into a cluster of
For each of the clusters, the candidate region determination unit corresponds to the feature amount of the partial region and the input image including the partial region for each of the partial regions classified into the cluster by the classification unit. Determining a candidate area that is a partial area representing the cluster based on the document data;
For each of the clusters, the classifier learning unit uses the feature amount of the candidate region determined by the candidate region determination unit as a positive example, and sets the feature amount of the partial region not classified into the cluster as a negative example. Learning and obtaining a discriminator for identifying whether or not the partial region belongs to the cluster, and outputting the discriminator obtained for each of the clusters as an image dictionary;
An image dictionary construction method including: Each of one or more images received as input, and an image dictionary configuration device that configures an image dictionary from document data corresponding to each of the images,
A partial area dividing unit that divides each of the input one or more images into one or more partial areas;
A feature amount extraction unit that extracts a feature amount for each of the partial regions included in the set of partial regions formed by the partial regions divided by the partial region dividing unit;
Each of the partial areas of the set of partial areas is classified into one of one or more clusters based on the similarity regarding the feature quantities of the partial areas extracted by the feature quantity extraction unit. A classification section;
For each of the clusters, based on the feature amount of the partial region for each of the partial regions classified into clusters by the classification unit, and the input document data corresponding to the image including the partial region, A candidate area determination unit that determines a candidate area that is a partial area representing the cluster;
For each of the clusters, the partial region is the cluster, with the feature amount of the candidate region determined by the candidate region determination unit as a positive example and the feature amount of the partial region not classified into the cluster as a negative example. A classifier learning unit that learns and acquires a classifier for identifying whether or not it belongs to, and outputs the classifier acquired for each of the clusters as an image dictionary;
An image dictionary construction apparatus including: An image expression method in an image expression device including a partial region dividing unit, a feature amount extraction unit, and an expression unit,
The partial region dividing unit divides the input image into one or more partial regions;
The feature amount extracting unit extracting a feature amount for each of the partial regions;
The each of the partial regions based on the feature amount of each of the partial regions extracted by the feature amount extraction unit and the image dictionary output by the image dictionary construction method according to claim 1. , Calculating the accuracy that the partial region belongs to each of the clusters, and based on the calculated accuracy, whether the partial region belongs to any of the clusters or does not belong to any of the clusters Determining and outputting, as an image representation of the input image, a histogram representing the frequency determined to belong to the cluster for each of the clusters based on the determination result;
An image representation method including: A partial area dividing unit that divides the input image into one or more partial areas;
A feature amount extraction unit that extracts a feature amount for each of the partial regions;
The partial area is determined for each of the partial areas based on the feature quantities of the partial areas extracted by the feature quantity extraction unit and the image dictionary output by the image dictionary construction device according to claim 2. Calculating the accuracy belonging to each of the clusters, and determining whether the partial region belongs to any of the clusters or does not belong to any of the clusters based on the calculated accuracy; An expression unit that outputs a histogram representing the frequency determined to belong to the cluster for each of the clusters as an image representation of the input image;
An image expression apparatus including:   The program for making a computer perform each step which comprises the image dictionary construction method of Claim 1, or the image expression method of Claim 3.

Images