JP5836779B2 - Image processing method, image processing apparatus, imaging apparatus, and program - Google Patents

Description

  The present invention relates to an image processing method, an image processing apparatus, an imaging apparatus, and a program.

  Conventionally, there is a method as described in Non-Patent Document 1 as a method of dividing an image for each subject. In the method of Non-Patent Document 1, feature quantities such as textures and pixel colors are stored for each category of each subject. Next, the likelihood of which category each pixel belongs to is obtained based on the feature amount, and region division is performed by determining the attribution of the pixel while considering the consistency between the pixels.

Pushmeet Kohli, L'ubor Ladicky and Philip H. et al. S. Torr, Int. J. et al. of Computer Vision, 82 (3), 302-324, 2009. Jitendra Malik, Serge Belongie, Thomas Leung and Jianbo Shi, Int. J. et al. of Computer Vision, 43 (1), 7-27, 2001. A. Rabinovich, T .; Lange, J .; Buhmann, S.M. Belongie, IEEE Conf. on ComputerVision and Pattern Recognition, 2006. J. et al. Carreira, C.I. Sminchiscu, IEEE Conf. on Computer Vision and Pattern Recognition, 2010. A. Gionis, P.M. Indyk, R.A. Motwani, Proc. of the 25th Very Large Database Conf. 1999. J. et al. Sivic, B.M. C. Russell, A.R. A. Efros, A.E. Zisserman, and W.C. T. T. et al. Freeman. In Proc. of 10th IEEE Int. Conf. on Computer Vision, 2005. A. Saxena, S .; H. Chung, A.D. Y. Ng, Int. J. et al. of Computer Vision, 76 (1), 53-69, 2008. J. et al. Sivic, A.M. Zisserman, Proc. of the 9th IEEE Int. Conf. on Computer Vision, 2003. O. Chum, M.M. Perdoc, J. et al. Matas, IEEE Conf. on Computer Vision and Pattern Recognition, 2009. B. C. Russell, W.M. T. T. et al. Freeman, A.M. A. Efros, J. et al. Sivic, A.M. Zisserman, Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2006. D. Blei, A.D. Ng, and M.M. Jordan. , J. et al. of Machine Learning Research, 3: 993.1022, 2003. G. Kim, A.A. Torralba, Annual Conf. on Neural Information Processing Systems, 2009. G. Wang, Y.W. Zhang. L. Fei-Fei, Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition, 2006. K. Barnard, D.D. Forsyth, Proc. of 8th IEEE Int. Conf. on Computer Vision, 2001.

  In the method of Non-Patent Document 1, it is necessary to prepare a plurality of case images and teacher data indicating to which subject each pixel in the case image belongs and to learn the features of the subject in advance. For this reason, the categories of subjects that can be divided into regions are limited, and a desired division result may not be obtained for an unknown subject.

  As a method that does not limit the subject, a method such as Non-Patent Document 2 in which region segmentation is performed based on similarity of local features or the like has been studied. However, this type of method has an unavoidable problem that when the subject is composed of a plurality of portions with different appearances, the subject is excessively divided into a plurality of portions.

  The present invention provides a technique that can divide an input image into different regions according to the type of subject without giving instructor data of the subject in advance.

An image processing device according to one aspect of the present invention that achieves the above object is an image processing device that divides an input image into regions for each subject in the input image,
Dividing means for dividing the input image into candidate region groups representing candidates for a plurality of regions for each subject;
Acquisition means for acquiring a similar image group similar to each candidate area of the candidate area group from an image database;
Extraction means for extracting features common to similar images from the similar image group as feature components of candidate regions;
Comparing the feature component with the feature amount of each pixel of the input image, identifying which candidate region each pixel belongs to a candidate region group, and outputting a region identification unit as a result of region division; It is characterized by providing.

  According to the present invention, it is possible to divide an input image into different areas according to the type of the subject without providing the subject's teacher data in advance.

1 is a diagram illustrating a configuration of an image processing apparatus according to a first embodiment. The figure explaining the concept of the extraction operation | movement of a region component. The figure explaining the comparison with the extraction operation of the area | region component of a conventional method, and the extraction operation | movement of the area | region component in embodiment. The figure explaining operation | movement of an area | region component extraction part. The figure explaining the concept of extraction operation | movement of a similar area | region. The figure explaining operation | movement of an area | region identification part. The figure explaining operation | movement of a similar image acquisition part. The figure explaining the further detailed operation | movement of a similar image acquisition part. The figure explaining the structure of the image processing apparatus of 2nd Embodiment. The figure explaining the operation | movement of the whole image processing apparatus of 2nd Embodiment. The figure explaining the structure of the image processing apparatus of 3rd Embodiment. The figure explaining operation | movement of an area | region label identification part. The figure explaining the structure of the image processing apparatus of 5th Embodiment. FIG. 10 is a diagram for explaining the overall operation of an image processing apparatus according to a fifth embodiment.

(First embodiment)
Hereinafter, an image processing method (an image region dividing method) and an image processing apparatus 10 according to an embodiment of the present invention will be described in detail with reference to the drawings. The imaging device 20 includes an imaging unit that captures an image and an image processing device 10 that uses the image captured by the imaging unit as an input image and divides the input image into regions for each subject in the input image. Note that the same reference numerals in the drawings perform the same operation, and the description thereof is omitted. An image processing method and an image processing apparatus according to an embodiment of the present invention relate to an image processing technique for appropriately dividing an input image into a plurality of regions according to a subject. Appropriately dividing the subject facilitates many image processes in subject recognition, scene recognition, image quality correction according to the subject, and the like. The input image here may be a still image or a moving image. The subject includes all objects such as living things such as people and dogs, artificial objects such as buildings and tools, and natural objects such as mountains and the sky.

  The operation of the image processing apparatus 10 according to the embodiment of the present invention will be described with reference to FIG. The area dividing unit 101 divides the input image into a plurality of candidate area groups representing a plurality of area candidates for each subject. Next, the similar image acquisition unit 102 acquires a plurality of similar images (similar image group) for each region of the candidate region group from the image database 103. The region component extraction unit 104 extracts K components as feature components common to similar images from the similar image group as feature components (region components) of candidate regions. Each region component is a component related to a main subject in the input image. The region identification unit 105 compares the information of the feature component (region component) with the feature amount of each pixel of the input image. Then, the region identification unit 105 determines which candidate region of each of the K region components each pixel belongs from the comparison result, performs labeling, and outputs the labeling result as a result of region division for each subject. . In this way, the area division result of the input image is finally obtained.

  The region dividing method according to the present invention is partially related to the method disclosed in Non-Patent Document 10. Here, the description of the common points and functional differences between the present method and Non-Patent Document 10 is effective in understanding the effects of the present method, and will be described in detail below.

  The schematic diagram of FIG. 3 conceptually explains the difference between the two methods. FIG. 3A shows the operation of Non-Patent Document 10. In the method of Non-Patent Document 10, the main component of the region is extracted for all image groups in the image database. In this method, since common components of all the images are extracted without giving importance to a specific image, components of subjects that generally appear widely in the image such as “sky” and “ground” are extracted as higher components. On the other hand, a subject such as “cow” that appears only in a relatively small number of images and has a change in appearance depending on the posture of the subject may not be extracted properly. The subject component as described above may be ignored as a surplus component, or may be extracted by mixing with other subject components. This can be understood similarly to the fact that a lot of noise is included in the lower component in the principal component analysis.

  On the other hand, in the area dividing method according to the present invention, as shown in FIG. 3B, the candidate area of the subject is extracted by dividing the area of one input image. Next, an area similar to the candidate area is acquired from the image database, and an image set (similar image group) limited to only images related to the input image is generated. The next region component extraction unit 104 can apply a method similar to the method of Non-Patent Document 10 to an image set (similar image group). However, since the image set (similar image group) is limited, it is possible to extract only the components related to the subject area of the input image with higher accuracy. The above is the difference between the method of Non-Patent Document 10 and this method.

(Area division unit 101)
Next, the operation of the area dividing unit 101 according to the embodiment of the present invention will be described. The area dividing unit 101 extracts a plurality of areas from the input image and sets them as candidates for the area of the subject in the image. The area dividing unit 101 can extract a plurality of areas that can be considered as a group in the image from the input image. The region extraction at this stage is performed by using the similarity of the local feature amount of the image such as the color of the pixel, the texture, and the position of the pixel as a clue. In many areas, it is conceivable that only a part of the subject is cut out or a plurality of subjects are included in the area. In consideration of such a failure in clipping, the region dividing unit 101 extracts a sufficient number of candidate regions for variations. Note that the candidate area allows multiple cutouts. That is, it is allowed to have a partial overlap with each other.

  As a specific method of the region dividing unit 101 satisfying such a function, there are a method of clustering as in Non-Patent Document 3 and a method of using a graph cut as in Non-Patent Document 4. The above-described method has controllable parameters such as the number of clusters to be divided and the position of the center pixel for performing the graph cut. In the above method, a plurality of candidate regions can be obtained by changing the control parameter and performing a plurality of region divisions.

(Similar image acquisition unit 102)
Next, the operation of the similar image acquisition unit 102 will be described. The similar image acquisition unit 102 acquires an image (similar image) including a region similar to the candidate region in the input image obtained in the previous process. Acquisition of similar images is performed for each candidate area, and a plurality of search result images are sent as a similar image group to subsequent processing.

  Similar image retrieval techniques are well studied in a field called image retrieval. For example, Non-Patent Document 8 and Non-Patent Document 9 describe a technique for searching a partial image having a unique appearance such as a logo mark from an image database at high speed. Although the above method may be used for the similar image acquisition unit 102, the above method may be vulnerable to changes in appearance such as a change in the posture of an object in the region. Therefore, in the present embodiment, the similar region is acquired more flexibly by using a method as described below.

  The specific operation of the similar image acquisition unit 102 will be described below with reference to the flowchart of FIG. First, the similar image acquisition unit 102 calculates the feature amount of each candidate region by a predetermined method (S701, S702). Next, similar data is searched from the image database 103 as inquiry data for searching for similar regions using this feature amount, and a similar region (region sample) is acquired (S703). A specific method for searching for similar regions will be described later. The similar image acquisition unit 102 ranks (sorts) the similar regions (region samples) in descending order of similarity (similarity) from the search result (S704). Next, images of similar regions (region samples) of higher rank are output as a set of similar images (S705). Images to be output are limited to those of a predetermined rank or higher. Alternatively, the similarity (similarity) score may be equal to or higher than a predetermined threshold, or the similarity (similarity) score and rank may be satisfied.

  Various features are conceivable as the region feature amount. Features used in past image recognition techniques can be used. For example, a SIFT (Scale Invariant Feature Transform) feature is calculated from a plurality of feature points in the region, and a histogram feature amount converted by the Bag-of-words method can be used. Further, for example, the position of the region in the image, the second moment representing the shape of the region, the color histogram of the pixels in the region, and the like can be used. Also, for example, HOG feature values, LBP feature values, and the like can be used. In addition, for example, a higher-order feature value obtained by concatenating a plurality of feature value vectors can be used. Note that since the shape of the candidate region is generally indefinite, when calculating the HOG feature value, the LBP feature value, etc., it is necessary to cut out the image for each peripheral part with a rectangle circumscribing the region and convert it into a square image. Details regarding each feature amount as described above have been widely disclosed in many previous studies such as Non-Patent Document 6 and Non-Patent Document 7, and the details will be given to them.

(Similar area search method (S703))
Next, details of a method for searching for similar regions will be described. Assume that the image database 103 stores in advance a set of sample images in the order of 100,000 or million. Therefore, here, a hash, which is a general technique for high-speed neighborhood search, is used. For example, a technique called local sensitive hash (LSH) disclosed in Non-Patent Document 5 is specifically used.

  LSH registers data offline for high-speed search. First, an image group in the image database 103 is divided into regions in advance by the same method as the region dividing unit 101. Further, a feature amount is calculated for each divided area. Next, the feature quantity is converted using a predetermined function to obtain a variable having a lower dimension than the original feature quantity called a hash key. Data of each area is registered using this hash key as an address, and a table called a hash table is created. Similar data samples are collected and registered on one address of the hash table. The above processing is the operation when registering data in the image database 103.

  Next, the operation at the time of searching will be described with reference to FIG. FIG. 8 is a more detailed flowchart of the similar image acquisition unit 102 described above with reference to FIG. First, a candidate area used for a query for searching for a similar area is converted into a feature amount (S801, S802), and further converted into a hash key (S803). Next, the hash table is referenced from the hash key. By referring to the entry of the corresponding address, if there is registered data, it is acquired as a similar area (S804, S805). If it is determined in step S805 that no similar region exists (S805-No), the process proceeds to step S812. On the other hand, if it is determined in step S805 that a similar region exists (S805-Yes), the process proceeds to step S806, and a high-dimensional feature value is obtained for each similar region that is the original before being converted into a hash key. (S806, S807). Based on the feature amount, an accurate distance between the query candidate region and the similar region is calculated (S808). This process is repeated for all similar regions. Then, the calculated distances are compared, the samples (similar areas) are sorted in order of increasing distance (S810), and similar areas that are closer than the predetermined threshold are output (S811). Here, as the distance, an Euclidean distance, a chi-square distance, or the like may be used. This process is repeated for all candidate areas (S812). The above is the detailed operation of the similar image acquisition unit 102. The design of a high-precision hash function for neighborhood search has been extensively studied, and details are given in Non-Patent Document 5.

  As another derivative form of the similar image acquisition unit 102, a form using a graph structure of image similarity may be considered. As a result, it can be expected that similar images can be inquired more flexibly. Here, the graph is a graph represented by a similarity matrix having each region as a node and the similarity between the regions as an edge. A specific operation example is as follows, for example. First, the neighborhood area is detected by hashing as described above. Next, another data sample closer to the neighboring region is searched using the similarity matrix between the regions. For each data sample, the distance from the candidate region is evaluated with a high-order feature amount. By such a device, it is possible to inquire similar images while considering the mutual relationship as a variety of images that are difficult to reflect only by hashing.

  As another derivative form of the image database 103, a form using an image search server installed on the Internet is also conceivable. This image search server has a function of operating an agent on a web service network, automatically collecting new images on the network, and adding them to a database. The similar image acquisition unit 102 and the region division unit 101 using the image database 103 having such a form can be expected to operate correctly even if a new subject is included in a given input image. For example, consider a situation in which an object with an unknown specific shape appears in the input image, such as a new product of a certain home appliance. Even if the user does not have any knowledge about this subject, it is possible to correctly divide the area if a sufficient number of image information of the product is posted on the web and collected in the database. I can expect that.

(Regional component extraction unit 104)
The operation of the region component extraction unit 104 will be described. The purpose of the operation of this module is to extract the components of the region from the set of similar images (similar image group) obtained by the similar image acquisition unit 102 in the previous stage.

In the present embodiment, what is called a component of the region will be specifically described with reference to FIG. FIG. 2 (a) shows several extracted regions relating to grassland in the image. Each grassland example image has a slight variation in appearance. The area also contains extra elements such as cow limbs. Each area in FIG. 2A is converted into a feature amount as shown in FIG. Here, an example in which the SIFT feature value and the Bag-of-words method described above are used for conversion to a histogram feature value is illustrated. It should be noted that _w 1 ··· _{w V} in the figure is the distribution of the code book. A main component as shown in FIG. 2C, which is common to the feature amounts, is extracted from the feature amount set, and is used as the overall components of the regions 1, 2, and 3. The components of the region are expressed on the codebook distribution by separating only the features common to the grassland.

  There are a plurality of methods for such component extraction. For example, the method of Latent Dilatation (LDA) of Non-Patent Document 10 can be used. In addition to LDA, models using the Dirichlet process have been well studied (Non-patent Document 13). Here, the description will focus on LDA having a certain evaluation with high accuracy of component extraction, but it should be noted that the embodiment according to the present invention does not particularly limit the method of component extraction to LDA.

  LDA is one of document generation models widely used for natural language processing, and is represented by the following equation (1).

The LDA will be briefly described below. The above expression represents a probability model when a document including K topics (topics) generates N vocabularies. Here, w represents a sequence of the generated N vocabulary w ₁ ,..., W _N. Among the K topics, each topic Z _k is a probability distribution that determines the generation probability of which vocabulary w ₁ ,..., W _V is generated. V is the total number of possible vocabularies. Z _k is generated from a multinomial distribution with β _k as a hyperparameter. z is a sequence of _N probability distributions z ₁ ,..., z _N. z _n is a distribution composed of a mixture of _K topics Z ₁ ,..., Z _K and is involved in the generation of the nth vocabulary. The mixing ratio is determined by the parameter θ, and θ is generated from the Dirichlet distribution with α as a hyperparameter. During learning, hyperparameters are learned by the variational method using the given case data.

By setting the topic _Zk as a latent variable of the feature amount of the image and placing the Bag-of-words codebook as the vocabulary w ₁ ,..., W _V , the LDA can be applied to image recognition. Each topic Z _k extracted by the LDA becomes a probability distribution in which each code book w ₁ ,..., W _V occurs. Since LDA is disclosed in Non-Patent Document 11 and details of its application to image recognition are disclosed in Non-Patent Document 10 and the like, details will be given to that.

  Next, the detailed operation of the region component extraction unit 104 to which LDA is applied will be described based on the flowchart of FIG. First, the region component extraction unit 104 receives a similar region group and a similar image group including a similar region from the similar image acquisition unit 102. Next, the region component extraction unit 104 reads the region division results for all similar images (S401, S402). The result of this area division is the same as that used when the similar image acquisition unit 102 is operated. A plurality of division results are obtained by changing the control parameters and stored in the image database 103. Next, an area that overlaps the similar area is extracted from all the read divided areas (S403, S404, and S405).

  The step of extracting a region that overlaps the similar region (S403, S404, S405) is important for improving the quality of the similar region, and is particularly schematically illustrated in FIG. FIG. 5 shows a candidate area 501 in the input image and a similar area 502 extracted as related to the candidate area 501. Also, regions 503a, 503b, and 503c that overlap the similar region 502 are shown. A region overlapping with a part of the similar region 502 is also extracted together with the similar region. By extracting the overlapping region 503 (503a, 503b, 503c) as a data sample, even if the similar region 502 is limited to a part of the subject, more suitable data for extracting the region component is obtained. I can expect that.

  Returning to FIG. 4, the processing from step S401 to step S406 is repeatedly executed for all similar images.

Next, the region component extraction unit 104 reads out the feature values of all the regions obtained in steps S403 to S405 from the image database 103, and executes LDA as the above-described method using this as a data sample (S407). As a result of the LDA, K topics Z ₁ ,..., Z _K are obtained and set as region components (S408). Here, although determined a predetermined number K as the number of domain components may delete some previously determined large numbers K, parameters of the mixing ratio theta _k is smaller topics. In that case, a variable number of topics of K or less can be obtained.

(Region identification unit 105)
Next, the operation of the area identification unit 105 will be described in detail. K region components Z ₁ ,..., Z _K extracted by the region component extraction unit 104 are sent to the region identification unit 105. Next, the region identification unit 105 identifies the category of each pixel of the input image based on this region component.

A specific operation flow will be described with reference to FIG. For each pixel in the input image, region division is performed around the pixel (S601 to S602). The feature amount of Bag-of-words of each region is calculated (S603). A Cullback-Lailer distance (KL distance) between this feature amount and K region components Z ₁ ,..., Z _K is obtained. The cullback / librarian distance is calculated for all K region components (S604 to S606).

In the determination in step S607, it is determined whether or not the minimum cullback / liver distance (KL distance) out of K is equal to or less than a threshold value. If it is determined in step S607 that the minimum cullback / librarian distance is not less than or equal to the predetermined threshold (S607-No), the process proceeds to step S608, and the target pixel is labeled with the area component of the minimum cullback / librar distance. . The k-th label is assigned assuming that the pixel of interest belongs to the closest region component Z _k (S608). On the other hand, if it is determined in step S607 that the minimum cullback / librarian distance is equal to or smaller than the predetermined threshold value (S607-Yes), the process proceeds to step 609 to set the target pixel as a blank area. A label is not assigned as an area that does not belong to any pixel whose Cullback-Lailer distance is equal to or less than a predetermined threshold (S609). Therefore, the number of types of labels here is K + 1 at the maximum including the area where the attribution is undetermined. The processing from step S601 to step S610 is repeated for all the pixels of the input image, and the final labeling result is output as a region division result (S611). It should be noted that here, the label given to the area is only the type number, and no semantic label such as “cow” or “grass” is identified.
By performing the processing as described above, it is possible to divide the input image into different regions according to the type of the subject without using the subject teacher data.

(Second Embodiment)
As a second embodiment according to the present invention, an extended method capable of further improving the accuracy of area division performed by the first embodiment will be described. FIG. 9 shows a block diagram of the image processing apparatus 10 of the present embodiment. The processing elements having the same reference numerals as those in FIG. In the configuration shown in FIG. 9, the processing from the region dividing unit 101 is repeated until the region division result of the region identifying unit 105 does not change (until the predetermined change rate or less). In this respect, the configuration of FIG. 9 is different from the configuration of FIG. 1 in the first embodiment.

  The operation of the image processing apparatus 10 of this embodiment will be described in detail based on the overall flow diagram of this embodiment of FIG. First, processing similar to that of the first embodiment is performed, and a region division result is obtained from the input image. The result of area division output from the area identification unit 105 is acquired as a candidate area group (S1001). Next, the region dividing unit 101 (second dividing unit) applies the graph cut algorithm to each candidate region obtained as a result of the region division acquired in the acquisition step of step S1001. Specifically, the pixels of each region and the pixels of the other regions are given as initial values of the foreground and the background, respectively, and the graph cut algorithm is applied. At this time, the parameters of the graph cut algorithm are changed to obtain a plurality of region division results (S1002 to S1006). The area dividing unit 101 (second dividing unit) sets the candidate area group thus obtained as a new candidate area group (second candidate area group). The new candidate area group (second candidate area group) is searched again for similar areas by the same process as in the first embodiment, and the area division process is performed (S1007).

  In step S1007, a similar image group similar to each candidate area of the new candidate area group is acquired from the image database. Features common to similar images are extracted from the similar image group as feature components of candidate regions. The extracted feature component is compared with the feature value of each pixel of the input image, and each pixel is labeled to indicate which candidate region the new candidate region group belongs to, and the labeling result is obtained for each subject. The result is output as a result of area division (result of second area division).

  In step S1008, it is determined whether or not the second region division result has converged with respect to the region division result. When it is determined that the result of the second area division has converged, the area identification step outputs the result of the second area division as a result of the area division for each subject. When it is determined that the result of the second region division has not converged, the above processing is repeated until the result of the region division does not change (for example, until it becomes a predetermined change rate or less) (S1008). If the above-described device is performed, it can be expected that the accuracy of the region division result is improved every time the iterative process proceeds.

  Although a method of generating a new candidate region group from the first region division result by performing graph cut is shown here, the method of generating the candidate region group is not limited to this. Another embodiment may be as follows. For example, the same processing as in the first embodiment is performed to obtain the first division result. Next, in the first candidate area group, only candidate areas that overlap the first division result with a predetermined ratio or more are extracted. This can be used as a second candidate area group, and the area can be divided again.

(Third embodiment)
As a third embodiment according to the present invention, a method for identifying subject names (subject recognition) such as “cow” and “grass” based on the result of area division performed by the first embodiment will be described. The subject recognition method in this embodiment is an extended method of the first embodiment. FIG. 11 shows a block diagram of the image processing apparatus 10 of the present embodiment. The image processing apparatus 10 illustrated in FIG. 11 further includes a region label identification unit 106 in addition to the configuration of the block diagram illustrated in FIG. The region label identifying unit 106 sets an annotation for the result of region segmentation by the region identifying unit 105 using the tag associated with each image stored in the image database, and recognizes the subject in the input image. Output as a result. Assume that the image database 103 in FIG. 11 stores a tag associated with each image. It is assumed that tags are extracted from related information such as web page headings when images are automatically collected from web pages, or that tags are manually added like image databases such as Flickr. .

  An operation flow of the area label identification unit 106 of the image processing apparatus 10 according to the present embodiment will be described with reference to FIG. First, it is assumed that K candidate area categories and area components have already been obtained (S1201). Next, the case data (similar image) of each area is obtained for each category. For this purpose, first, a hash key is created from the feature amount of the region component (S1202). This may be the same method as described in the first embodiment. Next, a similar area group is obtained from the hash table (S1203). Next, all the tags of the image including the similar region group are read from the image database 103 (S1204). Next, a vocabulary having a high appearance frequency is extracted from the tag set, and this is used as an annotation result (S1205). Note that the method based on the appearance frequency is the simplest method of vocabulary extraction, but if you consider the ontological distance between the vocabulary or consider the rareness of the vocabulary appearance frequency, the annotation with higher accuracy can be used. It can be performed. A method for setting such an annotation has been studied for a long time, and in the present embodiment, the realization means is not particularly limited to one. Here, a prior method such as Non-Patent Document 14 is cited as a reference method, and detailed description is omitted.

(Fourth embodiment)
As a fourth embodiment according to the present invention, another derivative form of the operations of the region component extraction unit 104 and the region identification unit 105 described in the first embodiment will be described. This embodiment is partially the same as the first embodiment, and a mode in which only the operations of the region component extraction unit 104 and the region identification unit 105 are different will be described. Therefore, here, only the operations of the region component extraction unit 104 and the region identification unit 105 will be described while avoiding repeated description.

  In the present embodiment, a graph structure of image similarity is used in the operations of the region component extraction unit 104 and the region identification unit 105. Non-Patent Document 12 discloses a technique called link analysis, which is an example of a method applicable to this purpose. This method is the same kind of algorithm as a method used when a search engine on the Internet extracts a web page on which links are concentrated as a highly important page.

  The outline of Non-Patent Document 12 will be described below. In this method, a plurality of regions are cut out from a database image group by a predetermined method. Each of the cut out regions is regarded as a node, and a graph is created in which the similarity between the regions is an edge between the nodes. Next, a node that is the core of this graph structure is found out and used as a representative case image of a subject included in the image database. Such a node is called a hub. There may be a plurality of hubs.

  Next, the detail of the form which applied the method of the nonpatent literature 12 to the area | region component extraction part 104 is demonstrated. The region component extraction unit 104 first cuts out a region group from the similar image group obtained in the preceding process. The method of cutting out the region group may be the same method as the region dividing unit 101 or a method using superpixels as described in Non-Patent Document 12. Next, a region is defined as a node, and a graph is created by obtaining the distance of the feature amount between the regions. Next, K nodes that are the hub of the graph are extracted. K hubs can be considered as representative case images of K categories.

  Next, the region identification unit 105 will be described. The area identifying unit 105 generates one area for each pixel of the input image by the same method as described in the first embodiment. A similar region most similar to this region is extracted from the image database 103. Next, it is determined which of the K categories the similar region belongs to. Specifically, the geodesic distance from the similar region to each hub is obtained, and the pixel of interest is assigned to the closest hub category. Here, it is also possible to take out a predetermined plurality of similar regions instead of one and determine category attribution by voting.

  The above is the description of the embodiment of the region segmentation method using the network structure of image similarity. In this embodiment, by using a graph of similarity between regions for extracting region components, it is possible to extract region components while more considering the properties of the image manifold.

(Fifth embodiment)
As another form of the image processing method (region division method) according to the present invention, a method for dividing a candidate region based on a component of a region category obtained by any means will be described. In the first embodiment, the component of the region is obtained based on the similar image. However, in the present embodiment, the configuration for obtaining the component of the region is not limited to this. In the present embodiment, another embodiment that can be applied when a component of a region is obtained by an external configuration will be described. A configuration example of the image processing apparatus 10 of the present embodiment is shown in FIG. FIG. 14 shows an operation flow of the image processing apparatus 10 of the present embodiment. The candidate region group divided by the region dividing unit 101 is sent to the region identifying unit 105. The region identification unit 105 reads K region components from the region component database 107, and then calculates a feature amount for each candidate region (S1401, S1402). The Cullback-Ribler distance with K region components is calculated (S1403, S1404, S1405). The above processing is repeatedly executed for all candidate regions of the input image (S1406).

  Next, the following processing is performed for each pixel of the input image. A candidate area including the target pixel is selected, and a candidate area having a minimum distance to any of the area components is selected (S1407, S1408). At this time, if the Cullback-Lailer distance (KL distance) is equal to or smaller than the threshold (S1409—Yes), the pixel is assigned to the category of the same region component (S1410). On the other hand, if it is determined in step S1409 that the distance is larger than the threshold (S1409-No), labeling is not performed with the target pixel as a blank area (S1411). The above process is repeated for all the pixels of the input image (S1412). Then, the result of labeling of all pixels is output as a region division result, and the process is terminated (S1413).

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An image processing method for dividing an input image into regions for each subject in the input image,
A dividing step of dividing the input image into candidate region groups representing a plurality of region candidates for each subject;
An acquiring unit that acquires a similar image group similar to each candidate area of the candidate area group from an image database;
An extraction step in which an extraction unit extracts a feature common to similar images from the group of similar images as a feature component of a candidate region;
A region that is identified by the region identification means by comparing the feature component with the feature amount of each pixel of the input image, identifying which candidate region each pixel belongs to, and outputting as a result of region division An identification process;
An image processing method comprising: A second dividing unit further comprising a second dividing step of dividing the result of the region division output in the region identification step into a new candidate region group representing a plurality of region candidates for each subject;
The acquisition step acquires a similar image group similar to each candidate region of the new candidate region group divided in the second division step from the image database,
The extraction step extracts features common to similar images from the similar image group as feature components of candidate regions,
The region identification step compares the extracted feature component with the feature amount of each pixel of the input image to identify which candidate region the new candidate region group belongs to, and The image processing method according to claim 1, wherein the image processing method is a result of area division. The region identification step includes a determination step of determining whether the result of the second region division has converged with respect to the result of the region division,
When it is determined in the determination step that the result of the second region division has converged, the region identification step outputs the result of the second region division as a result of region division for each subject,
When it is determined by the determination step that the result of the second region division has not converged, the second division step uses the result of the second region division as a plurality of region candidates for each subject. The image processing method according to claim 2, wherein the image processing method is divided into new candidate area groups to be represented.   The setting means sets an annotation to the result of area division in the area identification step using a tag associated with each image stored in the image database, and recognizes the subject in the input image. 4. The image processing method according to claim 1, further comprising a setting step of outputting as a first step. An image processing apparatus that divides an input image into regions for each subject in the input image,
Dividing means for dividing the input image into candidate region groups representing candidates for a plurality of regions for each subject;
Acquisition means for acquiring a similar image group similar to each candidate area of the candidate area group from an image database;
Extraction means for extracting features common to similar images from the similar image group as feature components of candidate regions;
A region identification unit that compares the feature component with a feature amount of each pixel of the input image, identifies which candidate region each pixel belongs to, and outputs as a result of region division;
An image processing apparatus comprising: A second dividing unit that divides the result of the region division output by the region identification unit into a new candidate region group representing a plurality of region candidates for each subject;
The acquisition means acquires a similar image group similar to each candidate area of the new candidate area group divided by the second dividing means from an image database,
The extraction means extracts features common to similar images from the group of similar images as feature components of candidate regions,
The region identification means compares the extracted feature component with the feature amount of each pixel of the input image to identify which candidate region the new candidate region group belongs to, and The image processing apparatus according to claim 5, wherein the image processing apparatus is a result of area division. The area identification means has a determination means for determining whether or not the result of the second area division has converged with respect to the result of the area division,
When the determination unit determines that the result of the second region division has converged, the region identification unit outputs the result of the second region division as a result of region division for each subject,
When the determination unit determines that the result of the second region division has not converged, the second division unit uses the second region division result as a plurality of region candidates for each subject. The image processing apparatus according to claim 6, wherein the image processing apparatus is divided into new candidate area groups to be represented.   A setting for setting an annotation on the result of area division by the area identification unit using a tag associated with each image stored in the image database and outputting the result as a recognition result of the subject in the input image The image processing apparatus according to claim 5, further comprising a unit. An imaging means for capturing an image;
The image processing apparatus according to any one of claims 5 to 8, wherein the image captured by the imaging unit is used as an input image, and the input image is divided into regions for each subject in the input image.
An imaging device comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 5 thru | or 8.

Images