JP2013054458A - Image identification information assigning program and image identification information assigning device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image identification information assigning program and an image identification information assigning device which assign a plurality of pieces of identification information to an image by using correlation information concerning images.SOLUTION: An image identification information assigning device 1 comprises: a feature extraction unit 3 for extracting respective feature amounts from a plurality of query images 120; an annotation score quantization unit 5 for calculating respective quantization values for a plurality of labels to be assigned to a query image 120 by using a learning model 130 on the basis of the feature quantity extracted by the feature extraction unit 3; and a label estimation unit 7 which has the number of random field models corresponding to the number of labels, and inputs the quantization value of each label into the plurality of random field models to output scores of the plurality of labels for each query image 120.

Description

  The present invention relates to an image identification information providing program and an image identification information providing apparatus.

  In recent years, image annotation technology has become one important technology for image retrieval systems, image recognition systems, and the like in image database management. With this image annotation technology, the user can search for an image that is semantically close to the required image, for example.

  As an image annotation technique, for example, there are those disclosed in Patent Documents 1 to 4. These assign a semantic label to an unknown image, but as a means, a feature image is extracted and then a nearest neighbor algorithm (NN: Nearest Neighbor) is used to search for a similar image. A level is assigned to the target image using a label attached to the image. However, the method of attaching a label only from an image extracted by the nearest neighbor algorithm has a problem that the accuracy of the annotation is not high.

  In order to improve the above problems, there are some proposed in Patent Documents 5 and 6. They estimate the probability of each label using a learned discriminator based on the appearance frequency of the label with respect to the image feature.

  In addition, in order to improve the existing classification method, there is a model that fills the gap between image features and semantic labels by modeling correlation information between labels and features using Canonical Correlation Analysis (CCA). (For example, refer nonpatent literature 1).

JP 2005-352882 A JP 2007-109067 A JP 2009-188951 A JP 2010-271769 A JP 2000-353173 A JP 2009-48334 A

T.BAilloeul, C.Zhu and Y.Xu, “Automatic image tagging as a random walk with priors on the canonical correlation subspace”, MIR 2008

  However, the methods disclosed in Patent Documents 5 and 6 have a problem that the classifiers are constructed for each class of objects and the posterior probabilities of each label are calculated independently, so that the correlation between classes cannot be used. . Further, in the method disclosed in Non-Patent Document 1, a label is estimated from a feature amount of a target image by a random walk from a graph model constructed by CCA, which may fall into a local minimum value. There is a problem that calculation time is also required.

  The subject of this invention is providing the image identification information provision program and image identification information provision apparatus which provide several identification information with respect to an image using the correlation information regarding an image.

[1] An extraction unit that extracts a feature amount from each of a plurality of images, and a plurality of pieces of identification information to be assigned to the image using a learning model from the feature amounts extracted by the extraction unit. Computation means for calculating a first evaluation value and random field models corresponding to the number of identification information, and the first information for each of the identification information calculated by the calculation means for the plurality of images An image identification information addition program for functioning as output means for inputting evaluation values to the plurality of random field models and outputting second evaluation values for the plurality of identification information for each of the images.

[2] The image identification information adding program according to [1], further including optimization means for optimizing the random field model of the output means based on correlation information between the plurality of images.

[3] The image identification information addition program according to [1], further including optimization means for optimizing the random field model of the output means based on correlation information between the plurality of identification information.

[4] Extraction means for extracting feature amounts from a plurality of images, respectively, and a plurality of pieces of identification information to be added to the images using a learning model from the feature amounts extracted by the extraction means. A calculation means for calculating an evaluation value; and a number of MRF models corresponding to the number of the identification information, and the first evaluation value for each of the identification information calculated by the calculation means for the plurality of images An image identification information providing apparatus comprising: output means for inputting a plurality of random field models and outputting a second evaluation value for the plurality of identification information for each of the images.

  According to the invention described in claim 1 or 4, a plurality of pieces of identification information can be given to the image using the correlation information regarding the image.

  According to the second aspect of the present invention, a plurality of pieces of identification information optimized for an image can be given based on correlation information between the plurality of images.

  According to the third aspect of the present invention, a plurality of identification information optimized for an image can be given based on correlation information between the plurality of identification information.

FIG. 1 is a block diagram showing a configuration example of an image identification information providing apparatus according to the first embodiment of the present invention. FIG. 2 shows a schematic configuration example of the label estimation unit, where (a) is a plan view and (b) is a side view. FIG. 3 is a flowchart illustrating an operation example of the first embodiment. FIG. 4 is a block diagram showing a configuration example of an image identification information providing apparatus according to the second embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, in each figure, about the component which has the substantially same function, the same code | symbol is attached | subjected and the duplicate description is abbreviate | omitted.

[First Embodiment]
FIG. 1 is a block diagram showing a configuration example of an image identification information providing apparatus according to the first embodiment of the present invention. The image identification information adding device 1 includes an image receiving unit 2, a feature extracting unit 3, a label posterior probability calculating unit 4, a quantizing unit 5, a node joining unit 6, a label estimating unit 7, a label adding unit 8, and an annotation information output unit. 9 and a storage unit 10.

  The conventional annotation means extracts a feature amount from a learning image in a learning corpus (a pair of a learning image and a label given to the learning image) by a known feature extraction method, The relation with the label is learned as an identification model. The learned identification model, that is, the learning model 130 is stored in a database. Then, in order to give a label to the query image (also referred to as an input image or an unknown image) 120, the posterior probability of the label is calculated for the query image 120 using the learning model 130, and the label having the highest value is selected. Estimated result.

  In this specification, “annotation” means that a label is assigned to the entire image. The “label” is an example of identification information, and is identification information that represents the contents of the entire image or a partial area, for example, a word.

  In the present embodiment, the label posterior probability calculation unit 4 calculates the posterior probability of the label, and then adjusts the rank of the label using the MRF model or the CRF model based on the correlation information between the image query images 120, thereby obtaining the query image 120. Give a label to Here, the “MRF model” refers to a Markov Random Field (MRF) model, and the “CRF model” refers to a conditional random field (CRF) model. These Markov random field models and conditional established field models are examples of random field models.

  Hereinafter, the characteristic part of the present embodiment, that is, the quantization unit 5, the node junction unit 6, and the label estimation unit 7 will be mainly described.

  The image receiving unit 2 receives a query image 120 of a target image to which a label is assigned.

  The feature extraction unit 3 is an example of an extraction unit, and extracts feature amounts from the query image. The feature amount is an arrangement of image features such as colors such as R, G, B, and texture.

  The label posterior probability calculation unit 4 calculates the posterior probability (P (c | f)) of each label c from the feature quantity f, and outputs it as an annotation score (analog value) for each label.

  The storage unit 10 stores various programs such as an image identification information adding program 110, various data such as a query image 120, a learning model 130, a label dictionary 140, and link information 150. The storage unit 10 includes, for example, a ROM, a RAM, an HDD, and the like.

  The annotation output unit 9 outputs the annotation information (label and score) given by the label giving unit 8 to the outside. For example, a display unit such as a liquid crystal display or a printing unit such as a printer can be used. .

(Quantization part)
Conventional image annotation technology using MRF or CRF is disclosed in, for example, non-patent document “Word co-occurrence and Markov Random Field for Improving Automatic Image Annotation” HJEscalante, M. Montes and LESucar, BMVC, 2007 Then, the MRF model is constructed using the label co-occurrence, the label probability is input as an observation value, and the label is estimated for the input image. In this prior art, the hidden node that estimates the label of an image is a node that selects one label from a plurality of labels, and therefore only one label can be assigned to an entire image or image area, and the entire image It cannot be applied to annotations with multiple labels.

  In order to solve this, this embodiment has one MRF or CRF model for each label, and the hidden node of each model has the probability of a quantized label. Then, the order of the labels is determined by the quantized value estimated by the MRF or CRF model, and a plurality of labels having high scores are assigned to one image.

The quantization unit 5 of the present embodiment quantizes the annotation score, which is an analog value calculated for each label by the label posterior probability calculation unit 4. The quantized value (quantized value) is a discretized value (discretized value). In order to uniformly quantize the annotation score level, it is determined by a histogram equalization method. The quantized value is set as an initial state of a hidden variable of each posterior probability calculation node 72 of MRF models 70 ₁ to 70 _N described later. Table 1 shows an example of the annotation score (analog value) and the corresponding quantization value. In Table 1, M is the number of images, and N is the number of labels. Here, the label posterior probability calculation unit 4 and the quantization unit 5 are examples of calculation means, and an annotation score that is an analog value calculated by the label posterior probability calculation unit 4 and a quantization value output by the quantization unit 5 Is an example of a first evaluation value.

(Node junction)
The node junction unit 6 according to the present embodiment generates position information (link information) 150 of the junction link 76 that joins the posterior probability calculation node 72 and the estimated posterior probability calculation node 73 based on the correlation information between images. The generated link information 150 is stored in the storage unit 10. The input side link 75 and the output side link 77 are attached in advance. As the correlation information between images, for example, the image capturing time, the similarity of image feature amounts, and the like can be used. An example of using a correlation between annotations (labels) as correlation information will be described later.

The node junction unit 6 inputs a series of query images (query image set) 120, calculates correlation information between the images, and then calculates the posterior probability calculation node 72 and the estimated posterior of the MRF models 0 ₁ to 70 _N. A joining method between the probability calculation nodes 73 is determined. As an example of the joining method, when the similarity of the feature amount between images is equal to or greater than a threshold value, a joining link 76 is provided between the posterior probability calculation node 72 and the estimated posterior probability calculation node 73 corresponding to the images, When the similarity is smaller than the threshold value, the joint link 76 is not provided between the corresponding nodes 72 and 73. Moreover, you may provide the joining link 76 with respect to the image whose imaging | photography time is near each other. An example of the link information 150 is shown in Table 2. The number of images is the same as the number of posterior probability calculation nodes 72 and the number of estimated posterior probability calculation nodes 73. In Table 2, “1” indicates a case where the junction link 76 exists between the nodes 72 and 73, and “0” indicates a case where the junction link 76 does not exist between the nodes 72 and 73.

  The link information 150 can be generated from the correlation information of the image in advance, but the link information 150 may be generated dynamically. That is, when the distance between the hidden variables or the difference in the quantized value is equal to or less than a certain threshold due to the state of the hidden variable of the post-establishment calculation node 72, the junction link 76 between the nodes 72 and 73 is automatically added. If the distance between them or the difference in quantization values is greater than a certain threshold, the junction link 76 between the nodes 72, 73 may be automatically excluded.

(Label estimation part)
FIG. 2 is a diagram illustrating a schematic configuration example of the label estimation unit 7. The label estimation unit 7 has MRF models 70 ₁ to 70 _N provided for each label, and inputs the initial state of the node hidden variables of the corresponding MRF models 70 ₁ to 70 _N and link information 150 connecting the nodes. Then, the quantization state of the label is optimized by a graph cut message passing method (Yuri Boykov, O. Veksler, R. Zabih, “Fast Approximate Energy Minimization via Graph Cuts”, PAMI2001).

Since each MRF model 70 ₁ to 70 _N has the same structure, the MRF model 70 ₁ will be described as a representative. MRF model 70 _1, as shown in FIG. 2 (a), the posterior probability of holding an input node ₇₁ 1 -71 _M quantized value Q is inputted, the posterior probability output from the annotation scoring quantization unit 5 The calculation nodes 72 _{1 to} 72 _M , the estimated posterior probability calculation nodes 73 _{1 to} 73 _M for calculating the estimation posterior probability, the output nodes 74 ₁ to 74 _M for outputting the label score, and the input nodes 71 _{1 to} 71 _M and the input-side link ₇₅ 1 to 75 _M for joining the posterior probability calculation nodes ₇₂ 1 to 72 _M, the posterior probability calculation nodes ₇₂ 1 to 72 _M and the estimated posteriori probability calculation nodes ₇₃ 1-73 junction link 76 joining the _M , it is schematically configured to have an output-side link ₇₇ 1 to 77 _M for bonding the estimated posterior probability calculation nodes ₇₃ 1 to 73 _M output node ₇₄ 1 to 74 _M. Further, since the image and the input nodes 71 _{1 to} 71 _M and the output nodes 74 ₁ to 74 _M correspond one-to-one, the input nodes 71 _{1 to} 71 _M and the output nodes 74 _{1 of} each MRF model 70 ₁ to 70 _N. to 74 the number of _M is the same as the number M of the image.

For example, the quantized values Q _{11 to} Q _1N of the _first image (Image ₁₎ are input to the input nodes 71 ₁ of the MRF models 70 ₁ to 70 _N , and the quantized values Q _{21 to} Q of the next image (Image 2) are input. _2N inputs to each input node 71 ₂ of MRF model ₇₀ 1 to 70 _N, the quantization value Q _M1 to Q _MN similarly M-th image (Imagem), each input of the MRF model ₇₀ 1 to 70 _N After inputting to the node 71 _M , the scores of the labels L _{1 to} L _N for the images ₁ to _M are output from the output nodes 74 ₁ to 74 _{M of the} MRF models 70 ₁ to 70 _N.

The input side links 75 ₁ to 75 _M and the output side links 77 _{1 to} 77 _M are given in advance. The junction link 76 is given by the node junction 6 based on the link information 150. The joint link 76 joins not only the posterior probability calculation nodes 72 _{1 to} 72 _{M of} one MRF model 70 and the estimated posterior probability calculation nodes 73 _{1 to} 73 _M but also the MRF models 70 ₁ to 70 _N.

With the above configuration, all the states of the corresponding nodes of the MRF model 70 are compared, and all labels are assigned to the images. That is, for the image M, the values of the output nodes 74 _M of the MRF models 70 ₁ to 70 _N are compared, and a higher level label is assigned to the image. Here, the label estimation unit 7 is an example of an output unit, and the scores of the labels L _{1 to} L _N output from the output node 74 ₁ are an example of a second evaluation value.

(Operation of the first embodiment)
FIG. 3 is a flowchart illustrating an operation example of the first embodiment. The present embodiment is characterized in that the junction links 76 are attached to the MRF models 70 ₁ to 70 _N based on the correlation information of the images.

  When the image reception unit 2 receives the query image 120, the feature extraction unit 3 extracts a feature amount from the query image 120.

  The label posterior probability calculation unit 4 calculates the posterior probability of each label with respect to the query image 120 using the learning model 130 stored by a known classifier (S1), and outputs the posterior probability as an annotation score.

  The quantization unit 5 quantizes the annotation score output by the label posterior probability calculation unit 4 according to a predetermined threshold (S4). The quantized value is set to the initial value of the hidden node, and then the estimation result of the final state of the hidden variable is held in the estimated posterior probability calculation node 73 by the graph cut message passing method.

  Next, after all the estimated posterior probability calculation nodes 73 have processed, the link information 150 connecting the nodes 72 and 73 is acquired. The node junction 6 attaches the junction link 76 based on the correlation information of the image (S5). When the correlation information of the image is time, if the difference between the shooting times of the image pair is equal to or less than a predetermined time (for example, 5 hours), the junction link 76 is attached between the nodes 72 and 73 of the corresponding pair. Further, when the difference between the shooting times of different images is larger than a predetermined time (for example, 5 hours), the junction link 76 is not attached between the nodes 72 and 73 of the corresponding pair.

  When the correlation information of the image is the similarity of the image, various feature amounts are extracted from the image. For example, RGB, normalized-RG, HSV (color space), LAB, robustHue feature (see van de Weijer, C. Schmid, “Coloring Local Feature Extraction”, ECCV 2006), Gabor feature, DCT (Direction Curve Tangent ) Feature amount, SIFT (Scale Invariant Feature Transform) feature amount, and GIST (Generalized Search Tree) feature amount, and any feature may be used. The similarity between images is the distance of the feature amount. When the normalized distance is 0.5 or less, the junction link 76 is attached between the pair of nodes 72 and 73 corresponding to the image pair. If it is greater than 0.5, the junction link 76 is not attached between the pair of nodes 72 and 73 corresponding to the image pair.

As described above, the MRF model 70 corresponding to one label is constructed. In the next step, the MRF model 70 is optimized (S6). That is, the hidden variable state calculated in step S4 is input to the posterior probability calculation node 72 of the MRF models 70 ₁ to 70 _N corresponding to the labels, and the link information 150 that joins the nodes 72 and 73 is input to the node. A joining link 76 is attached between 72 and 73. The above steps S4, S5, S6 are performed for all labels and nodes (S2, S3).

Finally, all the MRF models 70 ₁ to 70 _N corresponding to each label are optimized, and the estimated posterior probability calculation nodes 73 _{1 to} 73 _M of all the MRF models 70 ₁ to 70 _N corresponding to one image are changed. The final state was integrated, and as a result, all the annotation scores for the images could be adjusted. Then, the ranks of the adjusted annotation scores are given and labels are given to the query images in descending order (S7). For example, the quantization values _Q 11 _{to Q 1N} of one image (Image1), the input to each input node 71 ₁ of MRF model ₇₀ 1 to 70 _N, the estimated posterior probability of all MRF models ₇₀ 1 to 70 _N The hidden variable of the calculation node 73 ₁ is output from the output node 74 ₁ as the score of each label L _{1 to} L _N.

(Effects of the first embodiment)
According to the first embodiment, since the MRF model is optimized based on correlation information between a plurality of images, a plurality of labels are assigned to the images with higher accuracy than when this configuration is not adopted. can do.

[Second Embodiment]
FIG. 4 is a block diagram showing a configuration example of an image identification information providing apparatus according to the second embodiment of the present invention. As in the first embodiment, the image identification information providing apparatus 1 according to the present embodiment includes an image reception unit 2, a feature extraction unit 3, a label posterior probability calculation unit 4, a quantization unit 5, a node junction unit 6, A label estimation unit 7, a label addition unit 8, an annotation information output unit 9, and a storage unit 10 are schematically configured. The present embodiment is different from the first embodiment in that the node junction portion 6 is different, and other configurations are the same as those in the first embodiment, and the same operations are performed.

  The node junction 6 of the present embodiment generates a junction link 76 between the posterior probability calculation node 72 and the estimated posterior probability calculation node 73 of the MRF model 70 based on the correlation information of the label, and the position of the generated junction link 76 The link information, which is information, is stored in the storage unit 10. As the correlation information of labels, for example, the top five are listed by quantized annotation scores for a certain image pair, and the number of the same label of the image pair is counted regardless of the order. When the number of the same labels is one or more, the joint link 76 is provided between the corresponding nodes 72 and 73, and when the number of the same labels is zero, the joint link 76 is not provided between the corresponding nodes 72 and 73.

(Effect of the second embodiment)
According to the second embodiment, since the MRF model is optimized based on correlation information between a plurality of labels, a plurality of labels are assigned to an image with higher accuracy than when this configuration is not adopted. can do.

  Next, an embodiment of the present invention will be described by taking as an example the case where the number M of images is 100 and the range of quantization values is 1 to 2000. The annotation score of the analog value calculated by the label posterior probability calculation unit 4 is converted into a discretized value by the quantization unit 5. Table 3 shows a specific example of the annotation score (analog value) and the quantized value (discretized value).

  In Table 3, parentheses below the image ID indicate a correct label to be assigned to the image. From Table 3, it can be seen that if the first-ranked label is attached only by the annotation score (label posterior probability) output from the quantizing unit 5, the accuracy is not high because Image1 and ImageM are incorrect.

  An example of node information (node junction matrix) 150 created by the node junction 6 is shown in Table 4. In Table 4, “1” indicates that there is a temporal correlation between images, and thus indicates a case where there is a junction link 76 between the nodes 72 and 73, and “0” indicates that there is no temporal correlation between images. The case where there is no joining link 76 between 72 and 73 is shown.

  Table 5 shows the quantized annotation score (quantized value) before adjustment (input value of the MRF model) and the annotation score (quantized value) after adjustment (output value of the MRF model).

  Here, the number of images is 100, and the range of quantization values is 1 to 2000. The quantized values in Table 5 are obtained by converting the quantized values of analog values into discretized values by the histogram flattening method. The parentheses below the image ID in Table 5 indicate the correct answer label to be attached to the image. The second and third columns in Table 5 are before the MRF model 70 is optimized. The labels before optimization are arranged in descending order of quantization value. The quantization value before optimization is the same as in Table 3. The fourth and fifth columns in Table 5 are after the MRF model 70 has been optimized. The optimized labels are arranged in descending order of quantization values (adjusted annotation scores). From the above results, for the image ID Image1, the label “hug” was ranked first before optimization, but after optimization, the label “hand” was ranked first and the correct answer was obtained. In addition, for the image ID Image100, the first-ranked label “hand” does not change before and after optimization. However, the second place changes from the label “face” before optimization to the label “foot” after optimization, and the quantization value also increases from “117” to “148”, which is close to the correct answer. It turns out that it becomes high.

  According to the present embodiment, the F value (F-measure), which is a well-known evaluation value of information retrieval, for the query image set is improved from 0.536 to 0.549. Although the present embodiment uses image correlation, the same effect as the present embodiment can be expected when label correlation is used.

[Other embodiments]
In addition, this invention is not limited to the said embodiment, A various deformation | transformation is possible in the range which does not change the summary of this invention. For example, each function of the image reception unit 2, the feature extraction unit 3, the label posterior probability calculation unit 4, the quantization unit 5, the node junction unit 6, the label estimation unit 7, the label assignment unit 8, and the annotation information output unit 9 is a computer. It may be realized by the CPU operating in accordance with the readable image identification information adding program 110. In addition, the image receiving unit 2, the feature extracting unit 3, the label posterior probability calculating unit 4, the quantizing unit 5, the node joining unit 6, the label estimating unit 7, the label attaching unit 8, and the annotation information output unit 9 of the above embodiment. You may implement | achieve all or one part by hardware, such as ASIC.

  The program used in the above embodiment can be provided by being stored in a recording medium such as a CD-ROM. Moreover, replacement, deletion, addition, and the like of the steps described in the above embodiments are possible within a range that does not change the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Image identification information provision apparatus, 2 ... Image reception part, 3 ... Feature extraction part, 4 ... Label posterior probability calculation part, 5 ... Quantization part, 6 ... Node junction part, 7 ... Label estimation part, 8 ... Label assignment Part, 9 ... annotation information output part, 10 ... storage part, 70 ₁ to 70 _N ... MRF model, 71 _{1 to} 71 _M ... input node, 72 _{1 to} 72 _M ... posterior probability calculation node, 73 _{1 to} 73 _M ... estimation A posteriori probability calculation node, 74 ₁ to 74 _M ... output node, 75 ₁ to 75 _M ... input side link, 76 ... junction link, 77 _{1 to} 77 _M ... output side link, 110 ... image identification information adding program, 120 ... query Image 130 ... Learning model 140 ... Label dictionary 150 ... Link information

Claims (4)

Computer
Extraction means for extracting feature amounts from a plurality of images,
Calculation means for calculating a first evaluation value for each of a plurality of pieces of identification information to be added to the image using a learning model from the feature amount extracted by the extraction means;
There are a number of random field models corresponding to the number of identification information, and the first evaluation value for each of the identification information calculated by the calculation unit for the plurality of images is input to the plurality of random field models. An image identification information adding program for functioning as output means for outputting a second evaluation value for the plurality of identification information for each image.   The image identification information addition program according to claim 1, further comprising optimization means for optimizing the random field model of the output means based on correlation information between the plurality of images.   The image identification information adding program according to claim 1, further comprising optimization means for optimizing the random field model of the output means based on correlation information between the plurality of identification information. Extraction means for extracting feature amounts from a plurality of images,
Calculation means for calculating a first evaluation value for each of a plurality of pieces of identification information to be added to the image using a learning model from the feature amount extracted by the extraction means;
There are a number of random field models corresponding to the number of identification information, and the first evaluation value for each of the identification information calculated by the calculation unit for the plurality of images is input to the plurality of random field models. An image identification information providing apparatus comprising: output means for outputting a second evaluation value for the plurality of identification information for each image.

Images