JP2014048989A - Bof expression generation device and bof expression generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a device or a method for generating BoF expression capable of performing accurate image retrieval.SOLUTION: Distances between i-th subject local feature quantity on an object image and a plurality of clusters are calculated respectively. Then, a plurality of clusters that are relatively closer to the i-th subject local feature quantity than other clusters are determined. The BoF expression corresponding to the object image is generated using ID information of the plurality of determined clusters.

Description

  The present invention relates to a BoF expression generation device and a BoF expression generation method.

  As shown in Patent Document 1 below, an image recognition method using local feature amounts is known. In this method, an image is divided into a plurality of parts, a local feature amount is extracted corresponding to each part, and a similar image search can be performed using the local feature amount. These local feature quantities are usually multidimensional vector data in order to accurately represent the features of the image.

  By the way, as shown also in the following patent document, in the similar image search using the local feature amount, the calculation amount tends to be enormous.

  Therefore, a technique called a visual keyword or BoF (Bag of Features) has been proposed to search for local feature quantities at high speed. In this method, a local feature amount (that is, a visual keyword) representing a cluster is calculated by clustering a plurality of local feature amounts in advance. Further, an ID is assigned to each cluster. Thereby, each local feature-value can be linked | related with ID of the cluster to which it belongs.

  Then, a set of a plurality of local feature amounts corresponding to an image can be replaced by a cluster ID histogram (also referred to as a set). This cluster ID histogram is called BoF expression or simply BoF. One BoF expression can be obtained corresponding to one image. The generated BoF representation can generally be searched by a transposed file. The BoF expression is sometimes referred to as a “feature vector”. In this specification, the term “BoF expression” is used in order to avoid confusion with a feature vector indicating a feature amount of an image.

  Thus, by using the BoF expression, the local feature amount can be quantized and the data amount can be reduced. In addition, since a transposed file can be used for the search, a similar image search (that is, image recognition) can be performed faster than using the local feature amount itself.

  On the other hand, there is a problem that the search accuracy is lowered by quantizing the local feature amount. This is considered due to the fact that individual local feature values have been replaced with cluster IDs. Therefore, in Patent Document 1 below, when mapping the local feature quantity of an image to a cluster (that is, associating it with a cluster ID), weighting is performed according to the distance between the local feature quantity and the cluster center to generate a BoF expression To do. However, depending on the image capturing state, local feature amounts are mapped to clusters that should not be mapped originally, and as a result, search accuracy may deteriorate. This problem cannot be solved sufficiently by weighting. That is, the conventional technique has room for improvement in search accuracy.

JP 2011-128773 A (paragraphs 0003 to 0005, paragraph 0069)

  The present invention has been made in view of the above situation. A main object of the present invention is to provide an apparatus or a method for generating a BoF expression capable of performing an accurate image search.

  The present invention is configured as described in any of the following items.

(Item 1)
An apparatus for generating a BoF expression corresponding to a target image using a plurality of clusters generated by clustering for a plurality of learning local feature amounts,
A distance calculation unit that calculates distances between the i-th target local feature amount in the target image and the plurality of clusters;
A cluster identifying unit that identifies a plurality of clusters, the distance from the i-th target local feature is relatively closer than other clusters;
A BoF expression generation device, comprising: a BoF expression generation unit that generates BoF expression corresponding to the target image using ID information of the plurality of identified clusters.

  In the present invention, a specific local feature amount can be associated with a plurality of cluster IDs having a predetermined relationship with the local feature amount, and a BoF expression can be generated using the cluster ID. Such generation of the BoF expression can be performed in either or both of the image learning process and the recognition (that is, similarity search) process. In the learning step, the target image in the present invention is a learning image, and in the recognition step, the target image in the present invention is a query (search) image.

  Further, the cluster specifying unit of the present invention may not always specify a plurality of clusters. That is, the cluster specifying unit of the present invention may be configured to specify a plurality of clusters only when a predetermined condition is satisfied. In the present invention, the specification of the cluster means specifying the cluster ID.

(Item 2)
The BoF expression generation device according to Item 1, wherein the cluster specifying unit is configured to perform processing for specifying a plurality of clusters whose distance from the i-th target local feature is within a predetermined value.

  The cluster specifying unit may specify all the clusters that are within the predetermined value, or may specify the clusters that are within the predetermined range and within the predetermined number in close order.

(Item 3)
The BoF expression generation device according to Item 1, wherein the cluster specifying unit is configured to perform a process of specifying a plurality of clusters in order of increasing distance from the i-th target local feature.

(Item 4)
The cluster specifying unit is configured to vary the number of the clusters to be specified according to the genre difference between the learning image that is the original image of the learning local feature and the target image. 4. The BoF expression generation device according to any one of 3.

(Item 5)
The cluster specifying unit is configured to vary the number of the clusters to be specified according to the certainty of the search result obtained in the image search using the generated BoF expression. The BoF expression generation device according to any one of the above.

  Here, the certainty of the search result means the reliability of the search result, which can be determined by the number of images obtained by the search, for example. Alternatively, the certainty factor can be determined based on the degree of coincidence of coordinates obtained by collation using the coordinates of the local feature amount performed after the image search.

(Item 6)
The BoF expression generation unit is configured to perform weighting of the BoF expression according to a distance between the i-th target local feature and the identified cluster. The BoF expression generation device described in 1.

(Item 7)
A BoF expression generation method executed by using a BoF expression generation apparatus for generating a BoF expression corresponding to a target image using a plurality of clusters generated by clustering with respect to a plurality of learning local feature amounts,
The BoF expression generation device includes a distance calculation unit, a cluster specifying unit, and a BoF expression generation unit.
The distance calculating unit calculating an i-th target local feature amount in the target image and a distance between each of the plurality of clusters;
The cluster specifying unit specifying a plurality of clusters whose distance from the i-th target local feature is relatively closer than other clusters;
The BoF expression generation method includes: a step of generating a BoF expression corresponding to the target image using ID information of the plurality of identified clusters.

(Item 8)
A computer program for causing a computer to execute each step according to item 7.

  This computer program is stored in an appropriate recording medium (for example, an optical recording medium such as a CD-ROM or a DVD disk, a magnetic recording medium such as a hard disk or a flexible disk, or a magneto-optical recording medium such as an MO disk). Can be stored. This computer program can be transmitted via a communication line such as the Internet.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to provide the apparatus or method which produces | generates the BoF expression which can perform an accurate image search.

It is a schematic block diagram of the BoF expression production | generation apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the whole procedure of the BoF expression production | generation method which concerns on one Embodiment of this invention. It is a flowchart for demonstrating the procedure which produces | generates a cluster using a learning image. It is explanatory drawing which shows an example of a some learning image. It is explanatory drawing which shows an example of the local feature-value arrange | positioned in the feature-value space. It is explanatory drawing for demonstrating an example of the produced | generated cluster center. It is a flowchart for demonstrating the procedure which maps a local feature-value to cluster ID. It is explanatory drawing for demonstrating the state by which one local feature-value was mapped by three cluster ID. It is explanatory drawing for demonstrating the mode of the mapping by the method of the Example of this invention, and the method of a comparative example. FIG. 5A is an explanatory diagram for explaining BoF expression generated by the method of the comparative example. FIG. 5B is an explanatory diagram for explaining the BoF expression generated by the method of the embodiment. It is explanatory drawing which shows an example of the BoF expression produced | generated.

(Configuration of this embodiment)
An example of the BoF expression generation device according to an embodiment of the present invention will be described with reference to FIG. The BoF expression generation device includes a distance calculation unit 1, a cluster identification unit 2, and a BoF expression generation unit 3. Furthermore, this BoF expression generation device includes a local feature quantity extraction unit 4 and a cluster generation unit 5.

  The distance calculation unit 1 is configured to calculate the distances between the i-th target local feature amount in the target image and a plurality of clusters described later. Here, the target image is a learning image in the learning process or a query image in the search (recognition) process. In the following description, the description will be made with the learning process in mind, but the same description is valid in the search process. In addition, the i-th target local feature value means a specific (usually any one) local feature value in a specific target image. Therefore, i is usually an arbitrary natural number. For example, if the number of local feature amounts is 10, the value i takes is generally 1 ≦ i ≦ 10, but it is possible to take a value that is not a sequential number.

  The cluster specifying unit 2 is configured to specify a plurality of clusters whose distance from the i-th target local feature is relatively closer than other clusters. Of course, when the predetermined condition is not satisfied, the cluster specifying unit 2 can perform a process of specifying only one cluster or not specifying a cluster at all.

  More specifically, the cluster specifying unit 2 of the present embodiment specifies all or a predetermined number (for example, 3 or less) of a plurality of clusters whose distance from the i-th target local feature is within a predetermined value. It is the structure which performs a process.

  The BoF expression generation unit 3 is configured to generate a BoF expression corresponding to the target image using information of a plurality of identified cluster IDs. That is, the histogram of ID information corresponds to BoF expression. A specific example of the generated BoF expression will be described later.

  The local feature amount extraction unit 4 is configured to extract a local feature amount using data of the input target image (learning image or query image). With respect to the local feature extraction method, a technique similar to the conventional technique (for example, SIFT) can be used, and a detailed description thereof will be omitted. There are two methods for identifying the portion from which the local feature amount is to be extracted: a method of dividing the entire image into predetermined sizes, and a method of identifying a characteristic region in the image, either of which may be employed. In addition, since these techniques can also use the same technique as in the past, detailed description thereof is omitted.

  The cluster generation unit 5 is configured to generate a cluster by clustering the generated local feature amounts. As a clustering method, for example, k-means can be used, but is not particularly limited. Further, since an existing method can be used as the clustering method, a detailed description thereof will be omitted.

(Method for generating BoF expression according to this embodiment)
Next, a method for generating a BoF expression using the above-described apparatus will be described with further reference to FIGS. First, an overall procedure will be described with reference to FIG. 2, and then details of each procedure will be described.

(Step SA-1 in FIG. 2)
First, a local feature cluster is generated using the input learning image data.

(Step SA-2 in FIG. 2)
Next, each local feature amount of the learning image or the query image is mapped using the generated cluster. That is, the cluster ID corresponding to the local feature amount is specified.

(Step SA-3 in FIG. 2)
Next, a BoF expression for the learning image or the query image is generated using the histogram of the identified cluster ID.

  Hereinafter, each step will be described in detail.

(Cluster generation procedure)
(Step SB-1 in FIG. 3)
First, the local feature amount extraction unit 4 accepts learning image data (see FIG. 4). The image data is, for example, a photograph of a product to be searched, but is not particularly limited.

(Step SB-2 in FIG. 3)
Next, the local feature quantity extraction unit 4 extracts a local feature quantity from the received image data (see FIG. 5). This step is performed for each image data for learning.

(Step SB-3 in FIG. 3)
Next, the cluster generation unit 5 clusters local feature amounts in the feature amount space to generate a cluster. An example of the generated cluster is shown in FIG. Although FIG. 6 shows the cluster radius, it is not necessary to define the cluster radius. For example, the region can be divided by Voronoi division based on the cluster center. In short, once the cluster center is determined, the procedure described later can be executed.

  The cluster generation unit 5 generates a cluster ID for each obtained cluster. The cluster ID is, for example, a number assigned to the cluster, but is not particularly limited as long as it is information that can identify the cluster. Furthermore, the cluster generation unit 5 of the present embodiment calculates a representative local feature amount that represents each cluster. The representative local feature value is, for example, the local feature value closest to the cluster centroid, but the value of the cluster centroid can also be used as the representative local feature value.

(Step SB-4 in FIG. 3)
Next, the cluster generation unit 5 records information on the generated cluster in an appropriate recording unit (for example, a hard disk).

(mapping)
Next, a mapping method for generating a BoF expression using the generated cluster will be described with further reference to FIG.

(Step SC-1 in FIG. 7)
First, the local feature quantity extraction unit 4 receives an image (a learning image or a query image) to be quantized.

(Step SC-2 in FIG. 7)
Next, the local feature amount extraction unit 4 extracts a local feature amount in the received image. Note that the local feature amount extraction unit for the learning image and the local feature amount extraction unit for the query image may be implemented by different hardware or software, but in this specification, the local feature amount extraction is collectively performed. The expression part 4 is used.

(Step SC-3 in FIG. 7)
Next, the distance calculation unit 1 calculates the distance between each extracted local feature and each cluster center. Each local feature extracted in this way is referred to as the i-th local feature in this specification.

(Step SC-4 in FIG. 7)
Next, the cluster specifying unit 2 associates the local feature quantity with the cluster ID using information on the distance between the local feature quantity and the cluster center. Here, in the present embodiment, the cluster specifying unit 2 specifies a plurality of clusters whose distance from the i-th (that is, one specific target) target local feature is relatively closer than other clusters. In the example of FIG. 8, for example, three cluster IDs (that is, # 2, # 4, and # 5) within a predetermined distance are assigned to one local feature amount.

  By assigning a cluster ID for each local feature, a cluster ID histogram for the entire image can be obtained. The obtained histogram corresponds to the BoF expression. Therefore, a BoF expression for the learning image and a BoF expression for the query image can be generated, and similar image search and image recognition can be performed based on the similarity or identity of these BoF expressions.

(Examples and Comparative Examples)
Here, the method of specifying the cluster ID will be described in more detail with reference to FIG.

  An example of the image (for learning or query) is shown in FIG. FIG. 9B shows a state in which this figure is rotated 90 degrees clockwise. Here, the local feature amount of the region near the right eye in this figure is considered. In the comparative example, one local feature is associated with one cluster as in the conventional example. In this case, a situation occurs in which the local feature amount F2 in FIG. 9B is assigned to the cluster number # 2 even though the local feature amount F1 in FIG. 9A is assigned to the cluster number # 1. Yes (see FIG. 9C). This is because even if the images have the same contents, the local feature amount is different if there is a change in conditions such as a change in illumination conditions. When assigned in this way, the generated BoF expression differs between the image of FIG. 9A and the image of FIG. 9B, and as a result, the accuracy of the search using the BoF expression deteriorates. This causes a problem. An example of BoF expression formed by the method of this comparative example is shown in FIG. A set formed by cluster IDs shown in this figure corresponds to a BoF expression. In the example of this figure, for example, a vector (that is, a local feature amount) having an element of (24, 2, 4, 90, 3) is assigned to the cluster ID of # 39.

  In contrast, in the example, one local feature is associated with a plurality of clusters according to the method of the above-described embodiment. Then, the local feature amount F1 in FIG. 9A can be assigned to the cluster numbers # 1 and # 2. Similarly, the local feature amount F2 in FIG. 9B can also be assigned to the cluster numbers # 1 and # 2. As described above, according to the method of the embodiment, it is easy to assign the same cluster number to the local feature amounts F1 and F2 with respect to the partial images whose contents are the same even if the state of the image changes. Therefore, according to the method of this embodiment, there is an advantage that the accuracy of search using BoF can be improved. An example of BoF expression formed by the method of this embodiment is shown in FIG. A set formed by cluster IDs shown in this figure corresponds to a BoF expression. For example, a vector (24, 2, 4, 90, 3) is assigned to clusters # 39 and # 27. In this example, a maximum of three cluster IDs and a minimum of one cluster ID are assigned to local feature amounts. Note that the maximum value and the minimum value of the number of cluster IDs assigned to the local feature amount can be set as appropriate. The minimum value can also be set to zero. Also, a method of assigning all cluster IDs at a predetermined distance to the local feature amount is possible.

(Modification 1)
In the embodiment described above, a plurality of cluster IDs within a predetermined distance are assigned to one local feature amount. However, it is also possible to assign a predetermined number of cluster IDs in ascending order to one local feature. In this case, the number of cluster IDs assigned to one local feature amount is a predetermined value (for example, 2).

(Modification 2)
The cluster specifying unit 2 is configured to vary the number of the clusters to be specified according to the genre difference between the learning image that is the original image of the learning local feature and the target image (learning or query image). Also good. For example, when the genre of the learning image used for cluster generation is landscape and the genre of the image to be recognized is a person, the number of clusters assigned to one local feature amount is increased. In this way, improvement in image search accuracy can be expected. The genre to which the image belongs can be determined manually, for example. Further, genre information can be given to each image in advance.

(Modification 3)
The cluster specifying unit 2 may be configured to vary the number of clusters to be specified according to the certainty of the search result obtained in the search using the generated BoF expression. Here, the certainty of the search result means the reliability of the search result, which is, for example, the generated BoF expression (histogram vector data) and all BoF expressions (that is, learning data) in the DB. When the distance (for example, Euclidean distance) is calculated and the BoF expression having the shortest distance is used as the search result, the distance becomes the certainty factor. It can be considered that the certainty level is high when the distance is small and the certainty level is low when the distance is large. In addition, when a threshold is set for the distance and a BoF expression exceeding the threshold is used as a search result, the number of searches is a certainty factor. When the obtained number of searches is small, it can be considered that the certainty of the search results is low. Furthermore, after searching with the BoF expression, if there is a function for checking the positional relationship between the coordinates of the local feature values in the image obtained by this search and the coordinates of the local feature values in the recognition target image, It is also possible to set the degree of coincidence as a certainty factor. Since the existing method can be used as a matching method using the coordinates of the local feature amount, detailed description thereof is omitted.

  That is, in the search using the BoF expression, the search is performed based on the matching degree of the BoF expression. At this time, when the certainty of the search result is low (for example, the number of images obtained by the search is 0), the number of clusters assigned to one local feature amount can be increased. If the reference value of the BoF expression matching degree is simply lowered, the search accuracy may be deteriorated. However, according to the second modification, an improvement in the search accuracy can be expected.

(Modification 4)
The BoF expression generation unit 3 may be configured to perform weighting of the BoF expression according to the distance between the i-th (that is, any one) target local feature and the identified cluster. For example, it is assumed that two cluster IDs (ID numbers # 1 and # 2) are assigned to one local feature. At this time, the histogram of the cluster ID is weighted according to the distance between the local feature amount and the cluster center. The weighting method is not particularly limited. For example, when the weight value is W,
W = a · 1 / d
It can be. Here, d is a distance, and a is a positive proportionality constant. For convenience of calculation, it is preferable to normalize the maximum weight value W _max to 1.

  The degree of coincidence of the BoF expression can be calculated using a histogram in consideration of the weight value.

  The contents of the present invention are not limited to the above embodiment. In the present invention, various modifications can be made to the specific configuration within the scope of the claims.

  For example, each component described above may exist as a functional block, and may not exist as independent hardware. As a mounting method, hardware or computer software may be used. Furthermore, one functional element in the present invention may be realized by a set of a plurality of functional elements, and a plurality of functional elements in the present invention may be realized by one functional element.

  Moreover, the functional element may be arrange | positioned in the position physically separated. In this case, the functional elements may be connected by a network. It is also possible to realize functions or configure functional elements by grid computing or cloud computing.

DESCRIPTION OF SYMBOLS 1 Distance calculation part 2 Cluster specific part 3 BoF expression production | generation part 4 Local feature-value extraction part 5 Cluster production | generation part

Claims (8)

An apparatus for generating a BoF expression corresponding to a target image using a plurality of clusters generated by clustering for a plurality of learning local feature amounts,
A distance calculation unit that calculates distances between the i-th target local feature amount in the target image and the plurality of clusters;
A cluster identifying unit that identifies a plurality of clusters, the distance from the i-th target local feature is relatively closer than other clusters;
A BoF expression generation device, comprising: a BoF expression generation unit that generates BoF expression corresponding to the target image using ID information of the plurality of identified clusters. The BoF expression generation device according to claim 1, wherein the cluster specifying unit is configured to perform processing for specifying a plurality of clusters whose distance from the i-th target local feature is within a predetermined value. The BoF expression generation device according to claim 1, wherein the cluster specifying unit is configured to perform a process of specifying a plurality of clusters in order of increasing distance from the i-th target local feature. 2. The cluster specifying unit is configured to vary the number of clusters to be specified according to a difference in genre between a learning image that is an original image of the learning local feature and the target image. The BoF expression generation device according to any one of? The cluster specifying unit is configured to vary the number of the clusters to be specified according to a certainty factor of a search result obtained in an image search using the generated BoF expression. The BoF expression generation device according to any one of the above. The BoF expression generation unit is configured to weight the BoF expression according to a distance between the i-th target local feature and the identified cluster. The BoF expression generation device according to Item. A BoF expression generation method executed by using a BoF expression generation apparatus for generating a BoF expression corresponding to a target image using a plurality of clusters generated by clustering with respect to a plurality of learning local feature amounts,
The BoF expression generation device includes a distance calculation unit, a cluster specifying unit, and a BoF expression generation unit.
The distance calculating unit calculating an i-th target local feature amount in the target image and a distance between each of the plurality of clusters;
The cluster specifying unit specifying a plurality of clusters whose distance from the i-th target local feature is relatively closer than other clusters;
The BoF expression generation method includes: a step of generating a BoF expression corresponding to the target image using ID information of the plurality of identified clusters.   A computer program for causing a computer to execute each step according to claim 7.