KR101559457B1 - Method and apparatus of class-specific weighting based for image retrieval based on bag-of-word model - Google Patents

Abstract

There is provided an apparatus for learning a weight for each class in a visual word based image search, the apparatus comprising: an image storage unit for storing each of a plurality of images as a visual word vector and classifying and managing each image into at least one image class; A weight assignment function learning unit for assigning a weight to a weight word of each image class by extracting a weight value corresponding to the importance index of the visual word for each image class based on the weight assignment function, And an image class weight calculation unit for calculating the image class weight.

Description

[0001] METHOD AND APPARATUS OF CLASS-SPECIFIC WEIGHTING BASED FOR IMAGE RETRIEVAL BASED ON BAG-OF-WORD MODEL [0002]

The present invention relates to a weight learning method and apparatus.

Document retrieval can measure the similarity between documents using the Bag of Words methodology. A word bag refers to a document as a set of words, and expresses the appearance frequencies of words as a histogram and vectors them. Calculate the similarity between documents by measuring the distance between histograms.

In the field of image retrieval of computer vision, word bag methodology can be used as it is. However, the image domain does not have the concept of a word unlike a document. Therefore, a method for generating words in images has been proposed.

The word bag method of images quantizes the local descriptor space by clustering the local descriptors of the image and kmeans clustering. The center of a quantized cluster is taken as a visual word. As in the case of the document, the word bag method of an image expresses one image as an appearance frequency histogram of visual words, and the similarity between images is quantified by measuring the distance between the histograms. At this time, an inverted file structure, which is a data retrieval structure, is used as a retrieval algorithm in order to retrieve a huge amount of database images.

The Bag-of-Words Representation through visual words quantizes the local descriptor space to generate visual words, resulting in loss of information. Since the loss of information leads to the degradation of the distinction between images, it is important to reduce errors due to quantization as much as possible. Minimize the error by quantizing the local technician space very finely, usually to about one million spaces. However, this means that the number of dimensions of the histogram becomes one million dimensions, so it is very difficult to learn the weight vector to improve the distinction between histograms.

Therefore, a new method is needed to enable the learning of weight vectors to improve the distinction between histograms.

A problem to be solved by the present invention is to learn a weight assignment function based on statistical information of visual words in a sample image class and calculate a distance between images using a weight vector of an image class calculated on the basis of a weight assignment function And an apparatus.

An image storage unit for storing each of a plurality of images as visual word vectors and classifying each image into at least one image class and managing the weight; A weight assignment function learning unit for generating a weight assignment function mapped to each of the image classes and a weight value corresponding to importance indexes of visual words for each image class on the basis of the weight assignment function to calculate a visual word weight vector of each image class And an image class weight calculation unit.

The weight assignment function may be a function of assigning a higher weight value as the time words become more important.

Wherein the weighting function learning unit decides a certain number of image classes as a learning class and determines a function that maximizes an objective function based on the images included in the learning class as the weighting assignment function, May be a function of collecting vectors of different classes and moving vectors of different classes away from each other.

The importance index of the visual word may be an index related to the appearance frequency of each visual word among the images included in each image class.

The statistical index includes any one of standard deviation and variance, and the importance of the statistical index and the visual word may be inversely proportional.

The image class weight calculation unit compares the visual word vectors of the images included in the arbitrary image class and calculates the importance index of each visual word based on the appearance frequency of each visual word in the arbitrary image class.

Wherein the weight learning apparatus further comprises a search unit for calculating a distance between each of the query image and the stored image stored in the image storage unit and outputting a stored image having a minimum distance from the query image as a search result, The distance between two images can be calculated using the visual word weight vector of the image class in which the stored image is classified.

According to another embodiment of the present invention, there is provided a method of learning a weight of an image class, the method comprising: storing each of a plurality of images as a visual word vector, classifying each image into at least one image class, Generating a weight assignment function by mapping a weight to the importance index; extracting a weight value corresponding to the importance index of the visual word for each image class based on the weight assignment function; and calculating a visual word weight vector of each image class And calculating a distance between each of the plurality of images and the query image based on the visual word weight vector of each image class.

The step of generating the weighting assignment function may generate the weighting assignment function in such a manner that a higher weighting value is assigned to a more important visual word.

The importance index of the visual word may be an index related to the appearance frequency of each visual word among the images included in each image class.

The statistical indicator may include either standard deviation or variance.

Wherein the step of calculating the distance comprises calculating a first distance between the first image and the query image of the plurality of images using the visual word weight vector of the image class in which the first image is classified .

The weight learning method may further include outputting an image having a minimum distance from the query image among the plurality of images as a search result.

According to the embodiment of the present invention, since the weight assignment function has only a small number of parameters, it is simple to learn the weight assignment function. Therefore, according to the embodiment of the present invention, it is possible to solve the difficulty of learning according to the high-dimensional representation of the existing image search.

According to the embodiment of the present invention, different weights are learned for each class of images. Therefore, according to the embodiment of the present invention, it is possible to solve the problem that the conventional image search can not learn different weights for each class due to the difficulty of learning.

According to the embodiment of the present invention, the database image can calculate the weight vector using the learned weight model by calculating only a simple statistical value within the class. Therefore, according to the embodiment of the present invention, the weight vector can be calculated only with a very small amount of calculation, and the accuracy can be remarkably improved.

FIG. 1 is a diagram for explaining a method of expressing an image according to an embodiment of the present invention with visual words.
2 is a block diagram of an image retrieving apparatus according to an embodiment of the present invention.
3 is an exemplary graph of a weight assignment function according to an embodiment of the present invention.
4 is a diagram for explaining a method for obtaining a weight allocation function according to an embodiment of the present invention.
5 is a diagram illustrating a method of obtaining a visual word weight vector of an image class according to an embodiment of the present invention.
6 is a flowchart of an image retrieval method according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and similar parts are denoted by like reference characters throughout the specification.

Throughout the specification, when an element is referred to as "comprising ", it means that it can include other elements as well, without excluding other elements unless specifically stated otherwise.

FIG. 1 is a diagram for explaining a method of expressing an image according to an embodiment of the present invention with visual words.

Referring to FIG. 1, a Bag-of-Words model considers a document as a set of words, and displays the frequency of occurrences of words included in the document as a histogram to vectorize. The similarity between documents is calculated by measuring the distance between the histograms of the document.

In the image classification and search field, a word bag model using a visual word is used instead of a text word. That is, as shown in FIG. 1, the image is vectorized by displaying the frequency of occurrence of each visual word as a histogram.

A visual word is a 128-dimensional vector representation of image patches using a descriptor, for example, a SIFT descriptor. A single image is represented by several visual words (128-dimensional vectors).

In order to express the image as a vector of one fixed dimension using the word bag model, a codebook which can be called a visual word dictionary is used. The codebook spatters the visual words extracted from the multiple learning images into a 128-dimensional space and generates K visual vocabularies through arithmetic clustering based on the Euclidean distance between them. Based on a visual word dictionary with K visual vocabularies, a single image is represented by a histogram vector that indicates how many visual vocabularies occurred.

Image retrieval requires a vast number of visual vocabularies to find objects such as objects that the query image contains. For example, image retrieval is performed based on a visual word dictionary that contains one million visual vocabularies (K = million).

FIG. 2 is a block diagram of an image retrieval apparatus according to an embodiment of the present invention. FIG. 3 is an exemplary graph of a weight allocation function according to an embodiment of the present invention. FIG. And a method of obtaining a weight allocation function.

Referring to FIG. 2, the image search apparatus 100 generates a visual word weight vector for each image class based on a weighting mapping function (WMF). When the query image is input, the image search apparatus 100 calculates the distance between the query image and the stored image by applying the visual word weight vector of the stored image.

The image search apparatus 100 includes an image storage unit 110, a weight allocation function learning unit 130, an image class weight calculation unit 150, and a search unit 170.

]로 표현된다. The image storage unit 110 stores a plurality of images. The image storage unit 110 can classify and manage the stored images according to a certain criterion. Each image is classified into at least one image class, and each image class includes a plurality of images. The image is a visual word vector representing the appearance frequency of K visual words included in the visual word dictionary [

].

The weight assignment function learning unit 130 generates a weight assignment function by mapping a weight to an importance index of a visual word. The weight assignment function is a function for assigning a high weight to a visual word having a high importance.

)나 분산일 수 있다. 한 시각 단어에 대한 클래스내 출현 빈도 표준편차는 클래스내의 이미지들에서 그 시각 단어가 가장 많이 출현한 빈도수로 모두 나누는 방식으로 그 크기를 정규화한 후, 표준편차를 계산할 수 있다. 즉, 출현 빈도 편차가 적은 시각 단어는, 이미지에서 해당 시각 단어가 비슷한 빈도로 출현하므로, 해당 이미지 클래스의 특징을 포함하는 중요한 시각 단어로 볼 수 있다. 따라서, 가중치 할당 함수 학습부(130)는 출현 빈도 편차가 적은 시각 단어에 높은 가중치를 할당하는 가중치 할당 함수를 생성한다.The weight assignment function may be an increasing function form or a decreasing function form as shown in FIG. The decreasing function of the weight assignment function has a larger weight value as the value of the horizontal axis is smaller. Therefore, in the weight allocation function of the decreasing function shape, the index of the abscissa is an index whose importance of the visual word is lower as the value is larger. For example, the abscissa indicates the appearance frequency standard deviation of visual words

) Or may be dispersed. The standard deviation of the appearance frequency in a class for a visual word can be calculated by normalizing the size of the image in the class by dividing it by the frequency at which the visual word occurs most frequently. In other words, a visual word having a small deviation in appearance frequency appears as an important visual word including the characteristic of the corresponding image class because the visual word appears in the image at a similar frequency. Accordingly, the weighting assignment function learning unit 130 generates a weighting assignment function for assigning a high weighting to a visual word having a small appearance frequency deviation.

The weighting function can be created in various ways. For example, the weight assignment function learning unit 130 determines a certain number of image classes as a learning class (sample class). Referring to FIG. 4, the weight assignment function learning unit 130 determines a function that maximizes the objective function based on the images included in the learning class as a weight assignment function. The objective function is a function that collects vectors of the same class and distances vectors of different classes. The objective function is defined as Equation (1).

는 가중치 할당 함수의 매개변수들, D^tr은 학습 클래스들, c는 하나의 클래스, q는 c 클래스내에서 샘플링된 질의 이미지, i는 c클래스내에 있는 임의의 이미지를 의미한다. 수학식 1에서 WG(

)는

의 변화에 따라 얼마나 클래스 내부의 이미지들이 모이는지, BC(

)는

의 변화에 따라 얼마나 클래스간 이미지들이 서로 흩어지는지를 정량적으로 계산한다. 클래스 내부의 이미지들이 모일수록, WG(

)는 큰 값을 가지게 되고, 클래스간 이미지들이 서로 흩어질수록, BC(

)는 큰 값을 가지게 된다. 목적 함수를 최대화함으로써 최적의 매개변수를 갖는 가중치 할당 함수를 갖게 된다.In Equation (1)

Are the parameters of the function, the variable weight assigned, D ^tr is the learning class, c is a class, q is the query image sampled in the class c, i refers to any image in the class c. In Equation (1), WG

)

, How the images in the class are gathered, BC

)

Quantitatively calculate how the inter-class images are scattered with each other according to the change of the class. The more images the class contains, the more the WG

) Has a large value, and as the inter-class images are scattered, BC (

) Has a large value. By maximizing the objective function, we have a weight assignment function with optimal parameters.

]에 대응하는 가중치 값을 포함하는 벡터이다. 시각 단어 가중치 벡터는 이미지 클래스마다 계산된다.The image class weight calculation unit 150 calculates a visual word weight vector of each image class based on the weight assignment function. The visual word weight vector is expressed by the visual word vector [

]. ≪ / RTI > The visual word weight vector is calculated for each image class.

The image class weight calculation unit 150 compares the visual word vectors of the images included in the image class and calculates an statistical index related to the appearance frequency of each visual word. Statistical indicators can vary, and future standard deviations are used as examples.

)를 가중치 할당 함수에 매핑한다. 이미지 클래스 가중치 계산부(150)는 각 시각 단어에 대응하는 가중치 값을 추출한다. 이미지 클래스 가중치 계산부(150) 수학식 2와 같이, K개의 시각 단어 각각에 대응하는 가중치 값을 가중치 벡터로 생성한다. 이와 같이, 이미지 클래스 가중치 계산부(150)는 시각 단어들의 출현 빈도 통계를 기초로, 이미지 클래스마다 중요한 시각 단어를 간단히 계산할 수 있다.The image class weight calculation unit 150 calculates the appearance frequency standard deviation (

) To a weight assignment function. The image class weight calculation unit 150 extracts a weight value corresponding to each visual word. The image class weight calculation unit 150 generates a weight value corresponding to each of the K time words as a weight vector, as shown in Equation (2). In this manner, the image class weight calculation unit 150 can easily calculate important visual words for each image class based on the appearance frequency statistics of visual words.

]로 표현될 수 있다. When a query image is input, the search unit 170 searches an image corresponding to the query image among the stored images stored in the image storage unit 110. The query image is a visual word vector [

].

]의 거리를 계산한다. 이때, 검색부(170)는 저장 이미지가 포함된 이미지 클래스의 시각 단어 가중치 벡터를 기초로 각 시각 단어에 가중치를 부여한다. 여기서, 저장 이미지의 시각 단어 가중치 벡터는 저장 이미지가 속한 클래스의 시각 단어 가중치 벡터이다.The retrieval unit 170 retrieves the visual word vector of the query image and the visual word vector [

] Is calculated. At this time, the search unit 170 assigns weights to the respective visual words based on the visual word weight vector of the image class including the stored image. Here, the visual word weight vector of the stored image is a visual word weight vector of the class to which the stored image belongs.

The search unit 170 calculates the distance between two images using the visual word weight vector as shown in Equation (3).

The search unit 170 compares the query image with the distance of the stored images stored in the image storage unit 110 to search for an image corresponding to the query image. The search unit 170 outputs the stored image having the closest distance.

Conventionally, there is a method such as metric learning as a method of assigning weights to important visual words per class. However, image retrieval based on high-level expression is difficult because of the existing learning framework, because the image is represented by a very large number of dimensions of about one million dimensions. In particular, it is difficult to generate a million-dimensional weight vector for each class because it is computationally expensive, and if there is not enough learning data, it is likely to be over-fitting and each weight vector must be learned for each class at a high cost.

However, the image search apparatus 100 can easily calculate a weight representing the importance of a visual word for each image class, even if the vector is a million-dimensional vector, by using an statistical index (for example, standard deviation) related to the appearance frequency. In addition, the image search apparatus 100 may generate one weight assignment function, and may calculate the visual word weight vector of each image class in correspondence with the weighting function of the statistical index for each image class.

5 is a diagram illustrating a method of obtaining a visual word weight vector of an image class according to an embodiment of the present invention.

Referring to FIG. 5, for example, M images are included in image class 1. Each of the M images is represented by a visual word vector representing the appearance frequency of each visual word.

The image search apparatus 100 calculates an appearance frequency-related statistical index appearing in M images by visual words. Here, the statistical index related to the appearance frequency is information indicating importance of each visual word in the image class. The statistical indicator may be, for example, standard deviation or variance. In an image class, if the appearance frequencies of certain visual words are similar, the visual words are important visual words containing the characteristics of the image class. Conversely, in an image class, if the appearance frequency of a visual word varies from image to image, the visual word may be a visual word that is not significant in the image class.

)의 출현 빈도 표준 편차(

)부터 시각 단어K(

)의 출현 빈도 표준 편차(

)까지 계산한다. The image search apparatus 100 calculates the standard deviation of each visual word in image class 1. That is, the image search apparatus 100 searches the visual word 1 (

) Frequency standard deviation (

) To the visual word K (

) Frequency standard deviation (

).

)를 계산한다.The image search apparatus 100 extracts a weight value corresponding to the standard deviation of each visual word based on the weight assignment function. The image retrieval apparatus 100 calculates the visual word weight vector of image class 1 based on the weight value of each visual word

).

In this manner, the image search apparatus 100 calculates the visual word weight vector of each image class based on the statistical index for each visual word.

6 is a flowchart of an image retrieval method according to an embodiment of the present invention.

Referring to FIG. 6, the image search apparatus 100 generates a weight assignment function by mapping a weight to a significance index of a visual word based on an objective function (S110). The objective function is a function that collects vectors of the same class and distances vectors of different classes. The image search apparatus 100 determines a function that maximizes an objective function as a weight allocation function. The significance index of the visual word is an statistical index related to the appearance frequency of the visual word, for example, the appearance frequency standard deviation or variance.

The image search apparatus 100 calculates a visual word weight vector of each image class based on the weight assignment function (S120). The image retrieval apparatus 100 calculates an index of statistics related to the frequency of appearance of the image class per visual word. Then, the image search apparatus 100 extracts a weight value corresponding to the statistical index using the weight allocation function.

The image search apparatus 100 receives the query image (S130).

The image searching apparatus 100 calculates the distance between the query image and each stored image (S140). At this time, the image search apparatus 100 applies the weight to the distance between the visual words using the visual word weight vector of the image class including the stored image. That is, the distance between the important visual words is greater than the distance between the non-critical visual words.

The image search apparatus 100 outputs a stored image having the closest distance as a search result (S150).

As described above, according to the embodiment of the present invention, since the weight assignment function has only a small number of parameters, it is simple to learn the weight assignment function. Therefore, according to the embodiment of the present invention, it is possible to solve the difficulty of learning according to the high-dimensional representation of the existing image search.

The embodiments of the present invention described above are not implemented only by the apparatus and method, but may be implemented through a program for realizing the function corresponding to the configuration of the embodiment of the present invention or a recording medium on which the program is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, It belongs to the scope of right.

Claims (13)

An apparatus for learning a weight of an image class based on a visual word,
An image storage unit that stores each of the plurality of images as a visual word vector, classifies each image into at least one image class,
A weight assignment function learning unit for generating a weight assignment function in which a weight is mapped to a significance index of a visual word, and
An image class weight calculation unit for extracting a weight value corresponding to an importance index of a visual word for each image class based on the weight assignment function and calculating a visual word weight vector of each image class,
Lt; / RTI >
The image class weight calculation unit
And compares the visual word vectors of images included in the arbitrary image class and calculates the importance index of each visual word based on the appearance frequency of each visual word in the arbitrary image class. delete The method of claim 1,
The weight assignment function learning unit
Determining a certain number of image classes as a learning class, determining a function that maximizes an objective function based on the images included in the learning class as the weight assignment function,
Wherein the objective function is a function of collecting vectors of the same class and distancing vectors of different classes from each other. The method of claim 1,
The importance index of the visual word
A weight learning apparatus, which is an statistical index related to occurrence frequency of each visual word among images included in each image class. 5. The method of claim 4,
Wherein the statistical indicator includes any one of standard deviation and variance,
Wherein the importance of the statistical index and the visual word is in inverse proportion. delete The method of claim 1,
Further comprising a search unit for calculating a distance between each of the query images and each of the stored images stored in the image storage unit and outputting a stored image having a minimum distance from the query image as a search result,
Wherein the searching unit calculates a distance between two images using a visual word weight vector of an image class in which each stored image is classified. delete delete delete delete delete delete

Images