CN104331717B - The image classification method that a kind of integration characteristics dictionary structure is encoded with visual signature - Google Patents

Abstract

The invention discloses an image classification method integrating feature dictionary structure and visual feature coding, which comprises the following steps: visual feature extraction; feature dictionary learning; visual feature coding; spatial fusion of feature coding; training and classification. The invention can obtain more accurate image feature representation and improve the accuracy of image classification. Furthermore, by integrating the structural information in the feature dictionary into the visual feature encoding process, a more discriminative representation of image features is obtained, thus enabling more efficient classification of images. The invention realizes efficient and accurate image classification, and thus has high use value.

Description

An Image Classification Method Integrating Feature Dictionary Structure and Visual Feature Coding

technical field

The present invention relates to the field of image classification, especially an image classification method based on a codebook model (Bag-of-Words, BoW) that integrates feature dictionary structure and visual feature encoding

Background technique

With the continuous rapid development of information technology, various fields are generating various types of data at an alarming rate every day, including text, images, videos, music, etc. Among the rich and colorful data information, images are popular because of their intuitive and vivid performance, rich content, large amount of information, and convenient storage and transmission, and have become one of the most important information carriers in the 21st century. Especially with the increasing popularity of mobile devices with camera functions such as cameras, mobile phones, and tablets, and the rise of social networks, people have more and more ways to obtain images, which further promotes the rapid growth of image data, and quickly and accurately find the required information. Images and efficient management therefore become more and more difficult. People urgently hope that computers can help humans analyze the semantics contained in the massive images on the Internet, and fully understand the content expressed by the images, so as to manage, classify and label images more effectively, or retrieve images of interest.

Image classification, as one of the most important basic technologies for computer understanding of images, has been widely studied by various research institutions in academia and industry, and has been used as an important topic in authoritative journals and important academic conferences at home and abroad. It is an important topic in the field of computer vision. very important research topic. Image classification refers to the process of intelligently classifying images into a set of defined categories according to certain classification criteria, including object recognition, scene semantic classification, behavior recognition, etc. Image classification has become an important technical means to study image semantic understanding. Scientific researchers have gradually realized the importance of the above problems and continue to analyze them in depth. In recent years, the codebook model has brought new inspiration to the high-level semantic representation of images. Image classification with the codebook model as the key technology has achieved certain results, but there are still many research points that have not been covered, and there is still huge room for breakthroughs. The research on the image classification method based on the codebook model has become a cutting-edge hotspot in many intersecting fields such as artificial intelligence, computer vision, machine learning and data mining, and plays an important role in actively promoting social informatization. While creating irreplaceable social value, there are still many key technical issues in this field that have not been resolved, and there are still many functions that need to be further improved. Therefore, how to use the codebook model to understand and describe the high-level semantics of images more effectively, The research of realizing image classification more flexibly has far-reaching significance.

Contents of the invention

Purpose of the invention: the technical problem to be solved by this invention is to provide a kind of image classification method integrating feature dictionary structure and visual feature coding, and use the distribution information of visual words in the feature dictionary to assist visual feature coding, so as to make The encoding result is more discriminative, thus improving the accuracy of image classification.

In order to solve the above-mentioned technical problems, the present invention discloses an image classification method integrating feature dictionary structure and visual feature coding, which includes the following steps:

Step 1, extract the visual features of the image: perform local sampling on each image to obtain a set of area blocks, extract the visual features of each area, and obtain the set of visual features corresponding to each image, which is called the overall set of visual features of all images is the visual feature set of all images, denoted as set X;

Step 2, feature dictionary learning: take the set X as input, and use the feature dictionary learning method to obtain a feature dictionary composed of a group of representative visual words;

Step 3, visual feature coding: express each visual feature of each image as a linear combination of visual words, each visual word corresponds to a coefficient, and this group of coefficients is called the coding of visual features;

Step 4, spatial fusion of visual feature encoding: take the encoding of all visual features of each image as input, and use statistical methods to represent each image as a vector, which is the image feature representation of the corresponding image;

In step 5, the encoding of each image obtained in step 4 is used as an input, and the classification model is used for training and classification to obtain a classification result.

Step 1 specifically includes the following steps:

Local sampling is performed on each image I, and dense sampling is performed by means of equal steps to obtain several regional blocks of the same size, and a visual feature is extracted for each regional block, and a visual feature representing the local block is obtained by using the visual feature extraction method , Visual feature extraction methods include: Histogram of Oriented Gradient (HOG), Scale-invariant feature transform (Scale-invariant feature transform, SIFT) and so on. Get the visual feature set LFS _I of the image I, and finally get the overall X=[x ₁ ,x ₂ ,…,x _N ]∈R ^d×N of the visual feature set of all images, where d represents the dimension of the visual feature, and its The size is determined by the visual feature extraction technology, N represents the total number of visual features of all images, x _i represents the i-th visual feature, and i takes a value from 1 to N.

Step 2 specifically includes the following steps:

Take the set X as input, and use the feature dictionary learning method to obtain a feature dictionary composed of a group of representative visual words. The feature dictionary is recorded as: B=[b ₁ ,b ₂ ,…,b _M ]∈R ^{d ×M} , where M is the number of visual words; b _j is a column vector of dimension d, representing the jth visual word, and j takes a value from 1 to M. Commonly used feature dictionary learning methods include: k-means, K-SVD, etc.

Step 3 specifically includes the following steps:

In this step, each visual feature in the set X is encoded one by one. For the visual feature x _i , the encoding process is as follows:

First, select the p nearest neighbor visual words of x _i from the feature dictionary B, that is, the p visual words with the smallest distance to the visual feature x _i , record the feature dictionary composed of these p visual words as B _i , p The value is 1~M, and i takes the value 1~N.

Secondly, obtain the matrix D _i represented by the distance between the visual words in the feature dictionary B _i and the column vector d _i represented by the distance between the visual feature x _i and the visual words in the feature dictionary B _i , i takes the value 1 ~N. The elements of the mth row and s column of the matrix D _i are the distances between the corresponding visual words in B _i , m, s=1, 2,..., p; the nth component d _in of d _i represents the visual feature x _i and The distance between the nth visual word in Bi, _n =1,2,...,p. The distance calculation formula is: σ is a smoothing parameter that controls the descending speed of the weight, σ>0. dist( _xi ,B _i )＝[dist( _xi ,b _i1 ),dist( _xi ,b _i2 ),…,dist( _xi ,b _ip )] ^T , b _il represents the lth vision of Bi words, l=1,2,…,p; each component Indicates the distance between the visual feature x _i and the visual word b _il ; max(dist( _xi ,B _i )) indicates the maximum component of the vector dist( _xi ,B _i ), so that the value range of the component in d _i is (0,1]. When calculating the distance between a visual word and other visual words, the same strategy is used. In order to speed up the solution of D _i , the distance between each visual word in B is calculated at one time. Matrix D. Then D _i is a sub-matrix of D, and different D _i can be obtained by directly indexing D, i=1, 2,...,N.

Third, take x _i , d _i , D _i , B _i and two parameters λ and β as input, λ, β≥0, minimize the following formula to get the encoding of _xi on B _i

Restrictions:

in Represents the dot product, that is, the components corresponding to the two vectors are multiplied to obtain a new vector; solve to obtain the encoding result of _xi in the p visual words

Finally, to encode The components in are sorted to get the k largest coding coefficients and its corresponding feature dictionary composed of k visual words k=1,2,...,p, then the coding z _i of the visual feature x _i is an M-dimensional vector, and the vector is the same as The corresponding components are The rest of the components are set to 0.

Step 5 specifically includes the following steps:

Consider the spatial statistical information of each visual feature in each image, use a three-layer spatial pyramid matching model (Spatial Pyramid Matching, SPM), take the encoding of all visual features of an image I as input, and combine the maximum convergence technology, then The spatial pyramid outputs a vector with a dimension of (2 ⁰ +2 ² +2 ⁴ )*M, which is the image feature representation of I.

Step 6 specifically includes the following steps:

After obtaining image feature representations for each image, they can be used for training and classification. The set formed by the image feature representation of all images is divided into two parts: training set and test set. The training set is used to train the classification model, and the trained model is used to classify the test set. Usually support vector machine (Support Vector Machine, SVM) is used as the classifier model.

The present invention is aimed at the image visual feature encoding method in the field of image classification. The present invention has the following characteristics: 1) when the present invention encodes visual features, it not only considers the relationship between visual features and visual words, but also considers the relationship between visual words. 2) the visual feature coding obtained by the present invention is an analytical solution, and does not need an iterative optimization function, so the visual feature coding method of the present invention is fast.

Beneficial effects: the present invention fully considers the structural information of the distribution of visual words in the feature dictionary, and uses this information for coding of visual features, so that the coding of visual features can better reflect the distribution of visual words in the feature dictionary. Therefore, the image feature representation of the image is highly discriminative, thereby improving the accuracy of image classification.

Description of drawings

The advantages of the above and/or other aspects of the present invention will become clearer as the present invention will be further described in detail in conjunction with the accompanying drawings and specific embodiments.

Fig. 1 is the flow chart of the present invention.

Figure 2 is a schematic diagram of visual feature extraction

Figure 3 is a flow chart of encoding a visual feature

Fig. 4 is a schematic diagram of a three-layer space pyramid structure.

detailed description

As shown in Figure 1, the present invention discloses an image classification method integrating feature dictionary structure and visual feature coding, which includes the following steps:

Step 1, extract the visual features of the image: perform local sampling on each image to obtain a set of area blocks, extract the visual features of each area, obtain the visual feature set corresponding to each image, and combine the visual feature sets of all images as a whole denoted as set X;

Step 2, feature dictionary learning: take the set X as input, and use the feature dictionary learning method to obtain a feature dictionary composed of a group of representative visual words;

Step 3, visual feature encoding: express the visual features of each image as a linear combination of visual words, each visual word corresponds to a coefficient, and obtain a visual feature encoding set;

Step 4, spatial fusion of visual feature encoding: take all visual feature encodings of each image as input, and use statistical methods to represent each image as a vector, which is the image feature representation of the corresponding image;

In step 5, the encoding of each image obtained in step 4 is used as an input, and the classification model is used for training and classification to obtain a classification result.

1. Step 1 includes the following steps:

As shown in Figure 2, for an image I, a number of equal-sized regional blocks are usually extracted from I by means of equal-step dense sampling, and a visual feature is extracted for each regional block, where the visual feature is a d-dimensional vector. Commonly used visual feature extraction methods include: Histogram of Oriented Gradient (HOG), Scale-invariant feature transform (SIFT), etc. Finally, the overall X=[x ₁ ,x ₂ ,…,x _N ]∈R ^d×N of the visual feature set of all images is obtained, where d represents the dimension of visual features, N represents the total number of visual features of all images, x _i represents the i-th visual feature, and i takes a value from 1 to N. X is used as input in step 2 to learn a feature dictionary.

2. Step 2 includes the following steps:

In this step, the set X is used as input, and the feature dictionary B=[b ₁ ,b ₂ ,…,b _M ]∈R ^d×M composed of M d-dimensional visual words is obtained by using the feature dictionary learning method, where M is the visual The number of words; b _j is a column vector of dimension d, representing the jth visual word, and j takes a value from 1 to M. Taking the k-means method as an example, use k-means to cluster the set X into M classes, and the center of each class is a visual word.

3. Step 3 includes the following steps:

This step encodes each visual feature in the set X one by one.

The flow chart shown in Figure 3 describes a visual feature encoding process. For the visual feature x _i , select the p nearest neighbor visual words in the feature dictionary B obtained in step 2 of the visual feature x _i , that is, The p visual words with the smallest distance between features x _i , p takes a value of 1 to M, and the feature dictionary composed of these p visual words is B _i , i takes a value of 1 to N, and finds each visual word in the feature dictionary B _i The matrix D _i represented by the distance between, the element of the mth row s column of the matrix D _i is the distance between the corresponding visual words in B _i , m, s=1,2,...,p, and then calculate the visual features The column vector d _i represented by the distance between x _i and each visual word in the feature dictionary B _i , the nth component d _in of d _i represents the distance between the visual feature x _i and the nth visual word in B _i , n= 1,2,...,p; take x _i , d _i , D _i , B _i and two parameters λ and β as input, λ, β≥0, minimize the following formula to get the encoding of _xi on B _i

Restrictions:

in Represents the dot product, that is, the components corresponding to the two vectors are multiplied to obtain a new vector; solve to obtain the encoding result of _xi in the p visual words final pair encoding The components in are sorted to get the k largest coding coefficients and its corresponding k visual words k=1,2,...,p, then the encoding z _i of x _i is an M-dimensional vector, in which the The corresponding components are The rest of the components are set to 0.

The specific encoding method of visual feature x _i on B is as follows:

Input: image visual feature _xi , feature dictionary B=[b ₁ ,b ₂ ,…,b _M ]∈R ^d×M , M is the number of visual words in B and the dimension of encoding of _xi on B. The number p of nearest neighbor words of x _i , parameters k, λ and β.

Encoding process:

1) Calculate the M-dimensional vector d′ _i represented by the distance between the visual feature x _i and all visual words;

2) Sort the components in d′ _′ in ascending order, and select a set B _i composed of p visual words with the smallest distance, and the corresponding distance d _i ;

3) Find the matrix D _{i represented by the distance between each visual word in B i ;}

4) Calculate the code according to the following formula

Ψ＝(x _i 1 ^T -B _i ) ^T (x _i 1 ^T -B _i )

Θ＝Ψ+λ*diag ² (d _i )+βD _i

α＝-(1 ^T Θ ^-1 1)

where diag(d _i ) means that the diagonal vector is the diagonal matrix of di. Here 1 means a column vector whose components are all 1;

5) yes The components in are sorted in descending order to get the k largest coding coefficients and its corresponding feature dictionary composed of k visual words Then the encoding z _i of x _i is an M-dimensional vector, and the vector is the same as The corresponding components are The rest of the components are set to 0. Normalize z _i using the formula z _i =(1 ^T z _i ) ^-1 z _i ;

Output: Encoding _{zi of visual feature xi .}

4. Step 4 includes the following steps:

As shown in Figure 4, it is a three-layer spatial pyramid matching model. After obtaining all the visual feature encodings of an image, the spatial pyramid matching model (Spatial Pyramid Matching, SPM) is used, combined with the spatial convergence of MaxPooling. The technology uses all the visual feature codes of an image as input to obtain a vector, which is the image feature representation of the image. The specific operation is: take the center of the image as the origin, use different scales, and recursively divide it into several sub-regions. For example, in Figure 4, a three-layer spatial pyramid matching model is used, and there are a total of 2 ⁰ +2 ² +2 ⁴ =21 sub-regions. For the a-th area, a takes the value of 1 to 21, and the code of this area is obtained by using the maximum convergence technique This formula indicates that there are t visual features in this image sub-region; a _t represents the encoding of the h-th visual feature in this region, and h takes a value from 1 to t; z′ _a is a column vector with the same dimension as z _ah , That is, its dimension is M, and its qth component is a matrix The maximum value of the corresponding row of q takes a value from 1 to M. Further normalize z' _q , for example, use 2-norm normalization to get z' _q = z' _q /||z' _q || ₂ . Finally, the codes of each sub-region are sequentially spliced to obtain the image feature representation of the image.

5. Step 5 includes the following steps:

After obtaining the image feature representations of all images, use the image feature representations of the images in the training set to train the SVM classification model, and then use the trained SVM model to classify the image feature representations of the images in the test set.

Example 1

This embodiment includes the following parts:

1. First reduce the image to a size of no more than 300×300, and convert it into a grayscale image, and then use a dense sampling strategy to extract image blocks of 16×16 pixels from the image, every 6 pixels, for each Image patches extract a SIFT feature. Therefore, an image may contain hundreds or thousands of features, depending on the image block size and spacing when extracting features.

2. First use k-means to gather all image visual features into M clusters, and the center of each cluster represents a visual word. Set the number of nearest neighbor visual words p, the number of dense nearest neighbor visual words k, the distance smoothing parameter σ, and the regularization parameters λ and β. Encode each visual feature.

3. Use the spatial golden matching model and the maximum fusion technology to combine all the visual feature codes of each image into a vector as the image feature representation of the image. And use a support vector machine model to train and classify the images.

Example 2

For the image extraction dimension of 128 visual features, the size of the feature dictionary and the number of visual words are set to 1024. Set p and k to 10 and 5, respectively. Other parameter settings also include: λ=10 ^-4 , β=10 ^-4 . Use 3 layers of spatial pyramid matching and maximum convergence technology. A 21504-dimensional image feature representation is obtained for each image. Use the image feature representation of the image as the training set to train the SVM classification model, and use the trained model to classify the image feature representation of the image as the test set to obtain the final classification result.

The present invention provides an image classification method that integrates feature dictionary structure and visual feature coding. There are many methods and approaches to realize this technical solution. The above description is only a preferred embodiment of the present invention. Those of ordinary skill may make some improvements and modifications without departing from the principle of the present invention, and these improvements and modifications shall also be regarded as the protection scope of the present invention. All components that are not specified in this embodiment can be realized by existing technologies.

Claims (3)

1. the image classification method that a kind of integration characteristics dictionary structure is encoded with visual signature, it is characterised in that including following step Suddenly：

Step 1, the visual signature of image is extracted：Local sampling is carried out to each image, one group of region unit is obtained, every piece of area is extracted The visual signature in domain, obtains the corresponding visual signature set of each image, claims the visual signature set of all images generally The visual signature collection of all images, is designated as set X；

Step 2, characteristics dictionary learns：Using set X as input, using characteristics dictionary learning method, obtain that there is representative by one group Property vision word composition characteristics dictionary；

Step 3, visual signature is encoded：Each visual signature of each image is expressed as the linear combination of vision word, each Vision word one coefficient of correspondence, the coefficient is called the coding of visual signature；

Step 4, the space of visual signature coding is converged：Input is encoded to all visual signatures of each image, system is used Meter method, a vector is expressed as by each image, and the vector is exactly the image feature representation of correspondence image；

Step 5, the coding of each image step 4 obtained is trained and classified using disaggregated model as input, is obtained Classification results；

Step 1 comprises the following steps：

Local sampling is carried out for image I, sampling every time obtains a region unit, and each region unit extracts a visual signature, Obtain image I visual signature set LFS_I, finally give the visual signature set X=[x of all images₁,x₂,…,x_N]∈R^{d ×N}, wherein, d represents the dimension of visual signature, and N represents the sum of the visual signature of all images, x_iI-th of visual signature is represented, 1~N of i values；

Step 2 comprises the following steps：

Using set X as input, using characteristics dictionary learning method, the spy being made up of one group of representative vision word is obtained Dictionary is levied, this feature dictionary is designated as：B=[b₁,b₂,…,b_j,…,b_M]∈R^d×M, wherein M is the number of vision word；b_jIt is One dimension d column vector, represents j-th of vision word, 1~M of j values；

Step 3 comprises the following steps：

For visual signature x_i, choose visual signature x_iThe characteristics dictionary B obtained by step 2 in p arest neighbors vision list Word, i.e., with visual signature x_iThe minimum p vision word of distance, p 1~M of value remember the feature that this p vision word is constituted Dictionary is B_i, i 1~N of value obtain characteristics dictionary B_iIn matrix D represented by the distance between each vision word_i, matrix D_i's The element of m rows s row is characterized dictionary B_iIn the distance between m-th and s-th vision word, m, s=1,2 ..., p；Count again Calculate visual signature x_iTo characteristics dictionary B_iEach vision word the column vector d that represents of distance_i, d_iN-th of component d_inExpression is regarded Feel feature x_iWith B_iIn the distance between n-th vision word, n=1,2 ..., p, with x_i, d_i, D_i, B_iIt is with β with two parameter lambdas Input, λ, β >=0 minimizes following formula, obtains x_iIn B_iOn coding

Constraints：

WhereinRepresent that the corresponding component of the vector of dot product, i.e., two is multiplied and obtain a new vector；Solution obtains x_iRegarded at this p Feel the coding result of wordFinally to codingIn component sequence, obtain k maximum code coefficientAnd its it is corresponding The characteristics dictionary that k vision word is constitutedK=1,2 ..., p, then visual signature x_iCoding z_iIt is the vector of a M dimension, In vector withCorresponding component isRemaining component is all set to 0.

2. according to the method described in claim 1, it is characterised in that step 5 comprises the following steps：Matched using spatial pyramid Model, a vector is merged into as the image feature representation of the image using the coding of all visual signatures of each image.

3. method according to claim 2, it is characterised in that step 6 comprises the following steps：The image for obtaining all images is special Levy after the set for representing constituted, the set be divided into training set and test set two parts, training set is used for train classification models, Test set is classified with the model trained.