CN107085731A - An Image Classification Method Based on RGB‑D Fusion Feature and Sparse Coding - Google Patents

Abstract

The invention discloses an image classification method based on RGB-D fusion features and sparse coding. The specific implementation steps are: 1) extracting dense SIFT features and PHOG features of color images and depth images; (2) extracting features from the two images The feature fusion is performed in the form of linear series, and finally four different fusion features are obtained; (3) the K-means++ clustering method is used to cluster the different fusion features to obtain four different visual dictionaries; (4) in each Locally constrained linear coding is performed on a visual dictionary to obtain different image representation sets; (5) use linear SVM to classify different image representation sets, and use voting decision method to determine the final classification for multiple classification results obtained. The classification accuracy of the invention is high.

Description

An Image Classification Method Based on RGB-D Fusion Feature and Sparse Coding

technical field

The invention relates to technical fields such as computer vision and pattern recognition, and in particular to an image classification method based on RGB-D fusion features and sparse coding.

Background technique

Today's society is an era of information explosion. In addition to a large amount of text information, the multimedia information (pictures, videos, etc.) that human beings come into contact with is also growing explosively. In order to utilize, manage and retrieve images accurately and efficiently, it is necessary for the computer to understand the image content accurately in the way humans understand. Image classification is an important way to solve the problem of image understanding, and it plays an important role in promoting the development of multimedia retrieval technology. The acquired images may be affected by many factors such as viewpoint changes, lighting, occlusion, and background, which makes image classification a challenging problem in the fields of computer vision and artificial intelligence. Many image feature description and classification techniques are therefore be developed rapidly.

In the current image feature description and classification technology, the main algorithm is based on the Bag-of-Feature (BOF) algorithm. S. Lazebnik proposed a BOF-based spatial pyramid in the article "Spatial pyramid matching for recognizing naturalscene categories". Matching (Spatial Pyramid Matching, SPM) framework, this algorithm overcomes the spatial information lost in the BOF algorithm, and effectively improves the accuracy of image classification. However, BoW-based algorithms all use vector quantization (Vector Quantization, VQ) to encode features, and this hard-coding mode does not consider the relationship between visual words in the visual dictionary, resulting in a relatively large error after image feature encoding. Large, which in turn affects the performance of the entire image classification algorithm.

In recent years, with the maturity of the sparse coding (Sparse Coding, SC) theory, this theory has also become the most popular technology in the field of image classification. In the article "Linear spatial pyramid matching using sparse coding for image classification", Yang proposed a sparse coding spatial pyramid matching (Sparse coding Spatial Pyramid Matching, ScSPM), which replaces the hard allocation mode with sparse coding, which can optimize the visual The weight coefficient of the dictionary, so as to better quantify the image features, so that the accuracy and efficiency of image classification have been greatly improved, but due to the over-complete codebook, several features with a high similarity may be completely different Expressed differently, the stability of the ScSPM model is not good. Wang et al. improved ScSPM and proposed Locality-constrained Linear Coding (LLC) in the article "Locality-constrained linear coding for image classification", pointing out that locality is more important than sparsity. A base represents a feature descriptor, and similar feature descriptors get similar codes by sharing their local bases, which greatly improves the instability of ScSPM.

The above methods are all aimed at the classification of color images, ignoring the depth information of objects or scenes, and depth information is one of the important clues of image classification, because depth information can easily separate the foreground from the background according to the distance, and can directly reflect the object or 3D information of the scene. With the rise of Kinect, the acquisition of depth images has become easier and easier, and algorithms for image classification combined with depth information have also become popular. In the article "Kerneldescriptors for visual recognition", Liefeng Bo et al. proposed to extract the features of the image from the perspective of the kernel method and perform image classification. However, the defect of this algorithm is that the object must first be modeled in 3D, which is very time-consuming. The real-time performance of the algorithm is not high; N.Silberman first used the Scale Invariant Feature Transform (SIFT) algorithm to extract depth images (Depth images) and color images ( RGB image) features, and then perform feature fusion, and then use SPM encoding for image classification; A. Janoch used the Histogram of Oriented Gradient in the article "A Category-Level 3D Object Dataset: Putting the Kinect to Work" (Histogram of Oriented Gradient , HOG) algorithm to extract features from depth images and color images respectively, and achieve final image classification after feature fusion; MirdaniesM et al. will extract RGB images The SIFT features of the deep image are fused with the SURF features of the depth image, and the fused features are used for target classification. These algorithms are the fusion of RGB features and depth features in the feature layer, which can effectively improve the accuracy of image classification. However, this type of algorithm also has certain defects, that is, the features extracted from the RGB image and the depth image are a single feature, and when a single feature is used, there is insufficient information extraction from the image, and the obtained fusion features cannot be obtained. The reason for fully expressing the image content is that the RGB image is easily affected by illumination changes, viewing angle changes, image geometric deformation, shadows and occlusions, etc., and the depth image is easily affected by the imaging device, resulting in holes, noise, etc. in the image. The problem is that a single image feature extraction cannot maintain robustness to all factors in the image, which will inevitably lose the information in the image.

Therefore, it is necessary to design a more accurate image classification method.

Contents of the invention

The technical problem to be solved by the present invention is to provide an image classification method integrating RGB-D fusion features and sparse coding, which has high accuracy and good stability.

In order to solve the problems of the technologies described above, the technical solution provided by the present invention is:

An image classification method based on RGB-D fusion features and sparse coding, including training phase and testing phase:

The training phase includes the following steps:

Step A1. For each sample data, extract the denseSIFT (Scale-invariant feature transform) and PHOG (Pyramid Histogram of Oriented Gradients) features of its RGB image and Depth image (color image and depth image); The number of sample data is n;

Step A2. For each sample data, perform feature fusion on the features extracted from the two images in the form of two-two linear series to obtain four different fusion features; the same fusion features obtained from n sample data form a set, Get four fusion feature sets;

Through the above feature extraction, the denseSIFT and PHOG features of the RGB image, and the denseSIFT and PHOG features of the Depth image; then normalize the obtained features so that all features have similar scales; the present invention reduces the complexity of feature fusion Degree, the features are fused in a pairwise linear series, that is:

f=K ₁ ·α+K ₂ ·β (1)

Where K ₁ and K ₂ are weights corresponding to features, and K ₁ +K ₂ =1, and K ₁ =K ₂ in the present invention. α represents the feature extracted from the RGB image, and β represents the feature extracted from the Depth image; finally four different fusion features are obtained, namely: RGBD-dense SIFT feature, RGB-dense SIFT feature + PHOGD feature, RGB-PHOG feature + D-dense SIFT feature, RGBD-PHOG feature; respectively represent the fusion feature generated by the dense SIFT feature of the RGB image and the Depth image, the fusion feature generated by the denseSIFT feature of the RGB image and the PHOG feature of the Depth image, the PHOG feature of the RGB image and the dense of the Depth image Fusion features generated by SIFT features, fusion features generated by PHOG features of RGB images and Depth images.

Step A3, respectively clustering the fusion features in the four fusion feature sets to obtain four different visual dictionaries;

Step A4. On each visual dictionary, use a locally constrained linear coding model to perform feature coding on the fused features to obtain four different image representation sets;

Step A5, constructing classifiers according to four different fusion feature sets, image representation sets and corresponding class labels of sample data, to obtain four different classifiers.

The testing phase includes the following steps:

Step B1, extracting and fusing features of the image to be classified according to the method in steps A2 to A3 to obtain four fusion features of the image to be classified;

Step B2, on the four visual dictionaries obtained in step A3, use the locally constrained linear coding model to perform feature encoding on the four fusion features obtained in step B1, and obtain four different image representations of the image to be classified;

Step B3, use the four classifiers obtained in step A5 to classify the four image representations obtained in step B2 respectively to obtain four class labels (the four class labels may contain the same class labels, or may all be different class labels Label);

Step B4. Based on the obtained four class labels, use the voting decision method to obtain the final class label of the image to be classified, that is, select the class label with the most votes among the four class labels as the final class label.

Further, in the step A3, the K-means++ clustering method is used to cluster the fusion features in a certain fusion feature set.

The traditional K-means algorithm for building a visual dictionary has the advantages of simplicity and high performance. However, the K-means algorithm itself also has certain limitations. The algorithm selects the initial cluster center randomly, which leads to the clustering result being greatly affected by the initial center point. Falling into a local optimal solution is fatal to the result of correct image classification. Therefore, in view of this deficiency, the present invention uses the K-means++ algorithm to establish a visual dictionary, and adopts a method of probability selection instead of random selection of initial cluster centers. The specific implementation method of clustering for any fusion feature to obtain the corresponding visual dictionary is as follows:

3.1) Form a set of fusion features obtained from n sample data, that is, fusion feature set H _I ={h ₁ ,h ₂ ,h ₃ ,...,h _n }, and set the number of clusters to m;

3.2) Randomly select a point in the fusion feature set H _I ={h ₁ ,h ₂ ,h ₃ ,…,h _n } as the first initial cluster center S ₁ ; set the count value t=1;

3.3) For each point h _i in the fusion feature set H _I ={h ₁ ,h ₂ ,h ₃ ,…,h _n }, h _i ∈ H _I , calculate the distance _d (h _i );

3.4) Select the next initial clustering center S _t+1 :

formula based Calculate the probability that point h _i ' is selected as the next initial cluster center, where h _i '∈H _I ;

Select the point with the highest probability as the next initial cluster center S _t+1 ;

3.5) Make t=t+1, repeat steps (3) and (4), until t=m, that is, m initial cluster centers are selected;

3.6) Use the selected initial cluster centers to run the K-means algorithm, and finally generate m cluster centers;

3.7) Define each cluster center as a visual word in the visual dictionary, and the number of clusters m is the size of the visual dictionary.

Further, in the step A4, a locally constrained linear coding model is used to perform feature coding on the fusion feature, and the model expression is as follows:

In the formula: h _i is the fusion feature in the fusion feature set H _I , that is, the feature vector to be encoded, h _i ∈ R ^d , d represents the dimension of the fusion feature; B=[b ₁ ,b ₂ ,b ₃ …b _m ] is a visual dictionary established by the K-means++ algorithm, b ₁ ~ b _m are m visual words in the visual dictionary, b _j ∈ R ^d ; C=[c ₁ ,c ₂ ,c ₃ …c _n ] is the code The obtained image representation set, where c _i ∈ R ^m is the sparsely coded representation of an image after coding; λ is the penalty factor of LLC; Indicates that elements are multiplied correspondingly; 1 in 1 ^T c _i represents a vector with all elements being 1, then 1 ^T c _i =1 is used to constrain the LLC so that it has translation invariance; d _i is defined as:

where dist(h _i ,B)=[dist(h _i ,b ₁ ),dist(h _i ,b ₂ ),…dist(h _i ,b _m )] ^T , dist(h _i ,b _j ) means h The Euclidean distance between _i and _bj , σ is used to adjust the descent speed of the constraint weights for local locations.

The present invention adopts locality-constrained linear coding (Locality-constrained linear coding, LLC). Because the local position constraints of features must satisfy the sparsity of features, but the sparsity of features does not necessarily satisfy the local position constraints, so locality is more important than sparseness. LLC uses local constraints instead of sparse constraints and can achieve good performance.

Further, in the step A4, an approximate local constrained linear coding model is used to perform feature coding on the fused features; in the process of _solving the coding model in formula (2), the feature vector h to be _coded tends to select the visual dictionary Visual words that are closer in distance form a local coordinate system. Therefore, according to this law, a simple approximate LLC feature encoding method can be used to speed up the encoding process, that is, without solving equation (2), for any feature vector h _i to be encoded, use the k-neighborhood search to select the distance in the visual dictionary B The nearest k visual words are used as the local visual word matrix B _i , and the encoding is obtained by solving a smaller-scale linear system. Its expression is as follows:

in, is the image representation set obtained by approximate encoding, where It is the representation of sparse coding of an image after the approximate coding is completed. According to the analytical solution of formula (4), the approximate LLC feature coding can reduce the computational complexity from o(n ² ) to o(n+k ² ), where k<< n, but the final performance is not much different from LLC feature encoding. The approximate LLC feature encoding method can not only preserve local features, but also ensure the requirement of encoding sparsity, so the approximate LLC model is used for feature encoding in the present invention.

Further, take k=50.

Further, in the step A1, the denseSIFT feature uses a grid to divide the image into feature blocks (blocks) of equal size, and the blocks are overlapped, and the center position of each feature block is used as a feature point, through All the pixels in the same feature block form the SIFT feature descriptor of the feature point (the same feature descriptor as the traditional SIFT feature: gradient histogram), and finally these feature points based on the SIFT feature descriptor form the entire image. dense SIFT features;

The specific steps of PHOG feature extraction are as follows:

1.1) Statistical image edge information; use the Canny edge detection operator to extract the edge profile of the image, and use this profile to describe the shape of the image;

1.2) image is carried out pyramid level segmentation, the block number of image segmentation depends on the number of layers of pyramid level; Image is divided into 3 layers in the present invention, and the 1st layer is whole image; The 2nd layer divides image into 4 subregions, each The size of each area is the same; the third layer divides 4 sub-areas on the basis of the second layer, divides each area into 4 sub-areas, and finally obtains 4×4 sub-areas;

1.3) Extract the HOG feature vector (Histogram of OrientedGridients, directional gradient histogram) of each sub-region in each layer;

1.4) Finally, the HOG feature vectors of the sub-regions in each layer of the image are cascaded (serialized). After the concatenated HOG data are obtained, the data is normalized, and finally the PHOG features of the entire image are obtained.

Further, in the step A5, the classifier adopts a linear SVM classifier.

Further, for the voting decision method in step B4, there will be a problem that different class labels get the most and equal number of votes. In this case, a random selection method is used to randomly select among the class labels with equal number of votes. A class label serves as the final class label.

The beneficial effects of the present invention are:

The present invention selects a plurality of fusion features, which can make up for the shortcoming of insufficient information in a single fusion feature of an image, and effectively improves the accuracy of image classification. The Kmeans++ algorithm is used to build a visual dictionary, and the method of probability selection is used instead of randomly selecting the initial cluster center, which can effectively prevent the algorithm from falling into a local optimal solution. Finally, the voting decision method is used to vote for each class result, and the classification results with large differences are fused, and the final classification performance is determined by the voting decision, which ensures the stability of the result.

Description of drawings

Figure 1 is a flowchart of an image classification method integrating RGB-D fusion features and sparse coding.

Fig. 2 is the LLC feature encoding model in step A5 of the training phase of the present invention.

Fig. 3 is a test image classification decision-making module in step B4 of the test phase of the present invention.

Fig. 4 is the identification confusion matrix of the present invention on the RGB-D Scenes dataset.

detailed description

The present invention will be further described in detail below in conjunction with specific examples and with reference to the detailed drawings. However, the described examples are intended for the understanding of the present invention and do not have any limiting effect on it.

Figure 1 is a flow chart of the image classification system integrating RGB-D fusion features and sparse coding. The specific implementation steps are as follows:

Step S1: extract the dense SIFT features and PHOG features of the RGB image and the Depth image;

Step S2: Perform feature fusion on the features extracted from the two images in series, and finally obtain four different fusion features;

Step S3: use the K-means++ clustering method to cluster different fusion features to obtain four different visual dictionaries;

Step S4: Perform local constrained linear coding on each visual dictionary to obtain different image representation sets;

Step S5: use linear SVM to construct classifiers for different image representation sets, and finally determine the final classification by voting on the classification results of these four classifiers.

Based on the image classification method integrating RGB-D fusion features and sparse coding, the present invention uses experimental data to verify the method of the present invention.

The experimental data set used in the present invention is the RGB-D Scenes data set, which is a multi-view scene picture data set provided by the University of Washington. The data set is composed of 8 classified scenes, with a total of 5972 pictures. Acquired through the Kinect camera, the size is 640*480.

In the RGB-D Scenes dataset, all images are used for experiments and the image size is resized to 256*256. For feature extraction, the dense SIFT feature sampling interval of image extraction in this experiment is set to 8 pixels, and the image block is 16×16. The PHOG feature extraction parameters are set as follows: the image block size is 16×16, the sampling interval is 8 pixels, and the gradient direction is 9. When building a visual dictionary, the dictionary size is set to 200. The libsvm3.12 toolbox of the LIBSVM toolkit is used for SVM classification. 80% of the pictures in the data set are used for training and 20% of the pictures are used for testing.

In this experiment, the method of the present invention is considered from two aspects. First, compare the method of the present invention with the methods of some researchers with higher classification accuracy at present; second, investigate different RGB-D fusion features and The classification effect of the method of the present invention is compared.

Table 1 Comparison of classification results of RGB-D Scenes dataset

Classification Accuracy/% Linear SVM 89.6% Gaussian Kernel SVM 90.0% random forest 90.1% HOG 77.2% SIFT+SPM 84.2% The method of the invention 91.7%

The comparison of classification accuracy with other methods is shown in Table 1. Liefeng Bo integrated the three features in the article "Kerneldescriptors for visual recognition", and used linear SVM (LinearSVM), Gaussian kernel function SVM (Kernel SVM) and random forest (Random Forest) to train and classify them respectively. In this The accuracy rates of 89.6%, 90.0% and 90.1% are respectively obtained in the experiment. A. Janoch used the HOG algorithm in the article "A Category-Level 3D Object Dataset: Putting the Kinect to Work" to extract the features of the depth image and the color image respectively. After the feature fusion, the SVM classifier was used to achieve the final classification. In this This method achieves 77.2% accuracy in this experiment. In the article "Indoor scene segmentation using astructured light sensor" by N.Silberman, the SIFT algorithm is used to extract the features of the depth image and the color image respectively, and then the feature fusion is performed, and then SPM is used for feature encoding, and finally SVM is used for classification. Here In this experiment, this algorithm obtains 84.2% classification accuracy. However, the algorithm proposed by the present invention obtains an accuracy rate of 91.7%, which is 1.6% higher than the previous best result. It can be seen that the algorithm of the present invention has good classification performance.

Table 2 Comparison of classification results of different fusion features in RGB-D Scenes dataset

It can be seen from Table 2 that when combining depth information for image classification, the accuracy of the classification algorithm based on single fusion features is lower than that based on multi-fusion features, while the image classification algorithm based on multi-feature fusion can achieve better classification The accuracy rate is higher, but it is still slightly lower than the image classification algorithm based on multi-fusion feature decision fusion.

Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention .

Claims (8)

1. an image classification method based on RGB-D fusion feature and sparse coding, is characterized in that, comprises training stage and testing stage: The training phase includes the following steps: Step A1, for each sample data, extract the denseSIFT and PHOG features of its RGB image and Depth image; the number of sample data is n; Step A2. For each sample data, perform feature fusion on the features extracted from the two images in the form of two-two linear series to obtain four different fusion features; combine the same type of fusion features obtained from n sample data into one Set to get four fusion feature sets; Step A3, respectively clustering the fusion features in the four fusion feature sets to obtain four different visual dictionaries; Step A4. On each visual dictionary, use a locally constrained linear coding model to perform feature coding on the fused features to obtain four different image representation sets; Step A5, constructing classifiers according to four different fusion feature sets, image representation sets and corresponding class labels of sample data, to obtain four different classifiers; The testing phase includes the following steps: Step B1, extracting and fusing features of the image to be classified according to the method in steps A2 to A3 to obtain four fusion features of the image to be classified; Step B2, on the four visual dictionaries obtained in step A3, use the locally constrained linear coding model to perform feature encoding on the four fusion features obtained in step B1, and obtain four different image representations of the image to be classified; Step B3, using the four classifiers obtained in step A5 to classify the four image representations obtained in step B2, respectively, to obtain four class labels; Step B4. Based on the obtained four class labels, use the voting decision method to obtain the final class label of the image to be classified, that is, select the class label with the most votes among the four class labels as the final class label. 2. the image classification method based on RGB-D fusion feature and sparse coding according to claim 1, it is characterized in that, in described step A3, use K-means++ clustering method to carry out for the fusion feature in certain fusion feature set Clustering processing, the method of establishing a corresponding visual dictionary is as follows: 3.1) Record this kind of fusion feature set as H _I ={h ₁ ,h ₂ ,h ₃ ,…,h _n }, and set the number of clusters as m; 3.2) Randomly select a fusion feature in H _I as the first initial clustering center S ₁ ; set count value t=1; 3.3) For each fusion feature h _i in H _I , h _i ∈ H _I , calculate the distance d(h _i ) between it and S _t ; 3.4) Select the next initial clustering center S _t+1 : formula based Calculate the probability that point h _i ' is selected as the next initial cluster center, where h _i '∈H _I ; select the fusion feature with the highest probability as the next initial cluster center S _t+1 ; 3.5) Make t=t+1, repeat steps (3) and (4), until t=m, that is, m initial cluster centers are selected; 3.6) Use the selected initial cluster centers to run the K-means algorithm, and finally generate m cluster centers; 3.7) Define each cluster center as a visual word in the visual dictionary, and the number of clusters m is the size of the visual dictionary. 3. The image classification method based on RGB-D fusion feature and sparse coding according to claim 2, characterized in that, in the step A4, the fusion feature is characterized by using a locally constrained linear coding model, and the model expression is as follows : <mfenced open = "" close = ""> <mtable> <mtr> <mtd> <mrow> <munder> <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mi>C</mi> </munder> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mo>|</mo> <mo>|</mo> <msub> <mi>h</mi> <mi>i</mi> </msub> <mo>-</mo> <msub> <mi>Bc</mi> <mi>i</mi> </msub> <mo>|</mo> <msup> <mo>|</mo> <mn>2</mn> </msup> <mo>+</mo> <mi>&amp;lambda;</mi> <mo>|</mo> <mo>|</mo> <msub> <mi>d</mi> <mi>i</mi> </msub> <mo>&amp;CircleTimes;</mo> <msub> <mi>c</mi> <mi>i</mi> </msub> <mo>|</mo> <msup> <mo>|</mo> <mn>2</mn> </msup> </mrow> </mtd> </mtr> <mtr> <mtd> <mtable> <mtr> <mtd> <mrow> <mi>s</mi> <mo>.</mo> <mi>t</mi> <mo>.</mo> </mrow> </mtd> <mtd> <mrow> <msup> <mn>1</mn> <mi>T</mi> </msup> <msub> <mi>c</mi> <mi>i</mi> </msub> <mo>=</mo> <mn>1</mn> <mo>,</mo> <mo>&amp;ForAll;</mo> <mi>i</mi> </mrow> </mtd> </mtr> </mtable> </mtd> </mtr> </mtable> </mfenced> In the formula: h _i is the fusion feature in the fusion feature set H _I , that is, the feature vector to be encoded, h _i ∈ R ^d , d represents the dimension of the fusion feature; B=[b ₁ ,b ₂ ,b ₃ …b _m ] is a visual dictionary established by the K-means++ algorithm, b ₁ ~ b _m are m visual words in the visual dictionary, b _j ∈ R ^d ; C=[c ₁ ,c ₂ ,c ₃ …c _n ] is the code The obtained image representation set, where c _i ∈ R ^m is the encoding coefficient of the fusion feature h _i after encoding; λ is the penalty factor of LLC; Indicates that the elements are multiplied correspondingly; 1 in 1 ^T c _i represents a vector with all elements being 1, then 1 ^T c _i =1 is used to constrain the locally constrained linear coding model so that it has translation invariance; d _i is defined as: <mrow> <msub> <mi>d</mi> <mi>i</mi> </msub> <mo>=</mo> <mi>exp</mi> <mrow> <mo>(</mo> <mfrac> <mrow> <mi>d</mi> <mi>i</mi> <mi>s</mi> <mi>t</mi> <mrow> <mo>(</mo> <msub> <mi>h</mi> <mi>i</mi> </msub> <mo>,</mo> <mi>B</mi> <mo>)</mo> </mrow> </mrow> <mi>&amp;sigma;</mi> </mfrac> <mo>)</mo> </mrow> </mrow> where dist(h _i ,B)=[dist(h _i ,b ₁ ),dist(h _i ,b ₂ ),…dist(h _i ,b _m )] ^T , dist(h _i ,b _j ) means h The Euclidean distance between _i and _bj , σ is used to adjust the descent speed of the constraint weights for local locations. 4. The image classification method based on RGB-D fusion features and sparse coding according to claim 1, characterized in that, in the step A4, an approximate local constrained linear coding model is used to perform feature encoding on the fusion features, and the model expression The formula is as follows: <mfenced open = "" close = ""> <mtable> <mtr> <mtd> <mrow> <munder> <mrow> <mi>m</mi> <mi>i</mi> <mi>n</mi> </mrow> <mover> <mi>C</mi> <mo>~</mo> </mover> </munder> <munderover> <mo>&amp;Sigma;</mo> <mrow> <mi>i</mi> <mo>=</mo> <mn>1</mn> </mrow> <mi>n</mi> </munderover> <mo>|</mo> <mo>|</mo> <msub> <mi>h</mi> <mi>i</mi> </msub> <mo>-</mo> <msub> <mi>B</mi> <mi>i</mi> </msub> <mover> <msub> <mi>c</mi> <mi>i</mi> </msub> <mo>~</mo> </mover> <mo>|</mo> <msup> <mo>|</mo> <mn>2</mn> </msup> </mrow> </mtd> </mtr> <mtr> <mtd> <mtable> <mtr> <mtd> <mrow> <mi>s</mi> <mo>.</mo> <mi>t</mi> <mo>.</mo> </mrow> </mtd> <mtd> <mrow> <msup> <mn>1</mn> <mi>T</mi> </msup> <mover> <msub> <mi>c</mi> <mi>i</mi> </msub> <mo>~</mo> </mover> <mo>=</mo> <mn>1</mn> <mo>,</mo> <mo>&amp;ForAll;</mo> <mi>i</mi> </mrow> </mtd> </mtr> </mtable> </mtd> </mtr> </mtable> </mfenced> Among them, B _i is the local visual word matrix composed of _k visual words closest to the feature vector hi to be encoded in the visual dictionary B selected by k-neighbor search, is the image representation set obtained by approximate encoding, where is the encoding coefficient of the fusion feature _hi after the approximate encoding is completed. 5. the image classification method based on RGB-D fusion feature and sparse coding according to claim 4, characterized in that k=50. 6. The image classification method based on RGB-D fusion features and sparse coding according to any one of claims 1 to 5, wherein, in the step A1, the specific steps of PHOG feature extraction are as follows: 1.1) Statistical image edge information; use the Canny edge detection operator to extract the edge profile of the image, and use this profile to describe the shape of the image; 1.2) Segment the image in a pyramid level; divide the image into 3 layers, the first layer is the entire image; the second layer divides the image into 4 sub-regions, and the size of each region is the same; the third layer is the basis of the second layer Divide the 4 sub-regions above, divide each region into 4 sub-regions, and finally get 4×4 sub-regions; 1.3) Extract the HOG feature vector of each sub-region in each layer; 1.4) Finally, the HOG feature vectors of the sub-regions in each layer of the image are cascaded, and after the concatenated HOG data are obtained, the normalization operation of the data is performed, and finally the PHOG features of the entire image are obtained. 7. The image classification method based on RGB-D fusion features and sparse coding according to any one of claims 1 to 5, characterized in that, in the step A5, the classifier adopts a linear SVM classifier. 8. The image classification method based on RGB-D fusion features and sparse coding according to any one of claims 1 to 5, characterized in that, in the step B4, if there are multiple class labels with the largest number of votes, Then randomly select one of these class labels as the final class label.