CN101299241B - Method for detecting multi-mode video semantic conception based on tensor representation - Google Patents

Abstract

The invention discloses a multiple modal video semantic concept testing method based on tensor representation, including the following steps: 1. extracting image, audio and text modal bottom layer characteristic from the video lens in the training set and test set, wherein, each video tensor lens is expressed by the three bottom layer characteristics to form a third order tensor; 2. according to the manifold space eigen structure of the video tensor lens, implementing the dimension reducing and subspace embedding to the original high dimension tensor through conversion matrix searching; 3.using a supporting tensor machine to establish a categorizer model to the video tensor lens set after dimensionality reduction; 4. using the conversion matrix obtained by calculating the training set to project the test lens, and then executing semantic concept testing through a categorizer. The invention fully utilizes the multiple modal data in the video, represents the video lens as a third order tensor, and proposes a subspace embedded dimensionality reduction method based on the expression, thereby implementing the semantic concept testing to the video lens and executing better analysis and comprehension to the video semantic meaning.

Description

Method for detecting multi-mode video semantic conception based on tensor representation

Technical field

The present invention relates to a kind of method for detecting multi-mode video semantic conception based on tensor representation.This method is expressed as 3 rank tensors with video lens, and seeks effective dimension reduction method it is projected to the low-dimensional semantic space, thereby realizes the semantic concept of video tensor camera lens is detected by the training classifier model, belongs to video semanteme analysis and understanding field.

Background technology

Along with the development of various digital image apparatus with popularize, and the develop rapidly of film and television industry, computer technology, the communication technology, multimedia treatment technology, compression coding technology and internet etc., produced a large amount of video datas in fields such as news, film, historical document and monitoring.Video data has contained abundant semanteme such as task, scene, object and incident, and video is again a time series data simultaneously, has image, audio frequency and three kinds of medium data of text in the video, and presents the sequential correlation symbiotic characteristic.Simultaneously, the fusion of multiple modalities is also played important effect with " the semantic wide gap " of cooperating for reducing between low-level image feature and the high-level semantic.How effectively to utilize the multi-modal of video and sequential characteristic to excavate its semantic information, thereby support effective retrieval of video, the resource sharing advantage of performance video data is challenging studying a question.

For how expressing multi-modal medium in the video, traditional method is that image, audio frequency and text feature are represented with the splicing vector.But this high dimension vector tends to cause the problem of " dimension disaster ", and the relation of the sequential correlation symbiosis between multiple modalities in the video also can be left in the basket.In recent years, how much of polytenies-be that high order tensor has been widely applied to fields such as computer vision, information retrieval and signal Processing.Tensor is a kind of natural expansion and the extension to the vector sum matrix, and tensor geometry has defined a series of polyteny computings based on the vector space set.Simultaneously, finding the solution optimum solution with tensor as the supervising tensor learning framework employing alternating projection optimization step that has of input, is the combination of protruding optimization and polyteny geometric operation.Based on the supervising tensor learning framework is arranged, traditional support vector machine can be expanded to the tensor machine of supporting, realize the training and the application of sorter model.

Summary of the invention

The purpose of this invention is to provide a kind of method for detecting multi-mode video semantic conception based on tensor representation.Comprise the steps:

1) video lens in training set and the test set is all extracted the low-level image feature of image, audio frequency, three kinds of mode of text, each video tensor camera lens forms 3 rank tensors by these three kinds of low-level image features and expresses;

2) according to the stream shape space intrinsic structure of video tensor camera lens set, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding by seeking transition matrix;

3) adopt the video tensor camera lens set after supporting the tensor machine to dimensionality reduction to set up sorter model;

4) for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out the semantic concept detection by sorter model again.

The described low-level image feature that camera lens in training set and the test set is all extracted image, audio frequency, three kinds of mode of text: choose a key frame in each camera lens as representative image, extract color histogram, texture and Canny border then as characteristics of image; One section audio of camera lens correspondence is extracted as an audio example, and audio example is divided into the audio frame in short-term that contains superposition, extract each audio frame feature in short-term, comprise MFCC, barycenter, decay cutoff frequency, frequency spectrum flow and zero-crossing rate, form the frame proper vector, then the audio frequency characteristics of the statistical value of audio frame proper vector in short-term as camera lens; From video, extract the TF*IDF value as text feature through the transcribed text of identification.

The expression of described video tensor camera lens: based on the image that extracts in the video, audio frequency, text low-level image feature, with each video lens with one 3 rank tensor $S\∈R^{I_1\×I_2\×I_3}$ Represent.Wherein, I ₁, I ₂And I ₃It is respectively the dimension of characteristics of image, audio frequency characteristics and text feature.Each element s so _I1i2i3Value defined be: s _I1,1,1(1≤i ₁≤ I ₁) be the value of characteristics of image, s _{2, i2,2}(1≤i ₂≤ I ₂) be the value of audio frequency characteristics, s _{3,3, i3}(1≤i ₃≤ I ₃) be the value of text feature, the value of other element all initially is made as zero.

Described stream shape space intrinsic structure according to video tensor camera lens by seeking the transition matrix realization to the dimension reduction of original higher-dimension tensor and the method for subspace embedding is: given space R ^{I1 * I2 * I3}On lens data set X={X ₁, X ₂, LX _N, according to the stream shape space intrinsic structure and the spectrogram theory of tensor camera lens, be each the tensor camera lens X on the X _i| _I=1 ^NSeek three transition matrix: J ₁* I ₁The T of dimension ₁ ⁱ, J ₂* I ₂The T of dimension ₂ ⁱAnd J ₃* I ₃The T of dimension ₃ ⁱ, make it to shine upon this N data point to space R ^{J1 * J2 * J3}(J ₁＜I ₁, J ₂＜I ₂, J ₃＜I ₃) on Y={Y ₁, Y ₂, LY _N, satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}},$ Realize the dimension of original higher-dimension tensor is reduced and the subspace embedding with this; When asking for T ₁ ⁱ| _I=1 ^NThe time, by finding the solution generalized eigenvector problem (D _U-W _U) V ₁=λ D _UV ₁Calculate optimized intermediate conversion matrix V ₁, wherein, $D_U=\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{U}_1^{\mathrm{<figure-callout\; id="1"\; label="i\; U"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">i</figure-callout>}}{U}_{}^{}{}_1^{\mathrm{iT}},$ $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^{\mathrm{<figure-callout\; id="1"\; label="i\; U"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">i</figure-callout>}}{U}_{}^{}{}_1^{\mathrm{jT}},$ And W is that D is that the diagonal matrix of W is D according to the weight matrix of the constructed arest neighbors figure of training set X _Ii=∑ _jW _Ij, U ₁ ⁱBe to X _i| _I=1 ^NA mould to launch matrix mode-1 unfolding matrix be X ₍₁₎ ⁱCarry out SVD and decompose the left matrix that obtains, so finally can calculate transition matrix $T_1^i{|}_{i=1}^N=V_1^T{U}_1^i\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}};$ With asking for T with quadrat method ₂ ⁱWith T ₃ ⁱ

The method that sorter model is set up in video tensor camera lens set after described employing supports the tensor machine to dimensionality reduction is: the input of sorter model is the low-dimensional tensor through the subspace embeds and dimensionality reduction obtains $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}$ And corresponding classification logotype y _i∈+1 ,-1), output is the tensor lineoid parameter of sorter model $w_k{|}_{k=1}^3\&Element;R^{J_k}$ With b ∈ R; By the iterative optimization problem $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& \begin{array}{c}{y}_i\left[{w^T}_j(Y_i\underset{j=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_j{w}_j+b)\right]{\&GreaterEqual;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&GreaterEqual;0\end{array}\end{array}\end{array}\right]$ Obtain w _k| _K=1 ³And b, wherein parameter j is recycled to 3 from 1, and c is constant, and ξ is a relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2.$

Described for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out semantic concept by sorter model again and detect: the new data outside the training set $X_t\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; I"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{I_{}}$ By transition matrix $T_1^t=V_1^T{U}_1^t\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ $T_2^t=V_2^T{U}_2^t\&Element;R^{J_2\&times;I_2}$ And $T_3^t=V_3^T{U}_3^t\&Element;R^{J_3\&times;I_3}$ Be mapped as in the low n-dimensional subspace n $Y_t=X_t{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{tT}}{\&CircleTimes;}_2{T}_2^{\mathrm{tT}}{\&CircleTimes;}_3{T}_3^{\mathrm{tT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ Carry out classification by sorter model then and detect, promptly calculate z _t=sign (Y _t

₁w _{1 2}w ₂

₃w ₃)+b obtains the classification logotype z of test data _t∈+1 ,-1}.

Beneficial effect of the present invention:

1) the present invention has replaced the vectorial expression way of traditional video with tensor, can effectively reduce the problem that " dimension disaster " brings;

2) the present invention has considered the multiple modalities in the video: image, audio frequency and text, and the sequential correlation symbiotic characteristic of video data, play important effect based on the fusion of the multiple modalities of this characteristic with " the semantic wide gap " of cooperating for reducing between low-level image feature and the high-level semantic;

3) the present invention is according to the stream shape space intrinsic structure and the spectrogram theory that keep the set of tensor camera lens, the tensor camera lens subspace that is proposed embeds and dimension reduction method, not only solved the high-dimensional difficulty of bringing effectively, and owing to be linear method, the new data outer for training set can directly carry out projection mapping;

4) the present invention adopts and supports the tensor machine to come the training classifier model, has good classification detectability.

Description of drawings

Fig. 1 is based on the method for detecting multi-mode video semantic conception process flow diagram of tensor representation;

Fig. 2 is the testing result of the present invention to semantic concept " Explosion (blast) ", compares with ISOMAP and two kinds of methods of PCA respectively, is expressed as the ROC curve map;

Fig. 3 is the testing result of the present invention to semantic concept " Sports (sports) ", compares with ISOMAP and two kinds of methods of PCA respectively, is expressed as the ROC curve map.

Embodiment

Method for detecting multi-mode video semantic conception based on tensor representation.Comprise the steps:

1) video lens in training set and the test set is all extracted the low-level image feature of image, audio frequency, three kinds of mode of text, each video tensor camera lens forms 3 rank tensors by these three kinds of low-level image features and expresses;

2) according to the stream shape space intrinsic structure of video tensor camera lens set, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding by seeking transition matrix;

3) adopt the video tensor camera lens set after supporting the tensor machine to dimensionality reduction to set up sorter model;

4) for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out the semantic concept detection by sorter model again.

The described low-level image feature that camera lens in training set and the test set is all extracted image, audio frequency, three kinds of mode of text: low-level image feature is meant the feature of directly extracting from video source data, be different from the high-level characteristic of semantic concept representative.We extract low-level image feature respectively from each video lens, comprise image, audio frequency and text feature.

Characteristics of image: camera lens is a basic processing unit, chooses a key frame in each camera lens as representative image, and color histogram, texture and the Canny border of extracting key frame then are as characteristics of image;

Audio frequency characteristics a: section audio of camera lens correspondence is extracted as an audio example (audio clip), and audio example is divided into the audio frame in short-term that contains superposition, extract each audio frame feature in short-term, comprise MFCC, barycenter, decay cutoff frequency, frequency spectrum flow and zero-crossing rate, form the frame proper vector, then the statistical value of audio frame proper vector (average or variance) in short-term as the audio frequency characteristics of camera lens;

Text feature: we extract feature by (transcript) text of transcribing through identification from video.Because the dimension of text feature is much larger than other mode features, and comprised abundant semantic information in the text, (Latent Semantic Analysis LSA) does dimension-reduction treatment to text can to adopt implicit semantic analysis earlier.

The expression of described video tensor camera lens: based on the image that extracts in the video, audio frequency, text low-level image feature, with each video lens with one 3 rank tensor $S\&Element;R^{I_1\&times;I_2\&times;I_3}$ Represent.Wherein, I ₁, I ₂And I ₃It is respectively the dimension of characteristics of image, audio frequency characteristics and text feature.Each element s so _I1i2i3Value defined be: s _I1,1,1(1≤i ₁≤ I ₁) be the value of characteristics of image, s _{2, i2,2}(1≤i ₂≤ I ₂) be the value of audio frequency characteristics, s _{3,3, i3}(1≤i ₃≤ I ₃) be the value of text feature, the value of other element all initially is made as zero.

Described stream shape space intrinsic structure according to video tensor camera lens by seeking the transition matrix realization to the dimension reduction of original higher-dimension tensor and the method for subspace embedding is: given space R ^{I1 * I2 * I3}On lens data set X={X ₁, X ₂, LX _N, according to the stream shape space intrinsic structure and the spectrogram theory of tensor camera lens, be each the tensor camera lens X on the X _i| _I=1 ^NSeek three transition matrix: J ₁* I ₁The T of dimension ₁ ⁱ, J ₂* I ₂The T of dimension ₂ ⁱAnd J ₃* I ₃The T of dimension ₃ ⁱ, make it to shine upon this N data point to space R ^{J1 * J2 * J3}(J ₁＜I ₁, J ₂＜I ₂, J ₃＜I ₃) on set Y={Y ₁, Y ₂, L Y _N, and satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}},$ Low-dimensional data acquisition Y has just reflected how much topological structures of intrinsic in the stream shape space of set X so; Simultaneously, this mapping has also kept linear feature, that is to say for the outer data point X of training set _t, the transition matrix that can directly be obtained by training in advance calculates its mapping in low n-dimensional subspace n.

Order $X\&Element;i^{I_1\&times;I_2\&times;I_3}$ Represent one 3 rank tensor camera lens, given N tensor camera lens stream shape space that is distributed in ${M\&Element;\mathrm{<figure-callout\; id="1"\; label="R\; I"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{I_{}}$ On data acquisition X={X ₁, X ₂, Λ X _N, we can make up the local geometry that an arest neighbors figure G simulates M.The weight matrix W of definition G is as follows:

Wherein c is a constant.

For each tensor camera lens X _i, (1≤i≤N), (Higher-OrderSingular Value Decomposition, HOSVD), we can be respectively to X according to the high-order svd _iK| _K=1 ³Mould launches matrix (mode-kunfolding matrix) X ₍₁₎ ⁱ, X ₍₂₎ ⁱ, X ₍₃₎ ⁱCarry out singular value SVD and decompose, calculate left matrix U ₁ ⁱ, U ₂ ⁱ, U ₃ ⁱFor instance, U ₁ ⁱBe to X _iA mould launch matrix (mode-1 unfolding matrix) X ₍₁₎ ⁱCarry out SVD and decompose the left matrix that obtains.

Existing now $U_1^i\&Element;R^{I_1\&times;I_1},(1\&le;i\&le;N),$ We want to find I ₁* J ₁The matrix V of dimension ₁With U ₁ ⁱBe mapped to ${\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}\&Element;R^{I_1\&times;{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1},$ Just make $T_1^i=V_1^T{U}_1^i\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}.$ We will consider this problem from two angles.On the one hand, the intrinsic structure of shape be kept flowing, the optimum solution of following this objective function need be asked for:

$\underset{v_1}{\min }\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|V_1^T{U}_1^i-V_1^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}}$

That is to say, minimize $\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|{V_1}^T{U}_1^i-{V_1}^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}}$ Can guarantee to work as U ₁ ⁱAnd U ₁ ^jBe " close ", V so ₁ ^TU ₁ ⁱAnd V ₁ ^TU ₁ ^jAlso be " close ".

D is the diagonal matrix of W, i.e. D _Ii=∑ _jW _IjAnd for a matrix A, its " mark (trace) " ‖ A ‖ ²=tr (AA ^T), have so:

$\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{{\left|\right|{V_1^{T}U}_1^i-V_1^T{U}_1^i\left|\right|}^{2}W}_{\mathrm{ij}}$

$=\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}\mathrm{tr}\left((T_1^i-T_1^j)(T_1^i-T_1^j{)}^T\right)W_{\mathrm{ij}}$

$=\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}\mathrm{tr}{(T_1^i{T}_1^i+T_1^j{T}_1^j-T_1^i{T}_1^{j^T}-T_1^j{T}_1^{i^T})W}_{\mathrm{ij}}$

$=\mathrm{tr}(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{T}_1^i{T}_1^{i^T}-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{T}_1^i{T}_1^{j^T})$

$=\mathrm{tr}(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{V}_1^T{U}_1^i{U}_1^{i^T}{V}_1-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{V}_1^T{U}_1^i{U}_1^{j^T}{V}_1)$

$=\mathrm{tr}\left(V_1^T(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{U}_1^i{U}_1^{i^T}-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^i{U}_1^{j^T})V_1\right)$

Wherein $D_U=\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_1^i{U}_1^i}^T,$ $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^i{U_1^j}^T.$ From top derivation as can be seen, if want to find the solution $\underset{V_{}}{\min }$ $\underset{}{{}_1}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|V_1^T{U}_1^i-V_1^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}},$ Need minimize tr (V ₁ ^T(D _U-W _U) V ₁).

On the other hand, except keeping flowing the graph structure of shape, also need to maximize the overall variance on the stream shape space.Usually, the variance of a stochastic variable x is:

var(x)＝∫ _M(x-μ) ²dP(x)，μ＝∫ _MxdP(x)

Wherein M is the stream shape of data, and μ is an expectation value, and dP is a probability density function.According to spectrogram theory (spectralgraph theory), dP can be by the diagonal matrix D (D of sample point _Ii=∑ _jW _Ij) estimation of discretize obtains.We have following derivation so:

$\mathrm{var}\left(T_1\right)$

$=\underset{i}{\mathrm{\&Sigma;}}{\left|\right|T_1^i\left|\right|}^2{D}_{\mathrm{ii}}$

$=\underset{i}{\mathrm{\&Sigma;}}\mathrm{tr}\left(T_1^i{T}_1^{i^T}\right)D_{\mathrm{ii}}$

$=\underset{i}{\mathrm{\&Sigma;}}\mathrm{tr}\left({{V_1^T{U}_1^i{U}_1^i}^{T}V}_1\right)D_{\mathrm{ii}}$

$=\mathrm{tr}\left(V_1^T\left(\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_1^i{U}_1^i}^T\right)V_1\right)$

By the constraint condition of above two aspects, we have obtained following optimization problem:

$\underset{v_{}}{\mathrm{<figure-callout\; id="1"\; label="min\; v"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">min</figure-callout>}}\frac{\mathrm{tr}\left(V_1^T(D_U-W_U)V_1\right)}{\mathrm{tr}\left(V_1^T{D}_U{V}_1\right)}$

Obviously, V ₁Optimum solution be (D _U-W _U, D _U) generalized eigenvector.Therefore we can calculate following generalized eigenvector problem and obtain optimized V ₁:

(D _U-W _U)V ₁＝λD _UV ₁

When calculating v ₁After, by $U_1^i\&Element;i^{I_1\&times;I_1},(1\&le;i\&le;N),$ Can ask for $T_1^i=V_1^T{U}_1^i\&Element;i^{{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1\&times;I_1}.$ In like manner, for the intermediate conversion matrix V of audio frequency and these two kinds of mode of text ₂And V ₃Also can use the same method and calculate, so by $U_2^i\&Element;i^{I_2\&times;I_2},(1\&le;i\&le;N)$ And V ₂Just can ask for $T_2^i=V_2^T{U}_2^i\&Element;i^{J_2\&times;I_2},$ And by $U_3^i\&Element;i^{I_3\&times;I_3},(1\&le;i\&le;N)$ And V ₃Can ask for $T_3^i=V_3^T{U}_3^i\&Element;i^{J_3\&times;I_3}.$ The video tensor camera lens of lower dimensional space is gathered the data among the Y like this $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}.$

Be the algorithm that the subspace embeds and dimension reduces of tensor camera lens below.

Input: original training tensor camera lens set $X=\{{\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">X</figure-callout>}}_1,X_2,L{X}_N\}{R}^{I_1\&times;I_2\&times;I_3};$

Output: the low-dimensional tensor camera lens set after the mapping $Y=\{{\mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">Y</figure-callout>}}_1,Y_2,L{Y}_N\}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ The intermediate conversion matrix $V_{}$ ${}_1\&Element;R^{I_1\&times;{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1},$ $V_2\&Element;R^{I_2\&times;J_2}$ With $V_3\&Element;R^{I_3\&times;J_3},$ And transition matrix $T_1^i=V_1^T{U}_1^i\&Element;i^{{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1\&times;I_1},$ $T_2^i=V_2^T{U}_2^i\&Element;i^{J_2\&times;I_2}$ With $T_3^i=V_3^T{U}_3^i\&Element;i^{J_3\&times;I_3},$ And satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}};$ Arthmetic statement:

Step 1: make up an arest neighbors figure G;

Step 2: calculate weight matrix W;

Step 3:For k=1to3

Step 4:For i=1toN

Step 5: calculate X _iThe k mould launch matrix X _(k) ⁱThe left matrix U that decomposes of SVD _(k) ⁱ

Step 6:End;

Step 7: $D_U=\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_{\left(k\right)}^i{U}_{\left(k\right)}^i}^T;$

Step 8: $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W_{\mathrm{ij}}{U}_{\left(k\right)}^i{U}_{\left(k\right)}^j}^T;$

Step 9: find the solution following generalized eigenvector problem to obtain optimized V _k:

(D _U-W _U)V _k＝λD _UV _k；

Step 10:For i=1toN

Step 11: $T_{\left(k\right)}^i=V_{\left(k\right)}^T{U}_{\left(k\right)}^i\&Element;R^{J_k\&times;I_k};$

Step 12:End;

Step 13:End;

Step 14:For i=1toN

Step 15: $Y_i=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}};$

Step 16:End.

The method that sorter model is set up in video tensor camera lens set after described employing supports the tensor machine to dimensionality reduction is: in this step, we adopt and support the tensor machine to train the sorter of tensor camera lens.The input of training pattern is exactly a previous step through the low-dimensional tensor Y that the subspace embeds and dimensionality reduction obtains _i, rather than original X _iSuch processing can not only improve degree of accuracy, and can improve the efficient of training and classification.

Support that the algorithm of tensor machine training classifier is as follows.

Input: the low-dimensional tensor camera lens set after the mapping $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ And

Corresponding classification logotype y _i∈+1 ,-1};

Output: the tensor lineoid parameter of sorter model $w_k{|}_{k=1}^3\&Element;R^{J_k}$ With b ∈ R;

Arthmetic statement:

Step 1: w is set _k| _K=1 ³Be R ^JkIn the random units vector;

Step 2: repeating step 3-5 is until convergence;

Step 3:For j=1to3

Step 4: by finding the solution optimization problem $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& \begin{array}{c}{y}_i\left[{w^T}_j(Y_i\underset{k=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_k{w}_k+b)\right]{\&GreaterEqual;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&GreaterEqual;0\end{array}\end{array}\end{array}\right]$ Obtain $w_i\&Element;R^{I_j}$ And b, wherein c is a constant, ξ is a relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2;$

Step 5:End;

Step 6: check whether restrain: if ${\mathrm{\&Sigma;}}_{k=1}^3\left[\right|w_{k,t}^T{w}_{k,t-1}|\left({\left|\right|w_{k,t}\left|\right|}_{\mathrm{Fro}}^{-2}\right)-1\&rsqb;\&le;\mathrm{\&epsiv;}$ Calculate so

w _k| _K=1 ³Restrain.Here w _{K, t}Be current projection vector, w _{K, t-1}It is previous projection vector;

Step 7:End.

Described for testing lens, after adding up to the transition matrix obtain to carry out projection by training set, carrying out semantic concept by sorter model again detects: in this step, we will come according to the sorter model that the front training obtains the outer new data of training set is detected.Because our dimension reduction method is linear,, carries out classification by sorter then and detect so can map directly to low n-dimensional subspace n for new data.

Make X _tAs an outer detection example of training set, following algorithm provides testing process.

Input: camera lens to be detected $X_t\&Element;R^{I_1\&times;I_2\&times;I_3},$ The intermediate conversion matrix V ₁, V ₂, V ₃, classifier parameters $w_k{|}_{k=1}^3\&Element;R^{I_k}$ With b ∈ R;

Output: X _tClassification logotype z _t∈+1 ,-1};

Arthmetic statement:

Step 1:For k=1to3;

Step 2: calculate X _tThe k mould launch matrix X _(k) ^tThe left matrix U that decomposes of SVD _(k) ^t

Step 3: calculate $T_{\left(k\right)}^t=V_{\left(k\right)}^T{U}_{\left(k\right)}^t;$

Step 4:End;

Step 5: calculate $Y_t=X_t{\&CircleTimes;}_1{T}_1^{t^T}{\&CircleTimes;}_2{T}_2^{t^T}{\&CircleTimes;}_3{T}_3^{t^T};$

Step 6: calculate z _t=sign (Y _{t 1}w ₁

₂w ₂

₃w ₃)+b;

Step 7:End.

Claims (5)

1. the method for detecting multi-mode video semantic conception based on tensor representation is characterized in that comprising the steps:

1) video lens in training set and the test set is all extracted the low-level image feature of image, audio frequency, three kinds of mode of text, each video tensor camera lens forms 3 rank tensors by these three kinds of low-level image features and expresses;

2) according to the stream shape space intrinsic structure of video tensor camera lens set, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding by seeking transition matrix;

3) adopt the video tensor camera lens set after supporting the tensor machine to dimensionality reduction to set up sorter model;

4) for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out the semantic concept detection by sorter model again;

The expression of described video tensor camera lens: based on the image that extracts in the video, audio frequency, text low-level image feature, with each video lens with one 3 rank tensor

Represent, wherein, I ₁, I ₂And I ₃Be respectively the dimension of characteristics of image, audio frequency characteristics and text feature, each element so

Value defined be:

Be the value of characteristics of image, wherein 1≤i ₁≤ I ₁

Be the value of audio frequency characteristics, wherein 1≤i ₂≤ I ₂

Be the value of text feature, wherein 1≤i ₃≤ I ₃The value of other element all initially is made as zero.

2. a kind of method for detecting multi-mode video semantic conception according to claim 1 based on tensor representation, it is characterized in that the described low-level image feature that video lens in training set and the test set is all extracted image, audio frequency, three kinds of mode of text: choose a key frame in each video lens as representative image, extract color histogram, texture and Canny border then as characteristics of image; One section audio of video lens correspondence is extracted as an audio example, and audio example is divided into superposition audio frame in short-term, extract each audio frame feature in short-term, comprise MFCC, barycenter, decay cutoff frequency, frequency spectrum flow and zero-crossing rate, form the frame proper vector, then the audio frequency characteristics of the statistical value of audio frame proper vector in short-term as video lens; From video, extract the TF*IDF value as text feature through the transcribed text of identification.

3. a kind of method for detecting multi-mode video semantic conception according to claim 1 based on tensor representation, it is characterized in that described stream shape space intrinsic structure, by seeking the transition matrix realization be: given space the dimension reduction of original higher-dimension tensor and the method for subspace embedding according to video tensor camera lens

On lens data set X={X ₁, X ₂... X _N, according to the stream shape space intrinsic structure and the spectrogram theory of tensor camera lens, be each the tensor camera lens X on the X _i| _I=1 ^NSeek three transition matrix: J ₁* I ₁The T of dimension ₁ ⁱ, J ₂* I ₂The T of dimension ₂ ⁱAnd J ₃* I ₃The T of dimension ₃ ⁱ, make it to shine upon this N data point to the space

On Y={Y ₁, Y ₂... Y _N, satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{T}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}},$ Wherein, J ₁＜I ₁, J ₂＜I ₂, J ₃＜I ₃, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding with this; When asking for T ₁ ⁱ| _I=1 ^NThe time, by finding the solution generalized eigenvector problem (D _U-W _U) V ₁=λ D _UV ₁Calculate optimized intermediate conversion matrix V ₁, wherein, $D_U=\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{U}_1^i{U}_1^{\mathrm{iT}},$ $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^i{U}_1^{\mathrm{jT}}$ , and W is the weight matrix according to the constructed arest neighbors figure of training set X, D be W diagonal matrix promptly $D_{\mathrm{ii}}={\mathrm{\&Sigma;}}_j{W}_{\mathrm{ij}},$ U ₁ ⁱBe to X _i| _I=1 ^NA mould to launch matrix mode-1 unfolding matrix be X ₍₁₎ ⁱCarry out SVD and decompose the left matrix that obtains, so finally can calculate transition matrix

With asking for T with quadrat method ₂ ⁱWith T ₃ ⁱ

4. a kind of method for detecting multi-mode video semantic conception based on tensor representation according to claim 1, it is characterized in that video tensor camera lens set after described employing supports the tensor machine to dimensionality reduction sets up the method for sorter model and be: the input of sorter model is the low-dimensional tensor through the subspace embeds and dimensionality reduction obtains

And corresponding classification logotype y _i∈+1, and-1}, output is the tensor lineoid parameter of sorter model

By the iterative optimization problem $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ s.t.\begin{array}{c}{y}_i\left[w_j^T(Y_i\underset{j=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_j{w}_j+b)\right]{\&GreaterEqual;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&GreaterEqual;0\end{array}\end{array}\right]$ Obtain w _k| _K=1 ³And b, wherein parameter j is recycled to 3 from 1, and c is constant, and ξ is a relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2.$

5. a kind of method for detecting multi-mode video semantic conception according to claim 1 based on tensor representation, it is characterized in that described for testing lens, after adding up to the transition matrix obtain to carry out projection by training set, carry out semantic concept by sorter model again and detect: the new data outside the training set

By transition matrix

And

Be mapped as in the low n-dimensional subspace n

Carry out classification by sorter model then and detect, promptly calculate $Z_t=\mathrm{sign}\left(Y_t{\&CircleTimes;}_1{w}_1{\&CircleTimes;}_2{w}_2{\&CircleTimes;}_3{w}_3\right)+b,$ Obtain the classification logotype z of test data _t∈+1 ,-1}.

Images