CN101299241B - Multimodal Video Semantic Concept Detection Method Based on Tensor Representation - Google Patents

Abstract

The invention discloses a multimodal video semantic concept detection method based on tensor representation. Including the following steps: 1) Extract the bottom-level features of image, audio and text from the video shots in the training set and test set, and each video tensor shot is represented by a third-order tensor formed by these three bottom-level features ; 2) According to the eigenstructure of the manifold space of the video tensor lens set, the dimension reduction and subspace embedding of the original high-dimensional tensor are realized by finding the transformation matrix; 4) For the test shots, the transformation matrix calculated from the training set is projected, and then the semantic concept detection is performed through the classifier model. The present invention makes full use of the multimodal data in the video, expresses the video shot as a third-order tensor, and proposes a dimensionality reduction method of subspace embedding based on this expression, realizes the semantic concept detection of the video shot, and improves the video semantic better analysis and understanding.

Description

Method for detecting multi-mode video semantic conception based on tensor representation

Technical field

The present invention relates to a kind of method for detecting multi-mode video semantic conception based on tensor representation.This method is expressed as 3 rank tensors with video lens, and seeks effective dimension reduction method it is projected to the low-dimensional semantic space, thereby realizes the semantic concept of video tensor camera lens is detected by the training classifier model, belongs to video semanteme analysis and understanding field.

Background technology

Along with the development of various digital image apparatus with popularize, and the develop rapidly of film and television industry, computer technology, the communication technology, multimedia treatment technology, compression coding technology and internet etc., produced a large amount of video datas in fields such as news, film, historical document and monitoring.Video data has contained abundant semanteme such as task, scene, object and incident, and video is again a time series data simultaneously, has image, audio frequency and three kinds of medium data of text in the video, and presents the sequential correlation symbiotic characteristic.Simultaneously, the fusion of multiple modalities is also played important effect with " the semantic wide gap " of cooperating for reducing between low-level image feature and the high-level semantic.How effectively to utilize the multi-modal of video and sequential characteristic to excavate its semantic information, thereby support effective retrieval of video, the resource sharing advantage of performance video data is challenging studying a question.

For how expressing multi-modal medium in the video, traditional method is that image, audio frequency and text feature are represented with the splicing vector.But this high dimension vector tends to cause the problem of " dimension disaster ", and the relation of the sequential correlation symbiosis between multiple modalities in the video also can be left in the basket.In recent years, how much of polytenies-be that high order tensor has been widely applied to fields such as computer vision, information retrieval and signal Processing.Tensor is a kind of natural expansion and the extension to the vector sum matrix, and tensor geometry has defined a series of polyteny computings based on the vector space set.Simultaneously, finding the solution optimum solution with tensor as the supervising tensor learning framework employing alternating projection optimization step that has of input, is the combination of protruding optimization and polyteny geometric operation.Based on the supervising tensor learning framework is arranged, traditional support vector machine can be expanded to the tensor machine of supporting, realize the training and the application of sorter model.

Summary of the invention

The purpose of this invention is to provide a kind of method for detecting multi-mode video semantic conception based on tensor representation.Comprise the steps:

1) video lens in training set and the test set is all extracted the low-level image feature of image, audio frequency, three kinds of mode of text, each video tensor camera lens forms 3 rank tensors by these three kinds of low-level image features and expresses;

2) according to the stream shape space intrinsic structure of video tensor camera lens set, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding by seeking transition matrix;

3) adopt the video tensor camera lens set after supporting the tensor machine to dimensionality reduction to set up sorter model;

4) for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out the semantic concept detection by sorter model again.

The described low-level image feature that camera lens in training set and the test set is all extracted image, audio frequency, three kinds of mode of text: choose a key frame in each camera lens as representative image, extract color histogram, texture and Canny border then as characteristics of image; One section audio of camera lens correspondence is extracted as an audio example, and audio example is divided into the audio frame in short-term that contains superposition, extract each audio frame feature in short-term, comprise MFCC, barycenter, decay cutoff frequency, frequency spectrum flow and zero-crossing rate, form the frame proper vector, then the audio frequency characteristics of the statistical value of audio frame proper vector in short-term as camera lens; From video, extract the TF*IDF value as text feature through the transcribed text of identification.

The expression of described video tensor camera lens: based on the image that extracts in the video, audio frequency, text low-level image feature, with each video lens with one 3 rank tensor $S\∈R^{I_1\×I_2\×I_3}$ Represent.Wherein, I ₁, I ₂And I ₃It is respectively the dimension of characteristics of image, audio frequency characteristics and text feature.Each element s so _I1i2i3Value defined be: s _I1,1,1(1≤i ₁≤ I ₁) be the value of characteristics of image, s _{2, i2,2}(1≤i ₂≤ I ₂) be the value of audio frequency characteristics, s _{3,3, i3}(1≤i ₃≤ I ₃) be the value of text feature, the value of other element all initially is made as zero.

Described stream shape space intrinsic structure according to video tensor camera lens by seeking the transition matrix realization to the dimension reduction of original higher-dimension tensor and the method for subspace embedding is: given space R ^{I1 * I2 * I3}On lens data set X={X ₁, X ₂, LX _N, according to the stream shape space intrinsic structure and the spectrogram theory of tensor camera lens, be each the tensor camera lens X on the X _i| _I=1 ^NSeek three transition matrix: J ₁* I ₁The T of dimension ₁ ⁱ, J ₂* I ₂The T of dimension ₂ ⁱAnd J ₃* I ₃The T of dimension ₃ ⁱ, make it to shine upon this N data point to space R ^{J1 * J2 * J3}(J ₁＜I ₁, J ₂＜I ₂, J ₃＜I ₃) on Y={Y ₁, Y ₂, LY _N, satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}},$ Realize the dimension of original higher-dimension tensor is reduced and the subspace embedding with this; When asking for T ₁ ⁱ| _I=1 ^NThe time, by finding the solution generalized eigenvector problem (D _U-W _U) V ₁=λ D _UV ₁Calculate optimized intermediate conversion matrix V ₁, wherein, $D_U=\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{U}_1^{\mathrm{<figure-callout\; id="1"\; label="i\; U"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">i</figure-callout>}}{U}_{}^{}{}_1^{\mathrm{iT}},$ $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^{\mathrm{<figure-callout\; id="1"\; label="i\; U"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">i</figure-callout>}}{U}_{}^{}{}_1^{\mathrm{jT}},$ And W is that D is that the diagonal matrix of W is D according to the weight matrix of the constructed arest neighbors figure of training set X _Ii=∑ _jW _Ij, U ₁ ⁱBe to X _i| _I=1 ^NA mould to launch matrix mode-1 unfolding matrix be X ₍₁₎ ⁱCarry out SVD and decompose the left matrix that obtains, so finally can calculate transition matrix $T_1^i{|}_{i=1}^N=V_1^T{U}_1^i\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}};$ With asking for T with quadrat method ₂ ⁱWith T ₃ ⁱ

The method that sorter model is set up in video tensor camera lens set after described employing supports the tensor machine to dimensionality reduction is: the input of sorter model is the low-dimensional tensor through the subspace embeds and dimensionality reduction obtains $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}$ And corresponding classification logotype y _i∈+1 ,-1), output is the tensor lineoid parameter of sorter model $w_k{|}_{k=1}^3\&Element;R^{J_k}$ With b ∈ R; By the iterative optimization problem $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& \begin{array}{c}{y}_i\left[{w^T}_j(Y_i\underset{j=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_j{w}_j+b)\right]{\&GreaterEqual;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&GreaterEqual;0\end{array}\end{array}\end{array}\right]$ Obtain w _k| _K=1 ³And b, wherein parameter j is recycled to 3 from 1, and c is constant, and ξ is a relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2.$

Described for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out semantic concept by sorter model again and detect: the new data outside the training set $X_t\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; I"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{I_{}}$ By transition matrix $T_1^t=V_1^T{U}_1^t\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ $T_2^t=V_2^T{U}_2^t\&Element;R^{J_2\&times;I_2}$ And $T_3^t=V_3^T{U}_3^t\&Element;R^{J_3\&times;I_3}$ Be mapped as in the low n-dimensional subspace n $Y_t=X_t{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{tT}}{\&CircleTimes;}_2{T}_2^{\mathrm{tT}}{\&CircleTimes;}_3{T}_3^{\mathrm{tT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ Carry out classification by sorter model then and detect, promptly calculate z _t=sign (Y _t

₁w _{1 2}w ₂

₃w ₃)+b obtains the classification logotype z of test data _t∈+1 ,-1}.

Beneficial effect of the present invention:

1) the present invention has replaced the vectorial expression way of traditional video with tensor, can effectively reduce the problem that " dimension disaster " brings;

2) the present invention has considered the multiple modalities in the video: image, audio frequency and text, and the sequential correlation symbiotic characteristic of video data, play important effect based on the fusion of the multiple modalities of this characteristic with " the semantic wide gap " of cooperating for reducing between low-level image feature and the high-level semantic;

3) the present invention is according to the stream shape space intrinsic structure and the spectrogram theory that keep the set of tensor camera lens, the tensor camera lens subspace that is proposed embeds and dimension reduction method, not only solved the high-dimensional difficulty of bringing effectively, and owing to be linear method, the new data outer for training set can directly carry out projection mapping;

4) the present invention adopts and supports the tensor machine to come the training classifier model, has good classification detectability.

Description of drawings

Fig. 1 is based on the method for detecting multi-mode video semantic conception process flow diagram of tensor representation;

Fig. 2 is the testing result of the present invention to semantic concept " Explosion (blast) ", compares with ISOMAP and two kinds of methods of PCA respectively, is expressed as the ROC curve map;

Fig. 3 is the testing result of the present invention to semantic concept " Sports (sports) ", compares with ISOMAP and two kinds of methods of PCA respectively, is expressed as the ROC curve map.

Embodiment

Method for detecting multi-mode video semantic conception based on tensor representation.Comprise the steps:

1) video lens in training set and the test set is all extracted the low-level image feature of image, audio frequency, three kinds of mode of text, each video tensor camera lens forms 3 rank tensors by these three kinds of low-level image features and expresses;

2) according to the stream shape space intrinsic structure of video tensor camera lens set, realize the dimension of original higher-dimension tensor is reduced and the subspace embedding by seeking transition matrix;

3) adopt the video tensor camera lens set after supporting the tensor machine to dimensionality reduction to set up sorter model;

4) for testing lens, add up to the transition matrix obtain to carry out projection by training set after, carry out the semantic concept detection by sorter model again.

The described low-level image feature that camera lens in training set and the test set is all extracted image, audio frequency, three kinds of mode of text: low-level image feature is meant the feature of directly extracting from video source data, be different from the high-level characteristic of semantic concept representative.We extract low-level image feature respectively from each video lens, comprise image, audio frequency and text feature.

Characteristics of image: camera lens is a basic processing unit, chooses a key frame in each camera lens as representative image, and color histogram, texture and the Canny border of extracting key frame then are as characteristics of image;

Audio frequency characteristics a: section audio of camera lens correspondence is extracted as an audio example (audio clip), and audio example is divided into the audio frame in short-term that contains superposition, extract each audio frame feature in short-term, comprise MFCC, barycenter, decay cutoff frequency, frequency spectrum flow and zero-crossing rate, form the frame proper vector, then the statistical value of audio frame proper vector (average or variance) in short-term as the audio frequency characteristics of camera lens;

Text feature: we extract feature by (transcript) text of transcribing through identification from video.Because the dimension of text feature is much larger than other mode features, and comprised abundant semantic information in the text, (Latent Semantic Analysis LSA) does dimension-reduction treatment to text can to adopt implicit semantic analysis earlier.

The expression of described video tensor camera lens: based on the image that extracts in the video, audio frequency, text low-level image feature, with each video lens with one 3 rank tensor $S\&Element;R^{I_1\&times;I_2\&times;I_3}$ Represent.Wherein, I ₁, I ₂And I ₃It is respectively the dimension of characteristics of image, audio frequency characteristics and text feature.Each element s so _I1i2i3Value defined be: s _I1,1,1(1≤i ₁≤ I ₁) be the value of characteristics of image, s _{2, i2,2}(1≤i ₂≤ I ₂) be the value of audio frequency characteristics, s _{3,3, i3}(1≤i ₃≤ I ₃) be the value of text feature, the value of other element all initially is made as zero.

Described stream shape space intrinsic structure according to video tensor camera lens by seeking the transition matrix realization to the dimension reduction of original higher-dimension tensor and the method for subspace embedding is: given space R ^{I1 * I2 * I3}On lens data set X={X ₁, X ₂, LX _N, according to the stream shape space intrinsic structure and the spectrogram theory of tensor camera lens, be each the tensor camera lens X on the X _i| _I=1 ^NSeek three transition matrix: J ₁* I ₁The T of dimension ₁ ⁱ, J ₂* I ₂The T of dimension ₂ ⁱAnd J ₃* I ₃The T of dimension ₃ ⁱ, make it to shine upon this N data point to space R ^{J1 * J2 * J3}(J ₁＜I ₁, J ₂＜I ₂, J ₃＜I ₃) on set Y={Y ₁, Y ₂, L Y _N, and satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}},$ Low-dimensional data acquisition Y has just reflected how much topological structures of intrinsic in the stream shape space of set X so; Simultaneously, this mapping has also kept linear feature, that is to say for the outer data point X of training set _t, the transition matrix that can directly be obtained by training in advance calculates its mapping in low n-dimensional subspace n.

Order $X\&Element;i^{I_1\&times;I_2\&times;I_3}$ Represent one 3 rank tensor camera lens, given N tensor camera lens stream shape space that is distributed in ${M\&Element;\mathrm{<figure-callout\; id="1"\; label="R\; I"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{I_{}}$ On data acquisition X={X ₁, X ₂, Λ X _N, we can make up the local geometry that an arest neighbors figure G simulates M.The weight matrix W of definition G is as follows:

Wherein c is a constant.

For each tensor camera lens X _i, (1≤i≤N), (Higher-OrderSingular Value Decomposition, HOSVD), we can be respectively to X according to the high-order svd _iK| _K=1 ³Mould launches matrix (mode-kunfolding matrix) X ₍₁₎ ⁱ, X ₍₂₎ ⁱ, X ₍₃₎ ⁱCarry out singular value SVD and decompose, calculate left matrix U ₁ ⁱ, U ₂ ⁱ, U ₃ ⁱFor instance, U ₁ ⁱBe to X _iA mould launch matrix (mode-1 unfolding matrix) X ₍₁₎ ⁱCarry out SVD and decompose the left matrix that obtains.

Existing now $U_1^i\&Element;R^{I_1\&times;I_1},(1\&le;i\&le;N),$ We want to find I ₁* J ₁The matrix V of dimension ₁With U ₁ ⁱBe mapped to ${\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}\&Element;R^{I_1\&times;{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1},$ Just make $T_1^i=V_1^T{U}_1^i\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}.$ We will consider this problem from two angles.On the one hand, the intrinsic structure of shape be kept flowing, the optimum solution of following this objective function need be asked for:

$\underset{v_1}{\min }\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|V_1^T{U}_1^i-V_1^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}}$

That is to say, minimize $\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|{V_1}^T{U}_1^i-{V_1}^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}}$ Can guarantee to work as U ₁ ⁱAnd U ₁ ^jBe " close ", V so ₁ ^TU ₁ ⁱAnd V ₁ ^TU ₁ ^jAlso be " close ".

D is the diagonal matrix of W, i.e. D _Ii=∑ _jW _IjAnd for a matrix A, its " mark (trace) " ‖ A ‖ ²=tr (AA ^T), have so:

$\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{{\left|\right|{V_1^{T}U}_1^i-V_1^T{U}_1^i\left|\right|}^{2}W}_{\mathrm{ij}}$

$=\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}\mathrm{tr}\left((T_1^i-T_1^j)(T_1^i-T_1^j{)}^T\right)W_{\mathrm{ij}}$

$=\frac{1}{2}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}\mathrm{tr}{(T_1^i{T}_1^i+T_1^j{T}_1^j-T_1^i{T}_1^{j^T}-T_1^j{T}_1^{i^T})W}_{\mathrm{ij}}$

$=\mathrm{tr}(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{T}_1^i{T}_1^{i^T}-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{T}_1^i{T}_1^{j^T})$

$=\mathrm{tr}(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{V}_1^T{U}_1^i{U}_1^{i^T}{V}_1-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{V}_1^T{U}_1^i{U}_1^{j^T}{V}_1)$

$=\mathrm{tr}\left(V_1^T(\underset{i}{\mathrm{\&Sigma;}}{D}_{\mathrm{ii}}{U}_1^i{U}_1^{i^T}-\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^i{U}_1^{j^T})V_1\right)$

Wherein $D_U=\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_1^i{U}_1^i}^T,$ $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{U}_1^i{U_1^j}^T.$ From top derivation as can be seen, if want to find the solution $\underset{V_{}}{\min }$ $\underset{}{{}_1}\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{\left|\right|V_1^T{U}_1^i-V_1^T{U}_1^j\left|\right|}^2{W}_{\mathrm{ij}},$ Need minimize tr (V ₁ ^T(D _U-W _U) V ₁).

On the other hand, except keeping flowing the graph structure of shape, also need to maximize the overall variance on the stream shape space.Usually, the variance of a stochastic variable x is:

var(x)＝∫ _M(x-μ) ²dP(x)，μ＝∫ _MxdP(x)

Wherein M is the stream shape of data, and μ is an expectation value, and dP is a probability density function.According to spectrogram theory (spectralgraph theory), dP can be by the diagonal matrix D (D of sample point _Ii=∑ _jW _Ij) estimation of discretize obtains.We have following derivation so:

$\mathrm{var}\left(T_1\right)$

$=\underset{i}{\mathrm{\&Sigma;}}{\left|\right|T_1^i\left|\right|}^2{D}_{\mathrm{ii}}$

$=\underset{i}{\mathrm{\&Sigma;}}\mathrm{tr}\left(T_1^i{T}_1^{i^T}\right)D_{\mathrm{ii}}$

$=\underset{i}{\mathrm{\&Sigma;}}\mathrm{tr}\left({{V_1^T{U}_1^i{U}_1^i}^{T}V}_1\right)D_{\mathrm{ii}}$

$=\mathrm{tr}\left(V_1^T\left(\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_1^i{U}_1^i}^T\right)V_1\right)$

By the constraint condition of above two aspects, we have obtained following optimization problem:

$\underset{v_1}{\min }\frac{\mathrm{tr}\left(V_1^T(D_U-W_U)V_1\right)}{\mathrm{tr}\left(V_1^T{D}_U{V}_1\right)}$

Obviously, V ₁Optimum solution be (D _U-W _U, D _U) generalized eigenvector.Therefore we can calculate following generalized eigenvector problem and obtain optimized V ₁:

(D _U-W _U)V ₁＝λD _UV ₁

When calculating v ₁After, by $U_1^i\&Element;i^{I_1\&times;I_1},(1\&le;i\&le;N),$ Can ask for $T_1^i=V_1^T{U}_1^i\&Element;i^{{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1\&times;I_1}.$ In like manner, for the intermediate conversion matrix V of audio frequency and these two kinds of mode of text ₂And V ₃Also can use the same method and calculate, so by $U_2^i\&Element;i^{I_2\&times;I_2},(1\&le;i\&le;N)$ And V ₂Just can ask for $T_2^i=V_2^T{U}_2^i\&Element;i^{J_2\&times;I_2},$ And by $U_3^i\&Element;i^{I_3\&times;I_3},(1\&le;i\&le;N)$ And V ₃Can ask for $T_3^i=V_3^T{U}_3^i\&Element;i^{J_3\&times;I_3}.$ The video tensor camera lens of lower dimensional space is gathered the data among the Y like this $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}}.$

Be the algorithm that the subspace embeds and dimension reduces of tensor camera lens below.

Input: original training tensor camera lens set $X=\{{\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">X</figure-callout>}}_1,X_2,L{X}_N\}{R}^{I_1\&times;I_2\&times;I_3};$

Output: the low-dimensional tensor camera lens set after the mapping $Y=\{{\mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">Y</figure-callout>}}_1,Y_2,L{Y}_N\}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ The intermediate conversion matrix $V_{}$ ${}_1\&Element;R^{I_1\&times;{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1},$ $V_2\&Element;R^{I_2\&times;J_2}$ With $V_3\&Element;R^{I_3\&times;J_3},$ And transition matrix $T_1^i=V_1^T{U}_1^i\&Element;i^{{\mathrm{<figure-callout\; id="1"\; label="J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">J</figure-callout>}}_1\&times;I_1},$ $T_2^i=V_2^T{U}_2^i\&Element;i^{J_2\&times;I_2}$ With $T_3^i=V_3^T{U}_3^i\&Element;i^{J_3\&times;I_3},$ And satisfy $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}};$ Arthmetic statement:

Step 1: make up an arest neighbors figure G;

Step 2: calculate weight matrix W;

Step 3:For k=1to3

Step 4:For i=1toN

Step 5: calculate X _iThe k mould launch matrix X _(k) ⁱThe left matrix U that decomposes of SVD _(k) ⁱ

Step 6:End;

Step 7: $D_U=\underset{i}{\mathrm{\&Sigma;}}{D_{\mathrm{ii}}{U}_{\left(k\right)}^i{U}_{\left(k\right)}^i}^T;$

Step 8: $W_U=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W_{\mathrm{ij}}{U}_{\left(k\right)}^i{U}_{\left(k\right)}^j}^T;$

Step 9: find the solution following generalized eigenvector problem to obtain optimized V _k:

(D _U-W _U)V _k＝λD _UV _k；

Step 10:For i=1toN

Step 11: $T_{\left(k\right)}^i=V_{\left(k\right)}^T{U}_{\left(k\right)}^i\&Element;R^{J_k\&times;I_k};$

Step 12:End;

Step 13:End;

Step 14:For i=1toN

Step 15: $Y_i=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}};$

Step 16:End.

The method that sorter model is set up in video tensor camera lens set after described employing supports the tensor machine to dimensionality reduction is: in this step, we adopt and support the tensor machine to train the sorter of tensor camera lens.The input of training pattern is exactly a previous step through the low-dimensional tensor Y that the subspace embeds and dimensionality reduction obtains _i, rather than original X _iSuch processing can not only improve degree of accuracy, and can improve the efficient of training and classification.

Support that the algorithm of tensor machine training classifier is as follows.

Input: the low-dimensional tensor camera lens set after the mapping $Y_i{|}_{i=1}^N=X_i{\&CircleTimes;}_1{\mathrm{<figure-callout\; id="1"\; label="T"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">T</figure-callout>}}_1^{\mathrm{iT}}{\&CircleTimes;}_2{T}_2^{\mathrm{iT}}{\&CircleTimes;}_3{T}_3^{\mathrm{iT}}\&Element;{\mathrm{<figure-callout\; id="1"\; label="R\; J"\; filenames="177298DEST\_PATH\_H200810059125601E00021.png,570736DEST\_PATH\_H200810059125601E00012.png"\; state="\{\{state\}\}">R</figure-callout>}}^{J_{}},$ And

Corresponding classification logotype y _i∈+1 ,-1};

Output: the tensor lineoid parameter of sorter model $w_k{|}_{k=1}^3\&Element;R^{J_k}$ With b ∈ R;

Arthmetic statement:

Step 1: w is set _k| _K=1 ³Be R ^JkIn the random units vector;

Step 2: repeating step 3-5 is until convergence;

Step 3:For j=1to3

Step 4: by finding the solution optimization problem $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ \begin{array}{cc}s.t.& \begin{array}{c}{y}_i\left[{w^T}_j(Y_i\underset{k=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_k{w}_k+b)\right]{\&GreaterEqual;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&GreaterEqual;0\end{array}\end{array}\end{array}\right]$ Obtain $w_i\&Element;R^{I_j}$ And b, wherein c is a constant, ξ is a relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2;$

Step 5:End;

Step 6: check whether restrain: if ${\mathrm{\&Sigma;}}_{k=1}^3\left[\right|w_{k,t}^T{w}_{k,t-1}|\left({\left|\right|w_{k,t}\left|\right|}_{\mathrm{Fro}}^{-2}\right)-1\&rsqb;\&le;\mathrm{\&epsiv;}$ Calculate so

w _k| _K=1 ³Restrain.Here w _{K, t}Be current projection vector, w _{K, t-1}It is previous projection vector;

Step 7:End.

Described for testing lens, after adding up to the transition matrix obtain to carry out projection by training set, carrying out semantic concept by sorter model again detects: in this step, we will come according to the sorter model that the front training obtains the outer new data of training set is detected.Because our dimension reduction method is linear,, carries out classification by sorter then and detect so can map directly to low n-dimensional subspace n for new data.

Make X _tAs an outer detection example of training set, following algorithm provides testing process.

Input: camera lens to be detected $X_t\&Element;R^{I_1\&times;I_2\&times;I_3},$ The intermediate conversion matrix V ₁, V ₂, V ₃, classifier parameters $w_k{|}_{k=1}^3\&Element;R^{I_k}$ With b ∈ R;

Output: X _tClassification logotype z _t∈+1 ,-1};

Arthmetic statement:

Step 1:For k=1to3;

Step 2: calculate X _tThe k mould launch matrix X _(k) ^tThe left matrix U that decomposes of SVD _(k) ^t

Step 3: calculate $T_{\left(k\right)}^t=V_{\left(k\right)}^T{U}_{\left(k\right)}^t;$

Step 4:End;

Step 5: calculate $Y_t=X_t{\&CircleTimes;}_1{T}_1^{t^T}{\&CircleTimes;}_2{T}_2^{t^T}{\&CircleTimes;}_3{T}_3^{t^T};$

Step 6: calculate z _t=sign (Y _{t 1}w ₁

₂w ₂

₃w ₃)+b;

Step 7:End.

Claims (5)

1. a multimodal video semantic concept detection method based on tensor representation, is characterized in that comprising the steps: 1) For the video shots in the training set and the test set, the bottom-level features of the image, audio, and text modes are extracted, and each video tensor shot is represented by a third-order tensor formed by these three bottom-level features; 2) According to the eigenstructure of the manifold space of the video tensor lens set, the dimension reduction and subspace embedding of the original high-dimensional tensor are realized by finding the transformation matrix; 3) Use the support tensor machine to establish a classifier model for the video tensor lens set after dimensionality reduction; 4) For the test shot, the transformation matrix calculated from the training set is projected, and then the semantic concept detection is performed through the classifier model; The expression of the video tensor lens: based on the underlying features of images, audio, and text extracted from the video, use a 3rd-order tensor for each video lens

to represent, where, I ₁ , I ₂ and I ₃ are the dimensions of image features, audio features and text features respectively, then each element

The value of is defined as:

is the value of the image feature, where 1≤i ₁ ≤I ₁ ;

is the value of the audio feature, where 1≤i ₂ ≤I ₂ ;

is the value of the text feature, where 1≤i ₃ ≤I ₃ ; the values of other elements are initially set to zero. 2. A kind of multimodal video semantic concept detection method based on tensor representation according to claim 1, characterized in that the described video shots in the training set and the test set all extract three kinds of images, audio and text. The underlying features of the modality: select a key frame in each video shot as a representative image, and then extract the color histogram, texture, and Canny boundary as image features; extract a piece of audio corresponding to the video shot as an audio example, and use the audio The example is divided into superimposed short-term audio frames, and the features of each short-term audio frame are extracted, including MFCC, centroid, attenuation cut-off frequency, spectral flow and zero-crossing rate, to form a frame feature vector, and then the statistics of the short-term audio frame feature vector Values are used as audio features of video shots; TF*IDF values are extracted from recognized transcripts in videos as text features. 3. a kind of multimodal video semantic concept detection method based on tensor representation according to claim 1, is characterized in that described according to the manifold space eigenstructure of video tensor lens, realizes pairing by finding transformation matrix The method of dimension reduction and subspace embedding of the original high-dimensional tensor is: given space

The lens data set X on X={X ₁ , X ₂ ,...X _N }, according to the manifold space eigenstructure of the tensor lens and the spectrogram theory, for each tensor lens X _i | _i=1 on X ^N finds three transformation matrices: J ₁ × I ₁ -dimensional T ₁ ⁱ , J ₂ × I _2- dimensional T ₂ ⁱ and J ₃ × I ₃ -dimensional T ₃ ⁱ , to map these N data points to the space

Y={Y ₁ , Y 2 ,...Y N } on Y={Y 1 , Y ₂ ,...Y _N }, satisfy $Y_i{|}_{i=1}^N=x_i{\&CircleTimes;}_1{T}_1^i{\&CircleTimes;}_2{T}_2^i{\&CircleTimes;}_3{T}_3^i,$ Among them, J ₁ <I ₁ , J ₂ <I ₂ , J ₃ <I ₃ , so as to realize dimension reduction and subspace embedding of the original high-dimensional tensor; when T ₁ ⁱ | _i=1 ^N is obtained, The optimized intermediate transformation matrix V ₁ is obtained by solving the generalized eigenvector problem (D _U -W _U ) V ₁ =λD _U V ₁ , where, ${\mathrm{D.}}_u=\underset{i}{\mathrm{\&Sigma;}}{\mathrm{D.}}_i{u}_1^i{u}_1^i,$ $W_u=\underset{\mathrm{ij}}{\mathrm{\&Sigma;}}{W}_{\mathrm{ij}}{u}_1^i{u}_1^J$ , and W is the weight matrix of the nearest neighbor graph constructed according to the training set X, and D is the diagonal matrix of W, namely ${\mathrm{D.}}_i={\mathrm{\&Sigma;}}_j{W}_{\mathrm{ij}},$ U ₁ ⁱ is the left matrix obtained by SVD decomposition of the mode-1 unfolding matrix of X _i | _{i = 1} ^N , that is, X ₍₁₎ ⁱ , and finally the conversion matrix can be calculated

Calculate T ₂ ⁱ and T ₃ ⁱ in the same way. 4. A kind of multi-modal video semantic concept detection method based on tensor representation according to claim 1, characterized in that said support tensor machine is used to establish a classifier model for the video tensor lens set after dimensionality reduction The method is: the input of the classifier model is a low-dimensional tensor obtained through subspace embedding and dimensionality reduction

and the corresponding class identity y _i ∈ {+1, -1}, the output is the tensor hyperplane parameter of the classifier model

Solve optimization problems iteratively $\left[\begin{array}{c}\underset{w_j,b,\mathrm{\&xi;}}{\min }{J}_{C-\mathrm{STM}}(w_j,b,\mathrm{\&xi;})=\frac{\mathrm{\&eta;}}{2}{\left|\right|w_j\left|\right|}_{\mathrm{Fro}}^2+c\underset{i=1}{\overset{N}{\mathrm{\&Sigma;}}}{\mathrm{\&xi;}}_i\\ \mathrm{the\; s}.t.\begin{array}{c}{\mathrm{the\; y}}_i\left[w_j^T(Y_i\underset{j=1}{\overset{3}{\mathrm{\&Pi;}}}{\&times;}_j{w}_j+b)\right]{\&Greater\; Equal;\mathrm{\&xi;}}_i,1\&le;i\&le;N\\ \mathrm{\&xi;}\&Greater\; Equal;0\end{array}\end{array}\right]$ Get w _k | _{k = 1} ³ and b, where the parameter j cycles from 1 to 3, and c is a constant, ξ is the relaxation factor, $\mathrm{\&eta;}={\mathrm{\&Pi;}}_{1\&le;k\&le;3}^{k\&NotEqual;j}{\left|\right|w_k\left|\right|}_{\mathrm{Fro}}^2.$ 5. A kind of multimodal video semantic concept detection method based on tensor representation according to claim 1, characterized in that for the test shot, after the transformation matrix calculated by the training set is projected, and then classified Semantic Concept Detection with Detector Models: New Data Outside the Training Set

by the transformation matrix

and

is mapped to the low-dimensional subspace

Category detection is then performed through the classifier model, that is, computing $Z_t=\mathrm{sign}\left(Y_t{\&CircleTimes;}_1{w}_1{\&CircleTimes;}_2{w}_2{\&CircleTimes;}_3{w}_3\right)+b,$ To get the category identification z _t ∈ {+1, -1} of the test data.

Images