CN104133807A - Method and device for learning cross-platform multi-mode media data common feature representation - Google Patents

Abstract

The invention discloses a method and device for learning cross-platform multi-mode media data common feature representation by using a denoising auto-encoder. The method comprises the steps that S1, an optimal object equation is established, wherein in the object equation, media data features of different platforms and different modes are reestablished by adopting the single-layer denoising auto-encoder, mode correlation constraint and cross-platform constraint are considered during reestablishing, and the media data features of the different platforms and the different modes comprise image features and textual features; S2, an analytical solution to the optimal object equation is obtained, and the globally optimal solution is obtained by searching for the point of which a partial derivative is zero in the analytical solution obtaining process; S3, the analytical solution is solved through a marginalization method, wherein solving is conducted by marginalizing random noise of feature vectors through the weak law of large numbers in the solving process. According to the method and device for learning cross-platform multi-mode media data common feature representation, the random noise of the feature vectors is marginalized by means of the weak law of large numbers.

Description

Learn method and device that cross-platform multi-modal media data common trait represents

Technical field

The invention belongs to Social Media (social media) and analyze and represent field across media characteristic, be specifically related to a kind of method of utilizing the cross-platform multi-modal media data common trait of denoising own coding device study to represent.

Background technology

Along with popularizing fast of Web2.0, information can and be shared for user's issue in increasing Social Media website (for example Flickr, YouTube, Facebook and Google).This makes to occur in people's event around and is recorded at faster speed and propagates and produced thereupon the media data of a large amount of different modalities, for example image, text and video.According to statistics, within 1 minute, have 3125 pictures to be uploaded to Flickr, 700K bar message is sent out on Facebook, and 2MM video is viewed on YouTube.The information that user uploads is not only huge, and is present in different platform with different mode.In these social multi-medium datas, exist valuable information, and be used in a large amount of application.For example, Social Media data stream (Twitter) is used to semantic video recommendation, social event prediction and image labeling in real time.Pictorial information on Flickr is successfully used to predict America, presidential elections, the monitoring of production marketing branch and sales volume of the product prediction in 2008.Human face expression in Social Media image is used to monitor public feelings information in presidential election.

In the various application of Social Media, crucial problem is how from broad medium data, to extract effective feature.Current most method is all the contextual information based on media, for example time, position and textual description.These descriptive informations are easily extracted, but have a large amount of media datas not comprise these contextual informations, therefore cannot obtain effective character representation.Content-based Social Media information extraction can address these problems.But there are three difficult points in content-based character representation: (1) Social Media data have multi-modal attribute.For example the media sample in Social Media website is conventionally simultaneously by image, text representation.(2) the cross-platform characteristic of Social Media data.For example may be present in Flickr and Facebook about the image of particular social event simultaneously.(3) feature that traditional-handwork is set still can not represent the semantic information comprising in multi-medium data effectively.

Summary of the invention

The object of the invention is the cross-platform multi-modal characteristic for Social Media data, improve the expression ability of low-level image feature by denoising own coding device (denoising auto-encoder), excavate the common semantic feature of different modalities data by the correlativity between maximization different modalities, learn the common trait of multi-medium data in different platform by cross-platform constraint and represent.

For achieving the above object, the invention provides a kind of method of utilizing the cross-platform multi-modal media data common trait of denoising own coding device study to represent, the method comprises the following steps:

Step S1, sets up optimization aim equation; In target equation, the media data feature that adopts the denoising own coding device of individual layer to rebuild different platform and different modalities, in the time rebuilding, considers Modal Correlation constraint and cross-platform constraint; Wherein, the media data feature of described different platform and different modalities comprises characteristics of image and text feature;

Step S2, tries to achieve the analytic solution of described optimization aim equation, asking in analytic solution process, obtains globally optimal solution by finding the point that partial derivative is zero;

Step S3, utilizes marginalisation method to solve obtained analytic solution, and the random noise of wherein carrying out marginalisation proper vector by the weak law of large number in solution procedure solves.

The present invention also provides a kind of device that utilizes the cross-platform multi-modal media data common trait of denoising own coding device study to represent, it is characterized in that, this device comprises:

Set up module, it is for setting up optimization aim equation; In target equation, the media data feature that adopts the denoising own coding device of individual layer to rebuild different platform and different modalities, in the time rebuilding, considers Modal Correlation constraint and cross-platform constraint; Wherein, the media data feature of described different platform and different modalities comprises characteristics of image and text feature;

Analytic solution module, it asking in analytic solution process, obtains globally optimal solution by finding the point that partial derivative is zero for trying to achieve the analytic solution of described optimization aim equation;

Solve module, it utilizes marginalisation method to solve obtained analytic solution, and the random noise of wherein carrying out marginalisation proper vector by the weak law of large number in solution procedure solves.

Beneficial effect of the present invention: adopt denoising own coding device can improve the ability to express of low-level image feature, Modal Correlation constraint is conducive to find maximally related feature between different modalities data, and platform adapts to constraint can reduce the difference that the multi-medium data feature in different platform distributes.

Brief description of the drawings

Fig. 1 is the schematic diagram of cross-platform multi-modal own coding device of the present invention.

Embodiment

For making the object, technical solutions and advantages of the present invention clearer, below in conjunction with specific embodiment, and with reference to accompanying drawing, the present invention is described in further detail.

The present invention supposes that two kinds of modal datas are respectively image and text, and these two kinds of modal datas are distributed in again two Social Media platform Google and Flickr.We use with represent respectively the n on Google platform ₁the image of individual media data sample and text feature, use with represent respectively the n on Flickr platform ₂the image of individual media data sample and text feature, here n ₂=n-n ₁, the number of n presentation medium data sample, i.e. the number sum of characteristics of image and text feature.In addition, we use X=[X _s, X _t] and Y=[Y _s, Y _t] represent the image and the text feature that combine on two platforms.With with presentation video feature x and text feature y have added the later proper vector of noise. the matrix of m X composition, the matrix of m Y composition, with be with the version that plus noise is later, m is the number of times of plus noise, described plus noise refers to that a part of element of choosing at random in matrix is become 0.

Target of the present invention is to have proposed a kind ofly to utilize method described in the method that the cross-platform multi-modal media data common trait of denoising own coding device study represents to comprise three parts: 1) set up optimization aim equation, 2) try to achieve analytic solution, 3) rapid solving of marginalisation.Specifically, said method comprising the steps of:

Step S1, sets up optimization aim equation.For the common trait of learning different platform media data represents, first we adopt the denoising own coding device (denoising auto-encoder) of individual layer to rebuild input feature vector.In individual layer denoising own coding device, input data are mapped to output layer with a linear mapping matrix, and output is the reconstruction of input data later to plus noise.We use respectively W _xand W _ythe linear mapping matrix of the denoising own coding device that presentation video and text are corresponding, this linear mapping matrix is for being mapped to same characteristic vector space by image and Text eigenvector, and then makes between the data in different platform, there is no platform difference.In addition we use with represent respectively the reconstructed object equation of two kinds of modal datas, i.e. the reconstruction error to image and Text eigenvector, uses with represent Modal Correlation constraint and cross-platform constraint, i.e. semantic difference between difference between data in different platform, and different modalities data.Final optimization aim equation can be expressed as:

Here λ _x, λ _y, λ _mcand λ _cdall represent regularization parameter.The denoising own coding device of only considering a kind of modal data from tradition is different, and in (1) formula, we have not only considered respectively the reconstructed object equation of image and two kinds of modal datas of text with also add Modal Correlation constraint with cross-platform constraint by solving (1) formula, we just can the data-mapping of image and two kinds of different modalities of text to Modal Correlation maximize and the feature space of platform difference after minimizing in.(1) the reconstructed object equation of the view data in formula and text data be defined as follows respectively:

Wherein, the mark of Tr representing matrix.

Concerning the media data of every kind of mode, the sample on Flickr and Google platform is combined structure Reconstructed equation.The distributional difference of the data that the character representation obtaining so just can reduce same mode in different platform.

In order to consider the correlativity between different modalities data, by means of canonical correlation analysis (CCA), we maximize two kinds of correlativitys between mode in rebuilding image and text data, we be defined as:

Here the variance matrix of presentation video data, represent the variance matrix of text data, it is the covariance matrix between image and text data.

In order to consider the media data difference in different platform, we reduce the media data difference in different platform by means of maximizing average difference (MMD).Specifically, be defined as follows:

Here φ _x(x)=W _xx, φ _y(y)=W _yy, $G_x=(\frac{1}{{\mathrm{mn}}_1}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{\tilde{x}}_{\mathrm{ij}}-\frac{1}{{\mathrm{mn}}_2}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{\tilde{x}}_{\mathrm{ij}}),$ $G_y=(\frac{1}{{\mathrm{mn}}_1}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{\tilde{y}}_{\mathrm{ij}}-\frac{1}{{\mathrm{mn}}_2}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{\tilde{y}}_{\mathrm{ij}}).$

Step S2, tries to achieve analytic solution.Described in step 1 with four represented functions of expression formula are all protruding, and what therefore (1) formula represented is a convex quadratic programming problem.The globally optimal solution of this quadratic programming problem can obtain by finding the point that partial derivative is zero.

about W _xpartial derivative may be calculated:

Here and $Q_x=2(({\mathrm{\λ}}_x+{\mathrm{\λ}}_c)C_{\mathrm{xx}}+{\mathrm{\λ}}_m{G}_x{G}_x^T).$ Similarly, we can obtain about W _ylocal derviation:

Here and $Q_y=2(({\mathrm{\λ}}_y+{\mathrm{\λ}}_c)C_{\mathrm{yy}}+{\mathrm{\λ}}_m{G}_y{G}_y^T).$

By solving equation group work as Q _xwith all reversible time, we obtain W _xand W _yanalytic solution:

$W_y=2({2\mathrm{\λ}}_c{\mathrm{\λ}}_x{\stackrel{\‾}{C}}_{\mathrm{xx}}{Q}_x^{-1}{C}_{\mathrm{xy}}+{\mathrm{\λ}}_y{\stackrel{\‾}{C}}_{\mathrm{yy}}){(Q_y-{4\mathrm{\λ}}_c^2{C}_{\mathrm{xy}}^T{Q}_x^{-1}{C}_{\mathrm{xy}})}^{-1}---\left(8\right)$

$W_x=2({\mathrm{\λ}}_c{W}_y{C}_{\mathrm{xy}}^T+{\mathrm{\λ}}_x{\stackrel{\‾}{C}}_{\mathrm{xx}})Q_x^{-1}---\left(9\right)$

In actual applications, Q _xwith conventionally be all reversible.If they are irreversible, we obtain approximate solution by pseudoinverse.

Step S3, the rapid solving of marginalisation.Tradition own coding device easily study obtains a nonsensical unit matrix.In order to address this problem, denoising own coding device is rebuild original feature vector with having added the later input feature value of noise.Plus noise operation is that a part of element by input vector is set is at random zero to realize.The major function of denoising own coding device is with not having the element information of losing to predict the element information of loss.The added random noise of original feature vector is conducive to catch the statistical information in input vector.In the optimization aim equation of introducing in step 1, original feature vector is added to the noise of m time.M is larger, and the linear mapping that study obtains is more effective, but has also increased time cost simultaneously.Here we carry out the random noise of proper vector in marginalisation (1) formula by the weak law of large number.

Suppose the change probability of each element when p represents proper vector plus noise.Therefore the probability that proper vector element is not changed by noise is 1-p.P is that needs pass through the predetermined parameter of test in algorithm is realized, and in our test, the value of p is can obtain good effect at 0.7 o'clock.C below _xx, c _yy, c _xy, G _xand G _ythe solution that adopts marginalisation Noise Method to obtain.

Suppose S=XX ^t, we can obtain C _xxmarginalisation after solution:

$E\left(C_{\mathrm{xx}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{x}}_i{\tilde{x}}_i^T\right]---\left(10\right)$

$E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E[x_i,{\tilde{x}}_i^T]---\left(11\right)$

Wherein, E[] represent to ask expectation; Known according to step 2, C _xxwith have random noise, regard random noise as stochastic variable, the object of above-mentioned formula is stochastic variable corresponding to mode cancellation noise by asking expectation; E (C _xx) and in the element value of the capable β of α row can direct representation be:

$E{\left(C_{\mathrm{xx}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{S}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ S_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.---\left(12\right)$

$E{\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)}_{\mathrm{\α},\mathrm{\β}}{S}_{\mathrm{\α\β}}(1-p)---\left(13\right)$

Wherein, 1-p herein represents the probability that the element value of the capable β row of α of s is not changed by noise.

Suppose R=YY ^t, we can obtain C _yywith marginalisation after solution:

$E\left(C_{\mathrm{yy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{y}}_i{\tilde{y}}_i^T\right]---\left(14\right)$

$E\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E[y_i,{\tilde{y}}_i^T]---\left(15\right)$

E (C _yy) and in the element value of the capable β of α row can direct representation be:

$E{\left(C_{\mathrm{yy}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{R}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ R_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.---\left(16\right)$

$E{\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)}_{\mathrm{\α},\mathrm{\β}}{R}_{\mathrm{\α\β}}(1-p)---\left(17\right)$

Wherein, (1-p) herein represents the probability that the element value of the capable β row of α of R is not changed by noise.

Suppose U=XY ^t, we can obtain C _xymarginalisation after solution:

$E\left(C_{\mathrm{xy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{x}}_i{\tilde{y}}_i^T\right]---\left(18\right)$

$E{\left(C_{\mathrm{xy}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{U}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ U_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.---\left(19\right)$

Wherein, 1-p herein represents the probability that the element value of the capable β row of α of U is not changed by noise.

Suppose $V=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{x}_i-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{x}_i),$ $Z=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{y}_i-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{y}_i),$ We can obtain G _xand G _yedge after solution:

$E\left(G_x\right)=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}E\left({\tilde{x}}_i\right)-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}E\left({\tilde{x}}_i\right))---\left(20\right)$

$E\left(G_y\right)=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}E\left({\tilde{y}}_i\right)-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}E\left({\tilde{y}}_i\right))---\left(21\right)$

E(G _x) _α＝V _α(1-p)，E(G _y) _α＝Z _α(1-p)

Wherein, 1-p herein represents α the probability that element value is not changed by noise in V or Z.

So far we can obtain the marginalisation formal solution of the described optimization method of formula (1) in step 1:

$W_y=({\mathrm{\λ}}_c{\mathrm{\λ}}_{x}E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)E{\left(Q_x\right)}^{-1}E\left(C_{\mathrm{xy}}\right)+{\mathrm{\λ}}_{y}E\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)){(E\left(Q_y\right)-{\mathrm{\λ}}_c^{2}E{\left(C_{\mathrm{xy}}\right)}^{T}E{\left(Q_x\right)}^{-1}E\left(C_{\mathrm{xy}}\right))}^{-1}---\left(22\right)$

$W_x={({\mathrm{\λ}}_c{W}_{y}E{\left(C_{\mathrm{xy}}\right)}^T+{\mathrm{\λ}}_{x}E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right))E\left(Q_x\right)}^{-1}---\left(23\right)$

Here E (Q _x)=2 ((λ _x+ λ _c) E (C _xx)+λ _me (G _x) E (G _x) ^t),

E(Q _y)＝2((λ _y+λ _c)E(C _yy)+λ _mE(G _y)E(G _y) ^T)。

Above-described specific embodiment; object of the present invention, technical scheme and beneficial effect are further described; be understood that; the foregoing is only specific embodiments of the invention; be not limited to the present invention; within the spirit and principles in the present invention all, any amendment of making, be equal to replacement, improvement etc., within all should being included in protection scope of the present invention.

Claims (6)

1. a method of utilizing the cross-platform multi-modal media data common trait of denoising own coding device study to represent, is characterized in that, the method comprises the following steps:

Step S1, sets up optimization aim equation; In target equation, the media data feature that adopts the denoising own coding device of individual layer to rebuild different platform and different modalities, in the time rebuilding, considers Modal Correlation constraint and cross-platform constraint; Wherein, the media data feature of described different platform and different modalities comprises characteristics of image and text feature;

Step S2, tries to achieve the analytic solution of described optimization aim equation, asking in analytic solution process, obtains globally optimal solution by finding the point that partial derivative is zero;

Step S3, utilizes marginalisation method to solve obtained analytic solution, and the random noise of wherein carrying out marginalisation proper vector by the weak law of large number in solution procedure solves.

2. method according to claim 1, is characterized in that, the optimization aim equation in step S1 is to adopt denoising own coding device structure, and the optimization aim equation of constructing can be expressed as:

Wherein, λ _x, λ _y, λ _mcand λ _cdall represent regularization parameter; W _xand W _ythe linear mapping matrix of the denoising own coding device that presentation video feature and text feature are corresponding; with the reconstructed object equation of presentation video feature and text feature respectively, with represent Modal Correlation constraint and cross-platform constraint.

3. method according to claim 2, is characterized in that, be defined as follows respectively:

Wherein, m represents the number of times of plus noise, the number of n presentation medium data characteristics vector, the matrix of m X composition, the matrix of m Y composition, X=[X _s, X _t] and Y=[Y _s, Y _t] represent the image characteristic matrix and the text feature matrix that combine in two different platforms, X={x _i| i=1 ..., n}, Y={y _i| i=1 ..., n}, with be with the version that plus noise is later, x _ithe proper vector of the i row of matrix X, for i*n+j column element; be defined as follows:

Wherein, the variance matrix of presentation video feature, represent the variance matrix of text feature, the covariance matrix between characteristics of image and text feature, the mark of Tr representing matrix;

be defined as follows:

Wherein, φ _x(x)=W _xx, φ _y(y)=W _yy, $G_x=(\frac{1}{{\mathrm{mn}}_1}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{\tilde{x}}_{\mathrm{ij}}-\frac{1}{{\mathrm{mn}}_2}\underset{j=1}{\overset{m}{\mathrm{\Σ}}}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{\tilde{x}}_{\mathrm{ij}}),$ n1 and n2 represent respectively the number of the media data feature in different platform.

4. method according to claim 3, is characterized in that, in step S2, obtains globally optimal solution by finding the point that the partial derivative of optimization aim equation is zero, specifically comprises:

about W _xpartial derivative may be calculated:

Wherein, representative with and $Q_x=2(({\mathrm{\λ}}_x+{\mathrm{\λ}}_c)C_{\mathrm{xx}}+{\mathrm{\λ}}_m{G}_x{G}_x^T);$

about W _ylocal derviation:

Wherein, $C_{\mathrm{yy}}=\tilde{Y}{\tilde{Y}}^T,$ ${\stackrel{\‾}{C}}_{\mathrm{yy}}=\stackrel{\‾}{Y}{\left(\tilde{Y}\right)}^T$ And $Q_y=2(({\mathrm{\λ}}_y+{\mathrm{\λ}}_c)C_{\mathrm{yy}}+{\mathrm{\λ}}_m{G}_y{G}_y^T);$

By solving equation group work as Q _xwith all reversible time, obtain W _xand W _yanalytic solution:

$W_y=2({2\mathrm{\λ}}_c{\mathrm{\λ}}_x{\stackrel{\‾}{C}}_{\mathrm{xx}}{Q}_x^{-1}{C}_{\mathrm{xy}}+{\mathrm{\λ}}_y{\stackrel{\‾}{C}}_{\mathrm{yy}}){(Q_y-{4\mathrm{\λ}}_c^2{C}_{\mathrm{xy}}^T{Q}_x^{-1}{C}_{\mathrm{xy}})}^{-1},$ $W_x=2({\mathrm{\λ}}_c{W}_y{C}_{\mathrm{xy}}^T+{\mathrm{\λ}}_x{\stackrel{\‾}{C}}_{\mathrm{xx}})Q_x^{-1};$

Work as Q _xwith when irreversible, obtain approximate solution by pseudoinverse.

5. method according to claim 1, is characterized in that, step S3 specifically comprises:

Suppose that p represents the probability to each element is changed by noise in proper vector;

Suppose S=XX ^t, obtain C _xxwith marginalisation after solution:

$E\left(C_{\mathrm{xx}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{x}}_i{\tilde{x}}_i^T\right],E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[x_i{\tilde{x}}_i^T\right]$

E (C _xx) and in the element value of the capable β of α row can direct representation be:

$E{\left(C_{\mathrm{xx}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{S}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ S_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.,E{\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)}_{\mathrm{\α},\mathrm{\β}}=S_{\mathrm{\α\β}}(1-p)$

Wherein, X=[X _s, X _t] represent the image characteristic matrix that combines in two different platforms, X={x _i| i=1 ..., n}, x _iit is the proper vector of the i row in X; to x _iproper vector after plus noise; E[] represent to ask expectation; S _{α β}for the capable β column element of the α in S value;

Suppose R=YY ^t, obtain C _yywith marginalisation after solution:

$E\left(C_{\mathrm{yy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{y}}_i{\tilde{y}}_i^T\right],E\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[y_i{\tilde{y}}_i^T\right]$

E (C _yy) and in the element value of the capable β of α row can direct representation be:

$E{\left(C_{\mathrm{yy}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{R}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ R_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.,E{\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)}_{\mathrm{\α},\mathrm{\β}}=R_{\mathrm{\α\β}}(1-p)$

Wherein, Y=[Y _s, Y _t] represent the text feature matrix that combines in two different platforms, Y={y _i| i=1 ..., n}, y _iit is the i row characteristic of correspondence vector in Y; to y _iproper vector after plus noise; R _{α β}for the capable β column element of the α in R value;

Suppose U=XY ^t, obtain C _xymarginalisation after solution:

$E\left(C_{\mathrm{xy}}\right)=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}E\left[{\tilde{x}}_i{\tilde{y}}_i^T\right],E{\left(C_{\mathrm{xy}}\right)}_{\mathrm{\α},\mathrm{\β}}=\left\{\begin{array}{ccc}{U}_{\mathrm{\α\β}}{(1-p)}^2& \mathrm{if}& \mathrm{\α}\≠\mathrm{\β}\\ U_{\mathrm{\α\β}}(1-p)& \mathrm{if}& \mathrm{\α}=\mathrm{\β}\end{array}\right.$

Wherein, U _{α β}for the capable β column element of the α in U value;

Suppose $V=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{x}_i-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{x}_i),$ $Z=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}{y}_i-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}{y}_i),$ Obtain G _xand G _yedge after solution:

$E\left(G_x\right)=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}E\left({\tilde{x}}_i\right)-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}E\left({\tilde{x}}_i\right)),$ $E\left(G_y\right)=(\frac{1}{n_1}\underset{i=1}{\overset{n_1}{\mathrm{\Σ}}}E\left({\tilde{y}}_i\right)-\frac{1}{n_2}\underset{i=n_1+1}{\overset{n}{\mathrm{\Σ}}}E\left({\tilde{y}}_i\right))$

E(G _x) _α＝V _α(1-p)，E(G _y) _α＝Z _α(1-p)

Wherein, V _αand Z _αbe respectively α element in vectorial V and Z;

The marginalisation formal solution of optimization aim equation is expressed as follows:

$W_y=({\mathrm{\λ}}_c{\mathrm{\λ}}_{x}E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right)E{\left(Q_x\right)}^{-1}E\left(C_{\mathrm{xy}}\right)+{\mathrm{\λ}}_{y}E\left({\stackrel{\‾}{C}}_{\mathrm{yy}}\right)){(E\left(Q_y\right)-{\mathrm{\λ}}_c^{2}E{\left(C_{\mathrm{xy}}\right)}^{T}E{\left(Q_x\right)}^{-1}E\left(C_{\mathrm{xy}}\right))}^{-1}$

$W_x={({\mathrm{\λ}}_c{W}_{y}E{\left(C_{\mathrm{xy}}\right)}^T+{\mathrm{\λ}}_{x}E\left({\stackrel{\‾}{C}}_{\mathrm{xx}}\right))E\left(Q_x\right)}^{-1}$

Wherein, E (Q _x)=2 ((λ _x+ λ _c) E (C _xx)+λ _me (G _x) E (G _x) ^t),

E(Q _y)＝2((λ _y+λ _c)E(C _yy)+λ _mE(G _y)E(G _y) ^T)。

6. a device that utilizes the cross-platform multi-modal media data common trait of denoising own coding device study to represent, is characterized in that, this device comprises:

Set up module, it is for setting up optimization aim equation; In target equation, the media data feature that adopts the denoising own coding device of individual layer to rebuild different platform and different modalities, in the time rebuilding, considers Modal Correlation constraint and cross-platform constraint; Wherein, the media data feature of described different platform and different modalities comprises characteristics of image and text feature;

Analytic solution module, it asking in analytic solution process, obtains globally optimal solution by finding the point that partial derivative is zero for trying to achieve the analytic solution of described optimization aim equation;

Solve module, it utilizes marginalisation method to solve obtained analytic solution, and the random noise of wherein carrying out marginalisation proper vector by the weak law of large number in solution procedure solves.