CN104992188A - Distributed handwritten digit recognition method based on t mixed factor analysis - Google Patents

Abstract

The invention discloses a distributed handwritten digit recognition method based on t mixed factor analysis. The distributed handwritten digit recognition method comprises the steps that: features of acquired handwritten digits are extracted at each node; feature data corresponding to each digit used for training are initialized; each node calculates local statistics based on its training data and broadcasts the local statistics to its neighbour nodes; each node calculates joint statistics according to the received local statistics from all neighbour nodes, and estimates parameters in t mixed factor analysis on the basis of the joint statistics; and the distributed training process is completed. In the distributed recognition state, the data for test can be input to any mode, a log likelihood value of tMFA corresponding to each trained digit is calculated, and the digit corresponding to the maximum log likelihood value is taken as the recognition result. By adopting the tMFA, the distributed handwritten digit recognition method has high robustness for off-group points in the data, adopts the distributed training and recognizing method, and avoids network collapse caused by a center node.

Description

A kind of distributed Handwritten Digit Recognition method analyzed based on t hybrid cytokine

Technical field

The present invention relates to a kind of distributed Handwritten Digit Recognition method analyzed based on t hybrid cytokine, belong to the parallel and distributed process method of data and the technical field of application.

Background technology

At present, Handwritten Digit Recognition be in pattern-recognition category one have a challenging problem, main research be enable computing machine independently recognize the numeral that staff is write.Handwritten Digit Recognition has application in a lot, as: the many-sides such as extensive data statistics, finance, the tax, finance and sorting mail are all widely used.There is existing accomplished in many ways Handwritten Digit Recognition at present.But when data volume is larger, single computing machine cannot process fast and effectively, therefore needs handwritten numeral data to be divided into multiple part, stores on a different computer respectively, by the collaboration method between designing a calculating machine, thus achieve the distributed treatment of data.

In pattern-recognition, machine learning field, Handwritten Digit Recognition belongs to the category of supervised learning.In the training stage, based on markd sample, set up and train the model of the distribution that can describe handwritten numeral characteristic, at ensuing cognitive phase, numerical characteristic data to be identified and all kinds of characteristic model of representative trained are compared, thus find most suitable model, as final recognition result.Needing to carry out in the scene of distributed treatment, in the training stage, how by the cooperation between computer node, making each computer node can make full use of the data of other each computer nodes, thus estimating consistent model; At cognitive phase, whether handwritten numeral data to be identified can be inputted any one node, correct recognition result can be obtained, very crucial.The method that the present invention proposes can solve the problem effectively, obtains good recognition correct rate.

Summary of the invention

The object of the invention is the defect solving prior art, propose a kind of in sensor network the distributed clustering method based on hybrid cytokine analytical model.

The present invention solves the technical scheme that its technical matters takes: a kind of distributed Handwritten Digit Recognition method analyzed based on t hybrid cytokine, the method comprises the steps:

Step 1: the collection of data and feature extraction;

Be provided with M platform computing machine/computing node (that is: node), form a network, m node collects from numeral 0 ~ 9 by the handwriting pad be attached thereto, the raw data of totally 10 classes, handwriting pad records the two-dimensional coordinate position of each point on each character writing track automatically, gets the coordinate of 8 points in the first-class compartment of terrain of track as the characteristic s corresponding to each raw data, totally 16 dimensions.In order to represent convenient, if the collection of node m place the training dataset of the digital d obtained through feature extraction are wherein represent node m place, for n-th characteristic of handwritten numeral d of training, dimension is p, for the training data number of digital d.

Describe by t hybrid cytokine analytical model (tMFA) distribution, note, all Nodes carry out modeling about the public same tMFA of training data of digital d.TMFA is a component number is the mixture model of I; For each data it can be expressed as:

$s_{m,n}^{\left(d\right)}={\mathrm{\μ}}_i+A_i{u}_{m,n}^{\left(d\right)}+e_{m,n,i}^{\left(d\right)}$ With probability π _i(i=1 ..., I),

Wherein, μ _ibe i-th blending constituent mean value vector, dimension is p; for with the factor in corresponding lower dimensional space, its dimension is q (q < < p), obeys t distribution the value of q is chosen according to the size of p in particular problem, generally gets the arbitrary integer between q=p/5 ~ p/3; A _iit is the Factor load-matrix of (p × q) of i-th blending constituent; Error variance obey t distribution wherein D _ifor the diagonal matrix of (p × p), ν _iit is the degree of freedom of i-th blending constituent; The weight π of each blending constituent _imeet so the parameter sets Θ of tMFA is { π _i, A _i, μ _i, D _i, ν _i} _{i=1 ..., I}.Note, for all nodes, in its tMFA parameter sets to be estimated, parameters value is identical.Here it should be noted that t distribution can be launched into the integration of Gaussian distribution and Gamma distribution:

$t\left(u_{m,n}^{\left(d\right)}|0,I_q,{\mathrm{\&nu;}}_i\right)=\&Integral;N\left(u_{m,n}^{\left(d\right)}|0,I_q/w_{m,n,i}^{\left(d\right)}\right)\&CenterDot;Gamma\left(w_{m,n,i}^{\left(d\right)}|{\mathrm{\&nu;}}_i/2,{\mathrm{\&nu;}}_i/2\right){\mathrm{dw}}_{m,n,i}^{\left(d\right)}$

$t\left(e_{m,n,i}^{\left(d\right)}|0,I_q,{\mathrm{\&nu;}}_i\right)=\&Integral;N\left(e_{m,n,i}^{\left(d\right)}|0,D_i/w_{m,n,i}^{\left(d\right)}\right)\&CenterDot;Gamma\left(w_{m,n,i}^{\left(d\right)}|{\mathrm{\&nu;}}_i/2,{\mathrm{\&nu;}}_i/2\right){\mathrm{dw}}_{m,n,i}^{\left(d\right)}$

Wherein, be with corresponding integration hidden variable.

In addition, the data transmission range of each node is set to Dis, and for present node m, all nodes being less than Dis with its distance are its neighbor node, and the neighbor node set expression of node m is R _m.Illustrate the relation between each node in certain network in Fig. 1, wherein computer icon representation node, if there is limit to be connected between two nodes, then represents and can communicate mutually between two nodes, transmission information.The R of the m of the empty wire frame representation node in Fig. 1 _m.In the present invention, network topology determines in advance, only needs to ensure at least to exist between any two nodes one directly or the path that can arrive through multi-hop.

Step 2: distributed training, will for distributed training, obtain the tMFA parameter sets of every class numeral corresponding to d ${\mathrm{\&Theta;}}^{\left(d\right)}={\{{\mathrm{\&pi;}}_i^{\left(d\right)},A_i^{\left(d\right)},{\mathrm{\&mu;}}_i^{\left(d\right)},D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\}}_{i=1,...,I},(d=0,...,9);$

After the tMFA that network topology and data of description distribute establishes, then start distributed training.Here be trained for example with digital d, as shown in Figure 2, its concrete steps are as follows for training process:

Step 2-1: initialization; Setting tMFA in be mixed into mark I.Here I determines the complexity of tMFA model, and I can get the arbitrary integer in 3 ~ 8, gets I=5 and can obtain good performance in Handwritten Digit Recognition.The initial value of parameter in MFA is set according to the dimension p of I and data.Wherein, each Nodes $({\mathrm{\&pi;}}_1^0,...,{\mathrm{\&pi;}}_i^0,...,{\mathrm{\&pi;}}_I^0)=(1/I,...,1/I,...,1/I);\{{\mathrm{\&mu;}}_1^0,...,{\mathrm{\&mu;}}_i^0,...,{\mathrm{\&mu;}}_I^0\}$ Random selecting from the data that this node collects; with in each element generate from standardized normal distribution N (0,1); this group parameter gets the arbitrary integer between 1 ~ 5.In addition, and each node l (l=1 ..., M) data amount check that collected be broadcast to its neighbor node.After certain node m receives its data amount check of all neighbor nodes broadcast, this node calculate weight c _lm:

$c_{lm}=\frac{N_l^{\left(d\right)}}{{\mathrm{\&Sigma;}}_{l^{\&prime;}\&Element;R_m}{N}_{l^{\&prime;}}^{\left(d\right)}},$

The implication of this weight is each neighbor node l (the l ∈ R for weighing node m _m) each importance of information at node m place transmitted.After initialization completes, iteration count iter=1, starts iterative process.

Step 2-2: calculate local statistic and broadcast; This step does not need the information of neighbor node.At each node l place, based on the data that it collects first g is calculated _i, Ω _i, with

$g_i={\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;}^{-1}\&CenterDot;A_i^{old},$

${\mathrm{\&Omega;}}_i=I_q-g_i^T{A}_i^{old},$

${\stackrel{\&OverBar;}{u}}_{l,n,i}^{\left(d\right)}=g_i^T(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old}),$

$W_{l,n,i}^{\left(d\right)}={(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})}^T\&CenterDot;\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;\&CenterDot;(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})$

$<z_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&pi;}}_i^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_i^{old},A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old},{\mathrm{\&nu;}}_i^{old})}{{\&Sigma;}_{i^{\&prime;}=1}^I{\mathrm{\&pi;}}_{i^{\&prime;}}^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_{i^{\&prime;}}^{old},A_{i^{\&prime;}}^{old}{\left(A_{i^{\&prime;}}^{old}\right)}^T+D_{i^{\&prime;}}^{old},{\mathrm{\&nu;}}_{i^{\&prime;}}^{old})},$

$<w_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&nu;}}_i^{old}+p}{{\mathrm{\&nu;}}_i^{old}+W_{l,n,i}^{\left(d\right)}},$

Wherein for a front iteration complete after obtain parameter value (that is: first iteration time be the initial value of parameter represent node l place n-th data belong to the probability of i-th class (that is: blending constituent), for expectation value.

Had the result of calculation of above-mentioned variable, node calculate goes out local statistic (LS) ${\mathrm{LS}}_l={\{{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;\}}_{i=1,...,I},$ As follows:

${\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)},$

${\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)}\&CenterDot;{\left(s_{l,n}^{\left(d\right)}\right)}^T,$

${\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;log<w_{l,n,i}^{\left(d\right)}>.$

Finally, the local statistic LS that will calculate of each node l _lbroadcast is spread to its neighbor node, as shown in Figure 1.

Step 2-3: calculate associating statistic; When node m (m=1 ..., M) receive from its all neighbor node l (l ∈ R _m), node m calculates associating statistic ${\mathrm{CS}}_m={\{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;\}}_{i=1,...,I}:$

${\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;.$

Step 2-4: each parameter in estimation model; Node m (m=1 ..., M) CS that calculates according to previous step _m, estimate each parameter Θ={ π _i, A _i, μ _i, D _i, ν _i} _{i=1 ..., I}, wherein, { π _i, μ _i} _{i=1 ..., I}estimation procedure as follows:

${\mathrm{\&pi;}}_i=\frac{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}{{\mathrm{\&Sigma;}}_{i^{\&prime;}=1}^I{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i^{\&prime;}\&rsqb;},$

${\mathrm{\&mu;}}_i=\frac{{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;};$

For { A _i, D _i} _{i=1 ..., I}estimation, process is as follows:

$V_i=\frac{{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;-2{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;\&CenterDot;{\mathrm{\&mu;}}_i^T+{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;\&CenterDot;{\mathrm{\&mu;}}_i\&CenterDot;{\mathrm{\&mu;}}_i^T}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;},$

$A_i=V_i{g}_i{(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)}^{-1},$

$D_i=diag\{V_i-A_i(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)A_i^T\};$

In addition, for { ν _i} _{i=1 ..., I}, obtained by solution equation below:

$\log \left(\frac{{\mathrm{\&nu;}}_i}{2}\right)-\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i}{2}\right)+1-\frac{{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;-{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}-\log \left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)+\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)=0,$

Wherein ψ () the digamma function that is standard, specifically solves employing Newton method.

Step 2-5: judge whether convergence; Node m (m=1 ..., M) calculate log-likelihood under current iteration iter:

$\log p\left(S_m^{\left(d\right)}\right|\mathrm{\&Theta;})=\underset{n=1}{\overset{N_m}{\&Sigma;}}log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i\&CenterDot;t(s_{m,n}^{\left(d\right)}|{\mathrm{\&mu;}}_i,A_i{A}_i^T+D_i,{\mathrm{\&nu;}}_i\left)\right),$

If then algorithm convergence, stops iteration; Otherwise perform step 2-2, start next iteration (iter=iter+1; ).Wherein Θ represents the parameter value that current iteration estimates, Θ ^oldrepresent the parameter value estimated in last iteration.That is, the log-likelihood of adjacent twice iteration is less than threshold epsilon, algorithm convergence.ε gets 10 ^-5~ 10 ^-6in arbitrary value.It should be noted that because node each in network is parallel data processing, therefore allly can not to restrain in an iteration simultaneously.Such as, when when node l restrains, node m not yet restrains, then node l no longer sends LS _l, also no longer receive the information of neighbor node transmission.Node m is then with the LS that the node l received for the last time sends _lupgrade its CS _m.The node of not restraining continues iteration, until all nodes are all restrained in network.

After above-mentioned steps 2-1 ~ step 2-5, the corresponding model tMFA (that is: representing with parameter Θ during training convergence) obtained by the training data of handwritten numeral d.Repeat above-mentioned steps 10 times, thus obtain 10 numerals tMFA model corresponding separately, in order to represent convenient, and being distinguished, using ${\mathrm{\&Theta;}}^{\left(d\right)}={\{{\mathrm{\&pi;}}_i^{\left(d\right)},A_i^{\left(d\right)},{\mathrm{\&mu;}}_i^{\left(d\right)},D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\}}_{i=1,...,I},(d=0,1,...,9)$ The tMFA model that representative digit d is corresponding.Distributed training completes.

Step 3: Distributed identification; When the arbitrary computer acquisition in network to new for identify handwritten numeral time, first obtain its character pair by step (1), be expressed as s', then calculate about Θ ^(d)(d=0,1 ..., 9) log-likelihood logp (s'| Θ ^(d)) (d=0,1 ..., 9):

$\log p\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)})=log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i^{\left(d\right)}\&CenterDot;t(s^{\&prime;}|{\mathrm{\&mu;}}_i^{\left(d\right)},A_i^{\left(d\right)}{\left(A_i^{\left(d\right)}\right)}^T+D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\left)\right);$

Using the recognition result d' of sequence number corresponding for max log likelihood value as s':

$d^{\&prime;}=\arg {\max }_{d=0}^9\log p\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)}).$

The idiographic flow of Distributed identification method of the present invention as shown in Figure 2.

Beneficial effect:

1. the t hybrid cytokine analysis adopted in the present invention, to the outlier existed in data, there is higher robustness, and high dimensional data can be described better, thus better obtain the model corresponding with data, thus also can obtain better training and recognition performance.

2. the distributed training process based on t hybrid cytokine analytical model adopted in the present invention, the information comprised in each computer node in network can be made full use of data that other computer node collects, thus the more accurate model of training place.

3. the distributed training process based on t hybrid cytokine analytical model adopted in the present invention; in computer node cooperating process; exchange local statistic instead of directly transmit raw data; because the quantity of local statistic and dimension are much smaller than data; therefore this mode saves the expense of communication on the one hand; on the other hand, be conducive to the privacy information adequately protected in data, improve the security performance of the system adopting this method.

4. the Distributed identification process based on t hybrid cytokine analytical model adopted in the present invention, can gather new data in any computer node place in a network, can obtain identical recognition result.

Accompanying drawing explanation

Fig. 1 is the neighbor node collection R of nodes m of the present invention _m, and between node, receive and dispatch the schematic diagram of local statistic.

Fig. 2 is the process flow diagram of the distributed Handwritten Digit Recognition method based on the analysis of t hybrid cytokine that the present invention relates to.

Fig. 3 is method of the present invention and centralized tMFA, without confusion matrix (that is: ConfusionMatrix) the hinton schematic diagram that cooperation tMFA method recognition result is corresponding.

Fig. 4 is method of the present invention and centralized tMFA, without average and the variance schematic diagram of the recognition correct rate of cooperation tMFA method.

Embodiment

Below in conjunction with Figure of description, the invention is described in further detail.

In order to a kind of distributed Handwritten Digit Recognition method analyzed based on t hybrid cytokine that the present invention relates to is described better.We describe with a concrete application example.

(1) collection of data and feature extraction: establish and always have 44 people, everyone each digital handwriting 25 times, is total up to 25*10*44=11000 raw data.20 computing machine/nodes (M=20) altogether in network, the neighbor node number of each node is 3, has directly or multihop path intercommunication between any two nodes.The raw data (250*30=7500) of the wherein handwritten numeral of 30 people, for distributed training, is divided into 20 parts, is assigned randomly in 20 nodes.For each raw data, equally spaced get the coordinate of 8 points on its track as the characteristic s corresponding to this raw data, totally 16 dimensions.In order to represent convenient, if the training dataset of digital d that node m place obtains through feature extraction is wherein represent node m place, for n-th characteristic of handwritten numeral d of training, dimension is p=16, for the training data number of digital d.

(2) distributed training: after step (1) completes, start distributed training.Here be trained for example with digital d, carry out the distribution of Modelling feature data with tMFA as shown in Figure 2, its concrete steps are as follows for training process:

(2-1) initialization: setting tMFA in be mixed into mark I=5.The initial value of parameter in MFA is set according to the dimension p of I and data.Wherein, each Nodes $({\mathrm{\&pi;}}_1^0,...,{\mathrm{\&pi;}}_i^0,...,{\mathrm{\&pi;}}_I^0)=(1/I,...,1/I,...,1/I);\{{\mathrm{\&mu;}}_1^0,...,{\mathrm{\&mu;}}_i^0,...,{\mathrm{\&mu;}}_I^0\}$ Random selecting from the data that this node collects; with in each element generate from standardized normal distribution N (0,1); in addition, and each node l (l=1 ..., M) data amount check that collected be broadcast to its neighbor node.After certain node m receives its data amount check of all neighbor nodes broadcast, this node calculate weight c _lm:

$c_{lm}=\frac{N_l^{\left(d\right)}}{{\mathrm{\&Sigma;}}_{l^{\&prime;}\&Element;R_m}{N}_{l^{\&prime;}}^{\left(d\right)}},$

The implication of this weight is each neighbor node l (the l ∈ R for weighing node m _m) each importance of information at node m place transmitted.After initialization completes, iteration count iter=1, starts iterative process.

(2-2) calculate local statistic and broadcast: this step does not need the information of neighbor node.At each node l place, based on the data that it collects first intermediate variable g is calculated _i, Ω _i, ${\stackrel{\&OverBar;}{u}}_{l,n,i}^{\left(d\right)},W_{l,n,i}^{\left(d\right)},<z_{l,n,i}^{\left(d\right)}>$ With $<w_{l,n,i}^{\left(d\right)}>(n=1,...,N_l^{\left(d\right)};i=1,...,I):$

$g_i={\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;}^{-1}\&CenterDot;A_i^{old},$

${\mathrm{\&Omega;}}_i=I_q-g_i^T{A}_i^{old},$

${\stackrel{\&OverBar;}{u}}_{l,n,i}^{\left(d\right)}=g_i^T(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old}),$

$W_{l,n,i}^{\left(d\right)}={(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})}^T\&CenterDot;\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;\&CenterDot;(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})$

$<z_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&pi;}}_i^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_i^{old},A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old},{\mathrm{\&nu;}}_i^{old})}{{\&Sigma;}_{i^{\&prime;}=1}^I{\mathrm{\&pi;}}_{i^{\&prime;}}^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_{i^{\&prime;}}^{old},A_{i^{\&prime;}}^{old}{\left(A_{i^{\&prime;}}^{old}\right)}^T+D_{i^{\&prime;}}^{old},{\mathrm{\&nu;}}_{i^{\&prime;}}^{old})},$

$<w_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&nu;}}_i^{old}+p}{{\mathrm{\&nu;}}_i^{old}+W_{l,n,i}^{\left(d\right)}},$

Wherein for last iteration complete after the parameter value that obtains (be the initial value of parameter first during iteration represent node l place n-th data belong to the probability of i-th class (blending constituent), for expectation value.

Had the result of calculation of above-mentioned intermediate variable, node calculate goes out local statistic (LS) ${\mathrm{LS}}_l={\{{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;\}}_{i=1,...,I},$ As follows:

${\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)},$

${\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)}\&CenterDot;{\left(s_{l,n}^{\left(d\right)}\right)}^T,$

${\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;log<w_{l,n,i}^{\left(d\right)}>.$

Finally, the local statistic LS that will calculate of each node l _lbroadcast is spread to its neighbor node, as shown in Figure 1.

(2-3) calculate associating statistic: when node m (m=1 ..., M) receive from its all neighbor node l (l ∈ R _m), node m calculates associating statistic ${\mathrm{CS}}_m{\left\{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;\right\}}_{i=1,...,I}:$

${\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;.$

(2-4) each parameter in estimation model: node m (m=1 ..., M) CS that calculates according to previous step _m, estimate Θ={ π _i, A _i, μ _i, D _i, ν _i} _{i=1 ..., I}, wherein, { π _i, μ _i} _{i=1 ..., I}estimation procedure as follows:

${\mathrm{\&pi;}}_i=\frac{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}{{\mathrm{\&Sigma;}}_{i^{\&prime;}=1}^I{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i^{\&prime;}\&rsqb;},$

${\mathrm{\&mu;}}_i=\frac{{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;};$

For { A _i, D _i} _{i=1 ..., I}estimation, process is as follows:

$V_i=\frac{{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;-2{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;\&CenterDot;{\mathrm{\&mu;}}_i^T+{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;\&CenterDot;{\mathrm{\&mu;}}_i\&CenterDot;{\mathrm{\&mu;}}_i^T}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;},$

$A_i=V_i{g}_i{(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)}^{-1},$

$D_i=diag\{V_i-A_i(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)A_i^T\};$

In addition, for { ν _i} _{i=1 ..., I}, obtained by solution equation below:

$\log \left(\frac{{\mathrm{\&nu;}}_i}{2}\right)-\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i}{2}\right)+1-\frac{{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;-{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}-\log \left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)+\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)=0,$

Wherein ψ () the digamma function that is standard, generally with the above-mentioned equation of Newton method solution.

(2-5) convergence is judged whether: node m (m=1 ..., M) calculate log-likelihood under current iteration iter:

$\log p\left(S_m^{\left(d\right)}\right|\mathrm{\&Theta;})=\underset{n=1}{\overset{N_m}{\&Sigma;}}log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i\&CenterDot;t(s_{m,n}^{\left(d\right)}|{\mathrm{\&mu;}}_i,A_i{A}_i^T+D_i,{\mathrm{\&nu;}}_i\left)\right),$

If then algorithm convergence, stops iteration; Otherwise perform step (2-2), start next iteration (iter=iter+1; ).Wherein Θ represents the parameter value that current iteration estimates, Θ ^oldrepresent the parameter value estimated in last iteration.That is, the log-likelihood of adjacent twice iteration is less than threshold epsilon, algorithm convergence.ε gets 10 ^-5~ 10 ^-6in arbitrary value.It should be noted that because node each in network is parallel data processing, therefore allly can not to restrain in an iteration simultaneously.Such as, when when node l restrains, node m not yet restrains, then node l no longer sends LS _l, also no longer receive the information of neighbor node transmission.Node m is then with the LS that the node l received for the last time sends _lupgrade its CS _m.The node of not restraining continues iteration, until all nodes are all restrained in network.

After above-mentioned steps (2-1) ~ (2-5), the corresponding model tMFA (representing with parameter Θ during training convergence) obtained by the training data of handwritten numeral d.Repeat above-mentioned steps 10 times, thus obtain 10 numerals tMFA model corresponding separately, in order to represent convenient, and being distinguished, using ${\mathrm{\&Theta;}}^{\left(d\right)}={\{{\mathrm{\&pi;}}_i^{\left(d\right)},A_i^{\left(d\right)},{\mathrm{\&mu;}}_i^{\left(d\right)},D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\}}_{i=1,...,I},(d=0,1,...,9)$ The tMFA model that representative digit d is corresponding.Distributed training completes.

(3) Distributed identification: when the arbitrary computer acquisition in network to new for identify handwritten numeral time, first obtain its character pair by step (1), be expressed as s', then calculate about Θ ^(d)(d=0,1 ..., 9) log-likelihood logp (s'| Θ ^(d)) (d=0,1 ..., 9):

$\log p\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)})=log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i^{\left(d\right)}\&CenterDot;t(s^{\&prime;}|{\mathrm{\&mu;}}_i^{\left(d\right)},A_i^{\left(d\right)}{\left(A_i^{\left(d\right)}\right)}^T+D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\left)\right);$

Using the recognition result d' of sequence number corresponding for max log likelihood value as s':

$d^{\&prime;}=\arg {\max }_{d=0}^9\log p\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)}),$

Distributed identification flow process of the present invention as shown in Figure 2.

Performance evaluation:

Because digital realistic value to be identified is known, compare adopting recognition methods involved in the present invention and its true value, obtain recognition correct rate (that is: quantity/(20*3500) of the handwritten numeral that recognition correct rate=all nodes correctly identify), thus can evaluate and weigh out validity and the accuracy of method involved in the present invention.In order to compare the present invention propose based on the distributed Handwritten Digit Recognition method (being called for short distributed tMFA) of tMFA and the performance of additive method, here with based on the centralized Handwritten Digit Recognition method (being called for short centralized tMFA) of tMFA, compare without the Handwritten Digit Recognition method (being called for short without cooperation tMFA) of cooperation between each node based on tMFA.It should be noted that; in centralized tMFA; all nodes need by original data transmissions to certain Centroid, adopt traditional MFA to carry out training and identified, then again result is returned to each node by Centroid; this mode in practice little; one is that transmission raw data communication expense is very large, once occur that packet loss or packet damage, very large on the impact of last recognition performance; two is the secret protections be unfavorable in data, and network security causes anxiety.Here object is whether the recognition methods in order to compare the distributed tMFA that the present invention proposes can reach the same performance of centralized tMFA.Recognition result represents by quantitative and qualitative analysis two kinds of modes respectively.In the qualitative representation of result, adopt the hinton figure of confusion matrix, as shown in Figure 3.In the figure, each recognition result of row representative digit 0 ~ 9 and the true value of each row representative digit 0 ~ 9.Blockage on principal diagonal represents correct situation about identifying, the size of blockage shows that more greatly the numeral of this identification correct is more, and other positions occurs blockage shows to exist the situation of wrong identification.As can be seen from this figure, centralized tMFA and distributed tMFA of the present invention (only provide node 3 wherein as space is limited, other nodes come to the same thing) better performances, and without cooperation tMFA poor-performing.In the quantificational expression of result, adopt average and the variance two indices of discrimination, as shown in Figure 4.In the figure, the recognition correct rate of the distributed tMFA of the present invention's design is substantially identical with the average of the recognition correct rate of centralized tMFA, and without cooperating the poor of tMFA, the variance of the recognition correct rate of distributed tMFA is also much smaller than nothing cooperation tMFA in addition.Therefore, adopt method of the present invention to overcome the shortcoming of the Handwritten Digit Recognition of traditional centralized tMFA, achieve Distributed identification and there is good performance.

The scope of request protection of the present invention is not limited only to the description of this embodiment.

Claims (2)

1., based on the distributed Handwritten Digit Recognition method that t hybrid cytokine is analyzed, it is characterized in that, described method comprises the steps:

Step 1: the collection of data and feature extraction: be provided with M platform computing machine, that is: node, form a network, the topology of network determines in advance, as long as it meets and to exist directly between any two nodes or multi-hop forwards and the path of intercommunication, the neighbor node set expression of node m is R _m; M node (m=1, ..., M) collect handwritten numeral 0 ~ 9 by the handwriting pad be attached thereto, the raw data of totally 10 classes, handwriting pad records the two-dimensional coordinate position of each point on each character writing track automatically, gets the coordinate of 8 points in the first-class compartment of terrain of track as the characteristic s corresponding to each raw data, totally 16 dimensions; If node m place gathers and the training dataset of the digital d obtained through feature extraction is wherein represent node m place, for n-th characteristic of handwritten numeral d of training, dimension is p=16, for the training data number of digital d;

Analyze (tMFA) with a public t hybrid cytokine and describe characteristic data set relevant to digital d in all nodes distribution; The parameter sets of tMFA is { π _i, A _i, μ _i, D _i, ν _i} _{i=1 ..., I}, wherein I is for being mixed into mark, π _ibe the weight of i-th blending constituent, A _ibe the Factor load-matrix of (p × q) of i-th blending constituent, q is the dimension of the low-dimensional factor, gets the arbitrary integer between q=p/5 ~ p/3, μ _ibe the p dimension mean value vector of i-th blending constituent, D _iit is the covariance matrix of (p × p) of the error of i-th blending constituent; ν _iit is the degree of freedom of i-th blending constituent;

Step 2: distributed training, will for distributed training, obtain the tMFA parameter sets of every class numeral corresponding to d $\begin{array}{cc}{\mathrm{\&Theta;}}^{\left(d\right)}={\{{\mathrm{\&pi;}}_i^{\left(d\right)},A_i^{\left(d\right)},{\mathrm{\&mu;}}_i^{\left(d\right)},D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\}}_{i=1,...,I}& (d=0,...,9)\end{array};$

Step 3: Distributed identification, when any one node in network collect new for identify handwritten numeral time, first obtain its character pair by above-mentioned steps 1, be expressed as s', then calculate s' about Θ ^(d)(d=0,1 ..., 9) log-likelihood logp (s'| Θ ^(d)) (d=0,1 ..., 9):

$logp\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)})=log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i^{\left(d\right)}\&CenterDot;t(s^{\&prime;}|{\mathrm{\&mu;}}_i^{\left(d\right)},A_i^{\left(d\right)}{\left(A_i^{\left(d\right)}\right)}^T+D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}));$

Using the recognition result d' of sequence number corresponding for max log likelihood value as s':

$d^{\&prime;}=\arg {\max }_{d=0}^{9}logp\left(s^{\&prime;}\right|{\mathrm{\&Theta;}}^{\left(d\right)}).$

2. a kind of distributed Handwritten Digit Recognition method analyzed based on t hybrid cytokine according to claim 1, it is characterized in that, described step 2 comprises the steps:

Step 2-1: initialization; Setting tMFA in be mixed into mark I, set the initial value of each parameter in MFA according to I, p and q ${\{{\mathrm{\&pi;}}_i^0,A_i^0,{\mathrm{\&mu;}}_i^0,D_i^0,{\mathrm{\&nu;}}_i^0\}}_{i=1,...,I};$ Wherein, each Nodes $({\mathrm{\&pi;}}_1^0,...,{\mathrm{\&pi;}}_i^0,...,{\mathrm{\&pi;}}_I^0)=(1/I,...,1/I,...,1/I);$ random selecting from the data that this node collects; with in each element generate from standardized normal distribution N (0,1); this group parameter gets the arbitrary integer between 1 ~ 5; In addition, and each node l (l=1 ..., M) data amount check that collected be broadcast to its neighbor node; After certain node m receives its data amount check of all neighbor nodes broadcast, this node calculate weight c _lm:

$c_{lm}=\frac{N_l^{\left(d\right)}}{{\mathrm{\&Sigma;}}_{l^{\&prime;}\&Element;R_m}{N}_{l^{\&prime;}}^{\left(d\right)}},$

After initialization completes, iteration count iter=1, starts iterative process;

Step 2-2: calculate local statistic and broadcast; At each node l place, based on the data that it collects first intermediate variable g is calculated _i, Ω _i, with $\begin{array}{cc}<w_{l,n,i}^{\left(d\right)}>& (n=1,...,N_l^{\left(d\right)};i=1,...,I)\end{array}:$

$g_i={\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;}^{-1}\&CenterDot;A_i^{old},$

${\mathrm{\&Omega;}}_i=I_q-g_i^T{A}_i^{old},$

${\stackrel{\&OverBar;}{u}}_{l,n,i}^{\left(d\right)}=g_i^T(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old}),$

$W_{l,n,i}^{\left(d\right)}={(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})}^T\&CenterDot;\&lsqb;A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old}\&rsqb;\&CenterDot;(s_{l,n}^{\left(d\right)}-{\mathrm{\&mu;}}_i^{old})$

$<z_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&pi;}}_i^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_i^{old},A_i^{old}{\left(A_i^{old}\right)}^T+D_i^{old},{\mathrm{\&nu;}}_i^{old})}{{\mathrm{\&Sigma;}}_{i^{\&prime;}=1}^I{\mathrm{\&pi;}}_{i^{\&prime;}}^{old}\&CenterDot;t\left(s_{l,n}^{\left(d\right)}\right|{\mathrm{\&mu;}}_{i^{\&prime;}}^{old},A_{i^{\&prime;}}^{old}{\left(A_{i^{\&prime;}}^{old}\right)}^T+D_{i^{\&prime;}}^{old},{\mathrm{\&nu;}}_{i^{\&prime;}}^{old})},$

$<w_{l,n,i}^{\left(d\right)}>=\frac{{\mathrm{\&nu;}}_i^{old}+p}{{\mathrm{\&nu;}}_i^{old}+W_{l,n,i}^{\left(d\right)}},$

Wherein for last iteration complete after the parameter value that obtains, that is: first iteration time be the initial value of parameter represent node l place n-th data belong to i-th class, that is: the probability of blending constituent, for the hidden variable in tMFA expectation value;

Had the result of calculation of above-mentioned intermediate variable, node calculate goes out local statistic, i.e. LS, ${\mathrm{LS}}_l={\{{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;\}}_{i=1,...,I},$ Comprise:

${\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>,$

${\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)},$

${\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;<w_{l,n,i}^{\left(d\right)}>\&CenterDot;s_{l,n}^{\left(d\right)}\&CenterDot;{\left(s_{l,n}^{\left(d\right)}\right)}^T,$

${\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{n=1}{\overset{N_l}{\&Sigma;}}<z_{l,n,i}^{\left(d\right)}>\&CenterDot;\log <w_{l,n,i}^{\left(d\right)}>,$

Finally, the local statistic LS that will calculate of each node l _lbroadcast diffusion is to its neighbor node;

Step 2-3: calculate associating statistic; When node m (m=1 ..., M) receive from its all neighbor node l (l ∈ R _m) LS _lafter, node m calculates associating statistic ${\mathrm{CS}}_m={\{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;,{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;\}}_{i=1,...,I}:$

${\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(1\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(2\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(3\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(4\right)}\&lsqb;i\&rsqb;,$

${\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;=\underset{l\&Element;R_m}{\&Sigma;}{c}_{lm}\&CenterDot;{\mathrm{LS}}_l^{\left(5\right)}\&lsqb;i\&rsqb;;$

Step 2-4: each parameter in estimation model; Node m (m=1 ..., M) CS that calculates according to previous step _m, estimate Θ={ π _i, A _i, μ _i, D _i, ν _i} _{i=1 ..., I}, wherein, { π _i, μ _i} _{i=1 ..., I}estimation procedure comprise:

${\mathrm{\&pi;}}_i=\frac{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}{{\mathrm{\&Sigma;}}_{i^{\&prime;}=1}^I{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i^{\&prime;}\&rsqb;},$

${\mathrm{\&mu;}}_i=\frac{{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;};$

For { A _i, D _i} _{i=1 ..., I}estimation, process comprises:

$V_i=\frac{{\mathrm{CS}}_m^{\left(4\right)}\&lsqb;i\&rsqb;-2{\mathrm{CS}}_m^{\left(3\right)}\&lsqb;i\&rsqb;\&CenterDot;{\mathrm{\&mu;}}_i^T+{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;\&CenterDot;\mathrm{\&mu;}\&CenterDot;{\mathrm{\&mu;}}_i^T}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;},$

$A_i=V_i{g}_i{(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)}^{-1},$

$D_i=diag\{V_i-A_i(g_i^T{V}_i{g}_i+{\mathrm{\&Omega;}}_i)A_i^T\};$

In addition, for { ν _i} _{i=1 ..., I}, obtained by solution equation below, comprising:

$\log \left(\frac{{\mathrm{\&nu;}}_i}{2}\right)-\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i}{2}\right)+1-\frac{{\mathrm{CS}}_m^{\left(5\right)}\&lsqb;i\&rsqb;-{\mathrm{CS}}_m^{\left(2\right)}\&lsqb;i\&rsqb;}{{\mathrm{CS}}_m^{\left(1\right)}\&lsqb;i\&rsqb;}-\log \left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)+\mathrm{\&psi;}\left(\frac{{\mathrm{\&nu;}}_i^{old}+p}{2}\right)=0,$

Wherein ψ () the digamma function that is standard, adopts Newton method when specifically solving;

Step 2-5: judge whether convergence; Node m (m=1 ..., M) calculate log-likelihood under current iteration:

$logp\left(S_m^{\left(d\right)}\right|\mathrm{\&Theta;})=\underset{n=1}{\overset{N_m}{\&Sigma;}}log(\underset{i=1}{\overset{I}{\&Sigma;}}{\mathrm{\&pi;}}_i\&CenterDot;t(s_{m,n}^{\left(d\right)}|{\mathrm{\&mu;}}_i,A_i{A}_i^T+D_i,{\mathrm{\&nu;}}_i)),$

If then algorithm convergence, stops iteration; Otherwise perform step 2-2, start next iteration (iter=iter+1; ); Wherein Θ represents the parameter value that current iteration estimates, Θ ^oldrepresent the parameter value estimated in last iteration; That is, the log-likelihood of adjacent twice iteration is less than threshold epsilon, algorithm convergence; ε gets 10 ^-5~ 10 ^-6in arbitrary value; Because node each in network is parallel data processing, therefore allly can not to restrain in an iteration simultaneously; When node l restrains, node m not yet restrains, then node l no longer sends LS _l, also no longer receive the information of neighbor node transmission; Node m is then with the LS that the node l received for the last time sends _lupgrade its CS _m; The node of not restraining continues iteration, until all nodes are all restrained in network;

After above-mentioned steps 2-1 ~ step 2-5, the corresponding model tMFA obtained by the training data of handwritten numeral d, that is: represent with parameter Θ during training convergence; Repeat above-mentioned steps 10 times, thus obtain 10 numerals tMFA model corresponding separately, in order to represent convenient, and being distinguished, using $\begin{array}{cc}{\mathrm{\&Theta;}}^{\left(d\right)}={\{{\mathrm{\&pi;}}_i^{\left(d\right)},A_i^{\left(d\right)},{\mathrm{\&mu;}}_i^{\left(d\right)},D_i^{\left(d\right)},{\mathrm{\&nu;}}_i^{\left(d\right)}\}}_{i=1,...,I}& (d=0,1,...,9)\end{array}$ The tMFA model that representative digit d is corresponding, so far, distributed training completes.