DE19526954C1 - Statistical estimator computation unit - Google Patents

Abstract

The computation unit (CE) employs a number statistical estimators (Si) provided by neural networks, with individual weighting of the estimator output signals, before input to an overall estimator (GS), providing an estimated result signal (ES). The weighting of the estimator output signals is effected via individual weighting functions, obtained in dependence on training data provided during a training phase and corresponding to the reciprocal of the variance of each estimator.

Description

With artificial neural networks and other estimators based on statistical methods, it is often very useful if a complex problem of mapping a combination of input signals x ∈ ^{D is divided} into a combination of output signals y ∈ ^K into a plurality of smaller maps becomes.

It is known to split a complex calculator supported statistical estimator, e.g. B. an artificial Chen neural network, in a plurality of simpler computers based statistical estimators and the Aus output signals of the majority of simpler computer-aided sta to weight the statistical estimator and the weighted out output signals in a total estimator to a result signal accumulate (R. Jacobs et al., Adaptive Mixtures of Local Ex perts, Neural Computation, Vol. 3, Massachusetts Institute of Technology, 1991, pp. 79-87).

It is known that the weight functions with the Training of the individual estimators can be determined and thus the training data are adjusted. The familiar Expectation-Maximization procedure, called EM procedure, for Formation of the weight functions used, the whole Most of the simpler estimators all with the same trai training data.

To improve the results of the accumulation, it is known, the output signals of all modules, i.e. all individual averaging an estimate and thus verifying the total estimate improve (M. Perrone, Improving Regression Estimates: Avera  went Methods for Variance Reduction with Extensions to Gene ral Convex Measure Optimization, PhD thesis, Brown Universi ty, USA, pp. 10-21, 1993).

The fact that the concept of a statistical estimate not limited to artificial neural networks, but includes all computer-based estimators based on sta Based on the statistical methods, is known to any person skilled in the art immediately visible.

An overview of computer-based estimators based on stati based on technical methods is described in (J. Hartung et al., Stati stik, Oldenbourg Verlag, Munich, 9th edition, ISBN 3-486- 22055-1, pp. 123-142, 1993).

The method according to the invention is computer-aided for all statistical estimators can be used. More statistical Estimators described in (J. Hartung et al., Statistics, Oldenbourg Verlag, Munich, 9th edition, ISBN 3-486-22055-1, pp. 123-142, 1993) are not specified, but for which the invention The procedure is of course applicable to everyone Known specialist.

There are different ways of calculating the Va rianz of a single module, i.e. a single estimator, known (B. Efron et al., An Introduction to the Bootstrap, Chapman & Hall Verlag, New York, USA, ISBN 0-412-04231-2, p. 39-45, 1993; and R. Tibshirani, A Comparison of Some Error Estimates for Neural Network Models, Department of Preventive Medicine and Biostatistics, University of Toronto, 1994).

From (B. Efron et al., An Introduction to the Bootstrap, Chap man & Hall Verlag, New York, USA, ISBN 0-412-04231-2, pp. 124- 140, 1993) are the basics about distortions that occur the estimator, the so-called bias, and their estimation for their compensation, known.  

The well-known method of weighting the computerized statistical "smaller" estimators mainly hold the night part of a considerable inaccuracy in the accumulation in the total estimator for a result signal, the estimator result in itself.

The invention is therefore based on the problem of a method indicate the inaccuracy of an overall estimator, the is caused by weight functions, the output signals from "smaller" computer-aided statistical estimators weight and as input signals of the total estimator for ver provision, reduced.

The problem is solved by the method according to claim 1 solved.

Weight functions are used in the method depend on the reliability of the individual estimators, which depend on the training data with which the one individual "smaller" computer-aided statistical estimators were trained in a training phase. This will make the Accuracy of the total estimator increased considerably.

This is mainly achieved by the fact that the dependent weight functions of the used training data the reliability of each "smaller" computer supported statistical estimators depending on the training data with which each individual "smaller" computer-based statistical estimators are taken into account in the accumulation on the output signals of the "smaller" computer-based sta statistical estimators in a generalization phase in the Ge total estimates of the result signal for the input output signals of the "smaller" computer-aided statistical Estimator.

Further developments of the invention result from the dependent Claims.  

In the following the invention will be based on only one preferred embodiment illustrating drawings explained.

Show it

Fig. 1 is a sketch in which the arrangement is shown schematically, in the output signals of many "smaller" statistical estimators are weighted, depending on the training data with which the estimators were trained, and who accumulates in a total estimator to a result signal the;

Fig. 2 is a sketch, are weighted in which the arrangement is provided schematically illustrate the output signals of many "small" sta tistical estimator beginnings data depending on the Trai with which the estimator trained using different estimators with different Trainingsda ten trained, and accumulated into a result signal in an overall estimator;

Fig. 3 is a flowchart illustrating the sequence of the inventive method;

Fig. 4 is a sketch showing the ways in which the weight functions are formed.

Referring to Figs. 1 to 4 procedural to the invention is further illustrated ren.

In Fig. 1, an arrangement is represented by "smaller" computerized statistical estimators Si, ie computer-aided estimators that are based on statistical methods. Here, an index i, which uniquely identifies each "smaller" computational statistical estimator Si, is a natural number in the interval [1. . . n].

So it is an arrangement with any number of n "smaller" computer-aided statistical estimators Si  for an arrangement for which the method according to the invention can be applied.

Among computer-aided statistical estimators Si are in this connection z. B. artificial neural networks, or other estimators based on statistical methods the z. B. from (J. Hartung et al., Statistics, Oldenbourg Ver lag, Munich, 9th edition, ISBN 3-486-22055-1, pp. 123-142, 1993) are known to understand.

In the following, each parameter adjustment is a statistical Estimator Si called training of the estimator Si.

The computer-aided statistical estimators Si are estimates zer, who realized with the help of a computer unit CE the. For this reason, the statistical estimators are Si computer-aided. The fact that e.g. B. neural networks and also all other types of statistical estimators nothing else represent as a separate type of data processing system, every specialist is a separate type of computer unit completely common.

In a training phase 1, training data x with which the estimators are trained are applied to the individual “smaller” computer-aided statistical estimators Si (cf. FIG. 3).

Algorithms for training statistical estimators are everyone Those skilled in the art are well known and can do so without limitation to train the "smaller" computer-aided statistics cal estimators Si are used. An overview of one Selection of training algorithms is described in (J. Hertz et al., In troduction to the Theory of Neural Computation, Addison- Wesley Publishing Company, ISBN 0-201-51560-1, pp. 89-156, 1991).  

It is also provided that not all "smaller" computer-aided statistical estimators Si are trained with the same training data x, but, as shown in FIG. 2, they can also be trained with only a part txj of the training data x. An index j, which uniquely identifies each part txj of the training data x, is an arbitrary natural number in the interval [1. . . m].

The entirety of several "smaller" computer-based statistical estimator Si is referred to below as a variety VZ designated by estimators Si.

Here, the parts Txj of the training data x do not denote only individual patterns that are used to train the large number of VZ Si estimators are used, but they can also be used individual parts of one or more mu used for training include.

The parts txj of the training data x can be in the trai ning data x overlap, but can also be disjoint.

It is also envisaged that a part txj of the trai ningsdaten x for exactly one "smaller" computer-based statistical estimator Si is used.

But it is also provided that in a further embodiment a part txj of the training data x for several Use "smaller" computer-aided statistical estimators Si det or several parts txj of the training data x for a "smaller" computer-based statistical estimator Si be used.

By weight functions hi (xg) output signals AS of the plurality VZ of estimators Si phase in a generalization GP 3 weighted, when set to be recognized or to CLASSIFICA yield or to be estimated data ED, to the plurality VZ of estimators Si (see Fig. . 3).

Here, a data point xg denotes a date that is in the Generalization phase GP to the large number of VZ of estimators Si is created.

The method can also be used without restrictions on the multiplicity VZ of the estimators Si applied data ED, the by a regression due to the large number VZ of the estimators Si and the total estimator GS is approximated to the data ED should.

The generalization phase GP is one of the training phase 1 temporally following phase, in which the data ED to the multitude VZ of the individual estimators Si are created. Here you have to the weight functions hi (xg) have already been formed. The Formation of the weight functions hi (xg) is differentiated into Chen embodiments described below.

This means that weighted output signals GASi of the lot number VZ of estimators Si result from

GASi = hi (xg) _* ASi.

Here, the index i in turn designates the output signals ASi, the weight functions hi (xg) and the weights output signals GASi clearly, each of which corresponds to the i- th estimator Si are assigned.

The weighted output signals GASi are aggregated estimators GS accumulate and form a result signal ES (xg). The result signal ES (xg) can also be any vector given dimension.

The weight functions hi (xg) are in training phase 1 formed, depending on the training data x or the parts txj of the training data x, as will be described below.  

Some variants A are described below for forming 2 of the weight functions hi (xg) (cf. FIGS . 3 and 4).

The weight functions hi (xg) are formed depending on the training data x or parts txj of the training data x (cf. FIG. 3).

An advantageous formation of the weight functions hi (xg) for each estimator Si takes into account the vari anz Var (Si (xg)) B of the respective estimator Si for an in the Generalization phase GP to the large number VZ of estimators Si Date created xg.

As described in the previous, different are possible to calculate the variance Var (Si (xg)) of an estimator Si known (B. Efron et al., An Introduction to the Bootstrap, Chapman & Hall Verlag, New York, USA, ISBN 0-412-04231-2, p. 39-45, 1993; and R. Tibshirani, A Comparison of Some Error Estimates for Neural Network Models, Department of Preventive Medicine and Biostatistics, University of Toronto, 1994).

The variance Var (Si (xg)) of an estimator Si is for each due to the weight function hi (xg) by taking the Ge weighting function hi (xg) is formed from the reciprocal of the Va rianz Var (Si (xg)) of an estimator Si, so each results Weight function hi (xg) off:

This type B of forming the weight functions hi (xg) is suitable especially in the event that the individual estimators Si the large number of VZ of estimators Si in training phase 1 different parts txj of the training data x trained be the different parts txj of the training data x but one and the same input-output behavior Describe a picture.  

This type of formation means weight radio tion hi (xg) that the output signals ASi of an unsafe Estimator, that is, an estimator Si of the plurality VZ from Schät zer Si, which has a large Va at the respective data point xg rianz Var (Si (xg)) has less weight than the output signals ASi of a safer estimator Si, that is of an estimator Si, which is a at the respective data point xg has lower variance Var (Si (xg)).

The fact that the type of education described above of the weight functions hi (xg) for the case of training with different parts txj of the training data x for each Estimators Si of the plurality VZ of estimators Si advantageously is settable, but in no case means that this type of Formation of the weight functions hi (xg) not also for everyone whose cases, e.g. B. for training the large number of VZ Estimators Si can be used with the same training data x is.

Another possibility of forming the weight functions hi (xg) is to take into account the probability P (i | xg), that is the probability that the data point xg belongs to a certain estimator Si, in the weight functions hi (xg) D (see Fig. 4).

So the weight functions hi (xg) result from D:

The probability P (i | xg) can be done in different ways estimated that are known to any person skilled in the art.

An advantageous way to estimate the probability P (x | i) consists of using a Gaussian model:

Here, the term D (xg; Par ^il ) denotes a multidimensional local density distribution of a parameter set Par ^il .

Each estimator Si has o different local density distributions D (xg; Par ^il ), each of which is clearly identified by an index 1, which in.

Another parameter set P ^il describes the "strength" of the individual local density distribution D (xg; Par ^il ) at the data point xg for the estimator Si.

In a special case, the term can also be used with a multiple di dimensional Gaussian normal distribution can be realized. In the This case results for:

D (xg; Par ^il ) = G (xg; c ^il ; Σ ^il ).

Here G (xg; c ^il ; Σ ^il ) denotes a Gaussian normal distribution with a center vector c ^il and a covariance matrix Σ ^il at the data point xg.

The further parameter set P ^il and the parameter set Par ^il can be determined, for example, by the known EM algorithm (A. Dempster et al., Maximum Likelihood from in complete data via the EM algorithm, Journal of the Royal Statistical Society B, 1989 ).

Any person skilled in the art is familiar with other options for determining the further parameter set P ^il and the parameter set Par ^il and can be used without restrictions.

This type of weight formation described above functions hi (xg) is particularly suitable, but by no means limited to the case that the individual estimators Si the Many VZ of estimators Si in training phase 1 with un different parts txj of the training data x trained were, with the different parts txj of the training data x different input-output behavior of a Estimator Si.

The result signal ES (xg) results in the be in the previous cases, known to every specialist, in the fol way:

For the types described so far, the weight functions hi (xg), it is assumed that the estimators Si of are independent of each other.

In the event that the individual estimators Si of the plurality VZ of estimators Si are not independent of one another, a further possibility is suitable for forming the result signal ES (xg) C (cf. FIG. 4).

This excludes the use of those described below Procedure for determining the result signal ES (xg) independent estimators Si by no means.

For a clearer explanation of this process variant some assumptions are made, but not the General validity of the variant described below restrict, as also described below becomes.  

Two estimators S1 and S2 are provided for illustration taken, their dependence given by the following relationship is:

In the following, functions Si (xg) denote the images of the estimators Si at the data point xg, that is to say the "predictions" of the estimators Si, which are formed from the actual image fi (xg) and an independent Gaussian noise ε _i , which the respective Figures f _i (xg) with the variance Var (Si (xg)) of a single estimator Si are subject to the plurality VZ of estimators Si:

S _i (xg) = f _i (xg) + ε _i .

The functions f1 (x) and f2 (x) are estimated, i.e. approximate through:

A coupling variance Var _c indicates the strength of the coupling between the two estimators Si, that is to say the dependencies of the estimators S1 and S2.

The greater the coupling variance Var _c , the less the estimators S1 and S2 are "coupled".

If the value of the coupling variance Var _c → ∞ goes, then the two estimators S1 and S2 are "decoupled", that is to say independent of one another.

If the coupling variance Var _{c has} the value Var _c = 0, the input-output behavior of the estimators S1 and S2 is identical.

The result signal ES (xg) then results in the following Wise:

ES (xg) = P (1 | xg) · f₁ (xg) + P (2 | xg) · f₂ (xg)

The case of Ab illustrated by two estimators S1 and S2 dependency of the estimators Si on each other is generalized in the following expression:

For the general case, the following dependency becomes the Estimator Si assumed:

In the event that there is only one training data record for the plurality VZ of estimators Si, that is to say the training data x, the overall estimate can be further improved by forming the weight functions hi (xg) in the following manner E (cf. FIG. 4) :

Here, with Res (Si (xg)) is due to statistical noise residual error caused.  

The residual error Res (Si (xg)) can be estimated by using a Training errors in training phase 1, which are caused by statisti noise is caused, for example with a new one ronal network is approximated.

In previous cases it was assumed that the individual Estimators Si are not correlated with one another.

In the event that the individual estimators Si in the training phase 1 not with completely different training data x have been trained, d. that is, the parts txj of the trai ningsdaten x, with which the individual estimators Si the lot number VZ of estimators Si at least partially overlap, the individual estimators Si of the variety VZ from Schät zer Si be correlated.

A covariance matrix Ω (xg) of dimension nxn describes exi constant covariances between the individual estimators Si der Many VZ of estimators Si for the data point xg.

In this case, an n-dimensional vector of all weight functions hi (xg) results from:

hi (xg) = Ω ^-1 (xg) u

Here, a vector u denotes an n-dimensional unit vector.

In the event that the individual estimators Si are subject to the plurality VZ of estimators Si distortions, the so-called bias, the n-dimensional vector of all weight functions hi (xg) results from:

hi (xg) = (Ω (x g) + B (xg)) ^-1 · u

Here, a bias matrix B (xg) of dimension nxn denotes a matrix in which distortions bias _i (xg) of the individual estimators Si of the plurality VZ of estimators Si are entered.

Non-diagonal elements B _st (x) of the bias matrix B (x) result from:

B _st (x) = bias _s (x) · bias _t (x)

with s ≠ t.

Claims (5)

1. Method for the computer-aided combination of a plurality (VZ) of estimators (Si; i = 1... N), which are based on statistical methods, to form a result signal (ES (xg)) in a total estimator (GS),

• - in which the estimators (Si) of the plurality (VZ) of estimators (Si) are trained with training data (x) or parts (txj; j = 1... m) in a training phase (1),
• - in which a weight function (hi (xg); i = 1... n) is formed for each estimator (Si), which specifies the weight with which each estimator (Si) in each case when combined in the total estimator (GS) the result signal (ES (xg)) is taken into account, and
• - in which each weight function (hi (xg)) depends on the training data (x) or the parts (txj) of the training data (x) which depends on an individual estimator (Si) of the plurality (VZ) of estimators (Si).

2. The method of claim 1, wherein the weight function (hi (xg)) is used, which results from where the term Var (Si (xg)) denotes the variance of the estimator (Si) for a data point (xg). 3. The method of claim 1, wherein the weight function (hi (xg)) is used, which results from hi (xg) = P (i | xg), where the term P (i | xg) indicates the probability that a Data point (xg) is assigned to an estimator (Si). 4. The method according to claim 1, in which the mutually dependent estimators (Si) of the plurality (VZ) of estimators (Si) 5. The method of claim 1, wherein the weight function (hi (xg)) is used, which results from where the term Var (Si (xg)) denotes the variance of the estimator (Si) for the data point (xg) and wherein the term Res (Si (xg)) denotes a residual error of the estimate caused by statistical noise of the estimator (Si) zers (Si) called.