EP2680158A2 - Method for determining a missing value among data collected for a plurality of individuals at a plurality of times - Google Patents

Abstract

The method involves calculating a dynamic estimator of a missing value from values collected for a given individual (i) at a given instant (t) and a preceding instant. A static estimator of the missing value is calculated from the values collected for individuals at a given instant. A combined estimator of the missing value is determined from the static estimator and the dynamic estimator, where the dynamic estimator is calculated according to the length of a successive series of missing values and the location of the missing value in the series.

Description

The invention relates to the correction of the missing values in load curves, in particular in the field of power consumption data collected from electricity consuming individuals, but also in the field of consumption data for natural gas or water.

Recently, the data acquisition devices installed by the electrical service providers make it possible to communicate data relating to the consumption of the individual supplied with electrical energy. With such systems, the amount of load curves available will increase considerably for the supplier. It may then be possible to use sampling techniques to estimate the instantaneous total consumption at each step, for example half-hour, at the perimeter of a group of customers.

For example, it is known to install a distributed management system to supervise the operation of data acquisition devices installed for example at each individual or company. Each management system is a computer that governs the operation of the data acquisition devices. Each data acquisition device includes many sensors adapted to monitor the consumption of each individual or company, possibly each equipment of each individual or company. For example, the management system can be linked to individual meters for electricity, gas, water and others.

Periodically, each management system collects consumption data via the various sensors and stores this data in their storage medium. The frequency with which the data are collected is determined for example by the service provider. In general, relatively stable operation equipment consumption may be sampled less frequently, compared to equipment with large consumption variations.

The collected data can be analyzed locally by the management system or transmitted via a communication link for analysis by a central server of the service provider. The communication link may for example be of the Internet type and / or include wireless connections. The central server belongs for example to an electric power supplier and collects the electrical consumption data transmitted from a plurality of management systems.

Like any mass industrial process, data collection is susceptible to all kinds of technical hazards along the chain of measurement and feedback. The data could thus contain missing values. These missing data can be likened to a "no answer" problem in polls. A treatment of this non-response will be necessary, on the one hand to limit as much as possible the loss of precision due to the absence of certain data and on the other hand to avoid bias (ie errors systematic) on the estimation of the total due to possible differences between the global population and individuals without missing values.

Existing methods of correcting a "no answer" can be classified into two categories; on the one hand so-called "static" methods and on the other hand so-called "dynamic" methods.

The static methods, derived from the theory of the surveys, consist in treating the nonresponse instant by moment from the other individuals present at the moment considered and the auxiliary information. The missing value can then be imputed by regression, by the ratio, by the class average, by the "hot deck", accompanied by weighting methods. These are the classical methods used in survey statistics. They are described for example in the book P. Ardilly, "Survey techniques", published by Technip in 2006 . For these methods, the bias on the total estimator is zero at any given time, and the variance can be estimated under certain design conditions.

Dynamic methods, derived from time series, correct the missing value from consumptions of the same individual at other dates. The missing value can then be imputed by linear interpolation, exponential smoothing, or other. These methods provide consistent curves at the individual level. However, for these methods, the instantaneous biases on the total estimate are not zero, and there is no explicit variance estimator. The publication of RT Clemen, "Combining forecasts: A review and annotated bibliography", International Journal of Forecasting 559-583, 1989 , describes the use of several dynamic estimators but without sampling problems and does not involve static estimators.

All these methods, static and dynamic, have the disadvantage of leaving aside some of the information available (the rest of the curve of the individual for static estimators and the consumption of other individuals at the same time for dynamic methods ). In addition, it should be noted that, because of the large amount of data collected to be processed, all these methods are preferentially, if not inevitably, implemented by computer, whether at the level of each management system or at the level of the data collected. the central server of the service provider.

There is therefore a need for a method to impute missing values, that is, to replace each of them with a "credible" value calculated from the rest of the database, so as to obtain the best possible precision on the total estimate.

For this purpose, the invention proposes a method for combining one or more static estimators and one or more dynamic estimators to maximize the use of the information and thus improve the accuracy of the final estimator.

• calcul d'un estimateur dynamique ${\hat{y}}_{i,t,c}^d$
  
  de la valeur manquante à partir de valeurs collectées pour l'individu donné à des instants précédents et suivants l'instant donné ;
• calcul d'un estimateur statique ${\hat{y}}_{i,t}^s$
  
  de la valeur manquante à partir de valeurs relatives à des individus répondants à l'instant donné ;
• détermination d'un estimateur combiné ${\hat{y}}_{i,t}^{\gamma }$
  
  de la valeur manquante à partir de l'estimateur statique ${\hat{y}}_{i,t}^s$
  
  et de l'estimateur dynamique ${\hat{y}}_{i,t}^d\mathrm{.}$
  

More particularly, the invention relates to a method for determining a missing value among data collected for a plurality of individuals at a plurality of times, the missing value being associated with a given individual i not responding at a given time t the method comprising the steps of:

• calculating a dynamic estimator ${\widehat{\mathrm{there}}}_{i,t,\mathrm{vs}}^d$
  
  the missing value from values collected for the given individual at previous times and following the given moment;
• calculation of a static estimator ${\widehat{\mathrm{there}}}_{i,t}^s$
  
  the missing value from values relating to individuals responding at the given moment;
• determination of a combined estimator ${\widehat{\mathrm{there}}}_{i,t}^{\gamma }$
  
  the missing value from the static estimator ${\widehat{\mathrm{there}}}_{i,t}^s$
  
  and the dynamic estimator ${\widehat{\mathrm{there}}}_{i,t}^d\mathrm{.}$
  

The method according to the invention thus advantageously makes it possible to estimate at least one missing value among the values collected and thus to obtain a set of values comprising both the values collected and at least one estimated value, thus a set comprising more than one value. 'information to exploit only the starting set. The method according to the invention finds an application to the specific problematic of time series type data collected by sampling. The method of the invention has a better accuracy compared to existing methods since more information is used simultaneously. In particular, the method according to the invention does not introduce bias, unlike known dynamic methods.

est calculé pour une configuration donnée de la valeur manquante, la configuration étant fonction de la longueur de la série de valeurs manquantes successives et de l'emplacement de la valeur manquante dans cette série ; et/ou l'estimateur statique ${\hat{y}}_{i,t}^s$

est calculé pour une classe d'imputation donnée à laquelle appartient l'individu donné non répondant à l'instant donné.According to one embodiment, the dynamic estimator ${\widehat{\mathrm{there}}}_{i,t,\mathrm{vs}}^d$

is calculated for a given configuration of the missing value, the configuration being a function of the length of the series of successive missing values and the location of the missing value in this series; and / or the static estimator ${\widehat{\mathrm{there}}}_{i,t}^s$

is calculated for a given imputation class to which the given individual who is not responding at the given moment belongs.

consiste en une combinaison linéaire des estimateurs statique ${\hat{y}}_{i,t}^s$

et dynamique ${\hat{y}}_{i,t}^d\mathrm{.}$

The method according to the invention adapts according to the relative performance of the estimators at each moment and for each type of missing data: the static methods are better if there are few missing individuals at a given date, whereas dynamic methods are better for small holes, or if the missing value is at the edge of the hole. The method according to the invention makes it possible to adapt the combination to the situation, giving more weight to the dynamic estimator for small holes than for large holes, for example. According to a first mode of implementation, the determination of the combined estimator ${\widehat{\mathrm{there}}}_{i,t}^{\gamma }$

consists of a linear combination of static estimators ${\widehat{\mathrm{there}}}_{i,t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{i,t}^d\mathrm{.}$

• pour chaque individu répondant à l'instant donné, calculs d'un estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d$
  
  et d'un estimateur statique ${\hat{y}}_{\mathit{iʹ},t}^s,$
  
  et détermination d'un estimateur combiné ${\hat{y}}_{\mathit{iʹ},t}^{\gamma }$
  
  comme la combinaison linéaire desdits estimateurs statique et dynamique : ${\hat{y}}_{\mathit{iʹ},t}^{\gamma }=\mathrm{a}+s{\hat{y}}_{\mathit{iʹ},t}^s+d{\hat{y}}_{\mathit{iʹ},t}^d;$
  
• détermination de coefficients de la combinaison linéaire à partir des individus répondants à l'instant donné, par régression linéaire des valeurs collectées y_i',t à l'instant donné sur lesdits estimateurs statique ${\hat{y}}_{\mathit{iʹ},t}^s$
  
  et dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d;$
  
• détermination de l'estimateur combiné ${\hat{y}}_{\mathit{i},t}^{\gamma }$
  
  pour l'individu donné non répondant à l'instant donné en appliquant lesdits coefficients à la combinaison linéaire des estimateurs statique ${\hat{y}}_{\mathit{i},t}^s$
  
  et dynamique ${\hat{y}}_{\mathit{i},t}^d\mathrm{.}$
  

For example, the method may include the steps of:

• for each individual responding at the given moment, calculations of a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$
  
  and a static estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s,$
  
  and determining a combined estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }$
  
  as the linear combination of said static and dynamic estimators: ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }=\mathrm{at}+s{\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s+d{\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d;$
  
• determining coefficients of the linear combination from the responding individuals at the given instant, by linear regression of the collected values y _{i ', t} at the given instant on said static estimators ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s$
  
  and dynamic ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d;$
  
• determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$
  
  for the given individual not responding at the given moment by applying said coefficients to the linear combination of the static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$
  
  and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d\mathrm{.}$
  

Missing values of several nonresponding individuals can be calculated simultaneously. For each responding individual, dynamic estimators are then calculated for each missing value configuration; then, for each configuration, the coefficients of the linear combination corresponding to the missing value configuration considered are determined.

• pour chaque individu répondant à l'instant donné, calculs d'un estimateur dynamique ${\hat{y}}_{\mathit{i},t,c}^d$
  
  et d'un estimateur statique ${\hat{y}}_{\mathit{iʹ},t}^s,$
  
  et calculs d'une perte dudit estimateur statique et d'une perte dudit estimateur dynamique, chaque perte L(ŷ) étant calculée comme : $$L\left(\hat{y}\right)={\displaystyle \sum _r}\frac{\left(1-{\hat{p}}_{\mathit{iʹ}}\right)w_{\mathit{iʹ}}}{{\hat{p}}_{\mathit{iʹ}}}{E_{\mathit{iʹ}}}^2,$$
  
  
  avec p̂_i , la probabilité de réponse estimée de l'individu à l'instant donné, avec w_i, le poids de sondage de l'individu, et
  avec E_i, l'erreur entre l'estimateur ŷ_i',t et la valeur collectée y_i',t pour ledit individu à l'instant donné ;
• détermination d'un paramètre de combinaison ϕ _c comme : $${\phi }_c=\frac{L\left({{\hat{y}}_c}^d\right)}{L\left({{\hat{y}}_c}^d\right)+L\left({\hat{y}}^s\right)}$$
  
• détermination de l'estimateur combiné ${\hat{y}}_{\mathit{i},t}^{\gamma }$
  
  pour l'individu donné non répondant à l'instant donné en appliquant le paramètre de combinaison ϕ _c à la combinaison linéaire des estimateurs statique ${\hat{y}}_{\mathit{i},t}^s$
  
  et dynamique ${\hat{y}}_{\mathit{i},t,c}^d$
  
  comme suit : $${\hat{y}}_{\mathit{iʹ},t}^{\gamma }={\phi }_c{\hat{y}}_{\mathit{i},t}^s+\left(1-{\phi }_c\right){\hat{y}}_{\mathit{i},t}^d\mathrm{.}$$
  

In another example, the method may comprise the steps of:

• for each individual responding at the given moment, calculations of a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$
  
  and a static estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s,$
  
  and calculating a loss of said static estimator and a loss of said dynamic estimator, each loss L ( ŷ ) being calculated as: $$\mathrm{The}\left(\widehat{\mathrm{there}}\right)={\displaystyle \underset{r}{\Sigma }}\frac{\left(1-{\hat{p}}_{\mathit{ee\text{'}}}\right)w_{\mathit{ee\text{'}}}}{{\hat{p}}_{\mathit{ee\text{'}}}}{E_{\mathit{ee\text{'}}}}^2,$$
  
  
  with p _i , the estimated probability of response of the individual at the given moment, with w _i , the probing weight of the individual, and
  with E _i , the error between the estimator ŷ _{i ', t} and the value collected y _{i', t} for the said individual at the given moment;
• determining a combination parameter φ _c such as: $${\phi }_{\mathrm{vs}}=\frac{\mathrm{The}\left({{\widehat{\mathrm{there}}}_{\mathrm{vs}}}^d\right)}{\mathrm{The}\left({{\widehat{\mathrm{there}}}_{\mathrm{vs}}}^d\right)+\mathrm{The}\left({\widehat{\mathrm{there}}}^s\right)}$$
  
• determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$
  
  for the given individual not responding at the given time by applying the combination parameter φ _c to the linear combination of the static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$
  
  and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$
  
  as following : $${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }={\phi }_{\mathrm{vs}}{\widehat{\mathrm{there}}}_{\mathit{i},t}^s+\left(1-{\phi }_{\mathrm{vs}}\right){\widehat{\mathrm{there}}}_{\mathit{i},t}^d\mathrm{.}$$
  

Similarly, missing values of several nonresponding individuals can be calculated simultaneously. For each responding individual, dynamic estimators are then calculated for each missing value configuration; then, for each configuration, a combination parameter φ _c corresponding to the missing value configuration considered is determined.

consiste en une combinaison séquentielle des estimateurs statique ${\hat{y}}_{\mathit{i},t}^s$

et * dynamique ${\hat{y}}_{\mathit{i},t}^d\mathrm{.}$

According to a second mode of implementation, the determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

consists of a sequential combination of static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and * dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d\mathrm{.}$

• pour chaque individu répondant à l'instant donné, calcul d'un estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d;$
  
• détermination de l'estimateur combiné ${\hat{y}}_{\mathit{i},t}^{\gamma }$
  
  pour l'individu donné non répondant à l'instant donné comme l'estimateur statique ${\hat{y}}_{\mathit{i},t}^s$
  
  de la valeur manquante calculée à partir de l'estimateur dynamique ${\hat{y}}_{\mathit{i},t}^d$
  
  de l'individu non répondant et d'une relation estimée entre estimateurs dynamiques ${\hat{y}}_{\mathit{iʹ},t,c}^d$
  
  et valeurs collectées y_i',t sur les individus répondants à l'instant donné.

For example, the method may include the steps of:

• for each individual responding at the given moment, calculating a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d;$
  
• determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$
  
  for the given individual not responding at the given moment as the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$
  
  value missing calculated from the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$
  
  the nonrespondent individual and an estimated relationship between dynamic estimators ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$
  
  and values collected y _{i ', t} on the individuals responding at the given moment.

Similarly, missing values of several nonresponding individuals can be calculated simultaneously. For each responding individual, dynamic estimators are then calculated for each missing value configuration; then, for each configuration, the combined estimator is determined from the dynamic estimator and an estimated relationship for each missing value configuration considered.

• pour chaque individu répondant à l'instant donné, calcul d'un estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d;$
  
• comparaison de l'estimateur dynamique ${\hat{y}}_{\mathit{i},t}^d$
  
  de l'individu donné non répondant à l'instant donné avec chaque estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d$
  
  des individus répondants à l'instant donné, et détermination d'un plus proche voisin comme étant l'individu répondant à l'instant donné ayant estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t,c}^d$
  
  le plus proche de l'estimateur dynamique ${\hat{y}}_{\mathit{i},t}^d$
  
  de l'individu donné non répondant à l'instant donné ;
• détermination de l'estimateur combiné ${\hat{y}}_{\mathit{i},t}^{\gamma }$
  
  pour l'individu donné non répondant à l'instant donné comme l'estimateur statique ${\hat{y}}_{\mathit{i},t}^s$
  
  de la valeur manquante fixée à la valeur collectée y_i',t du plus proche voisin à l'instant donné.

In another example, the method may comprise the steps of:

• for each individual responding at the given moment, calculating a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d;$
  
• comparison of the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$
  
  of the given individual not responding at the given moment with each dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$
  
  respondents at the given moment, and determination of a nearest neighbor as the individual responding at the given moment with dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$
  
  closest to the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$
  
  the given individual not responding at the given moment;
• determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$
  
  for the given individual not responding at the given moment as the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$
  
  the missing value set to the collected value y _{i ', t} of the nearest neighbor at the given moment.

Similarly, missing values of several nonresponding individuals can be calculated simultaneously. For each responding individual, dynamic estimators are then calculated for each missing value configuration; then, for each configuration, we compare the dynamic estimator of the non-responding individual with each dynamic estimator of the responding individuals for each missing value configuration considered.

According to one application of the method, the data collected is electrical consumption data.

• figure 1, une illustration d'une « non réponse » d'un individu donné à un instant donné parmi un ensemble de valeurs collectées ;
• figure 2, des schémas illustrant la détermination d'un estimateur combiné selon l'invention pour une première configuration de données manquantes ;
• figure 3, des schémas illustrant la détermination d'un estimateur combiné selon l'invention pour une deuxième configuration de données manquantes.

Other features and advantages of the invention will appear on reading the detailed description below, and the appended figures which represent:

• figure 1 , an illustration of a "no answer" of a given individual at a given moment among a set of values collected;
• figure 2 diagrams illustrating the determination of a combined estimator according to the invention for a first configuration of missing data;
• figure 3 , diagrams illustrating the determination of a combined estimator according to the invention for a second configuration of missing data.

The invention provides a method of determining a missing value with a combined estimator from a static estimator and a dynamic estimator.

The figure 1 illustrates the "non response" of an individual i given at a given instant t among a set of values collected for n individuals between times 1 to T. An estimator will be determined for this missing value from static estimators, c ie, the data provided by the other respondents i at the given time t, and from dynamic estimators, ie the data collected for the given individual i to instants preceding and following the instant t given.

The present invention seeks the optimal combination of one or more static estimators and one or more dynamic estimators.

According to a first mode of implementation, so-called "additive" methods are considered. The performance of the different estimators is evaluated by recreating holes on the individuals present, and the combined estimator is defined as a linear combination of the different estimators whose coefficients depend on the performances observed on the individuals present, with possibly a constant intended to eliminate the bias induced by the use of dynamic estimators.

According to a second embodiment of so-called "sequential" methods are considered. Dynamic imputation is used as input to a static imputation method. The roles of the methods are no longer symmetrical since one estimator is used after another.

Whichever method is chosen, the search for the optimal combination is done separately for each triplet (time stamp, imputation class, configuration). In addition, when dynamic imputation is likely to give too poor results (for too incomplete curves or too long series of missing values), the static estimator will be used instead of a combined estimator.

The figures 2 and 3 illustrate such triplets: given t-time of the missing value, c-configuration of the missing value, and imputation class k of the nonresponding individual.

In the following, we will call "configuration of no answer", or more simply "configuration" of a missing value the length of the series of missing values to which it belongs as well as its position in this series. These configurations can be grouped to limit calculations. For example, a configuration may consist of holes of length 1, another of holes of length 2, another of the values "close to the edges" (first 4 missing for example) in holes of length 10 to 20, another of "middle" values for holes with lengths 10 to 20. Two missing values of the same series will therefore not necessarily belong to the same configuration: if the hole is relatively long, we will separate the missing values close to the edge of the values. missing mid-hole for which uncertainty is much greater. Taking into account the configuration of the missing value when determining the optimal combination of estimators makes it possible to favor one estimator relative to the other depending on whether the hole of missing values is large or small. For example, dynamic estimators will be less accurate on large holes than on small holes, and in the middle of the hole than on the edges, while the accuracy of static estimators does not depend on the configuration since we work separately instant by moment (On the other hand, it depends on the rate of missing values at the moment considered).

In addition, each individual of the sample considered is contained in an imputation class k. The imputation classes are homogeneous and previously constituted. For example, in an application to the collection of data relating to electric consumption load curves, a given imputation class contains individuals that are similar in terms of consumption (level, temperature sensitivity, ...) and probability of non-consumption. reply.

Four methods will now be described in detail. It is understood that these methods are not limiting. The methods described are intended to treat a partial nonresponse (individuals whose values are present and some missing) and not a total nonresponse (individuals for whom all the values are missing). For missing points resulting from an individual curve with too many missing values and / or included in a series of missing values that are too long (both cases are defined by thresholds set by the user), the combined estimator will be replaced by the static estimator.

• Les individus sont indicés par i. Ils sont N dans la population et n dans l'échantillon, i.e. les individus dont on collecte les données.
• Les horodates sont indicées par t allant de 1 à T.
• Pour un individu i, on notera w_i son poids de sondage (constant au cours du temps et égal à l'inverse de sa probabilité d'inclusion dans l'échantillon), y_it sa consommation à l'instant t, a_it son indicatrice de présence à l'instant t (qui vaut 1 ssi la donnée y_it est connue et qui vaut 0 si la donnée y_it est manquante).
• On notera ${\hat{y}}_{\mathit{it}}^d$
  
  l'estimateur dynamique pour l'individu i à l'instant t. Celui-ci peut-être fourni pour les non répondants comme pour les répondants.
  Cet estimateur est dépendant de la configuration de la série de valeurs manquantes considérée, c'est pourquoi on parle en fait de ${\hat{y}}_{\mathit{it},c}^d:$
  
  l'estimateur dynamique dans le cas d'une configuration de valeurs manquantes c. L'indice c sera parfois omis pour alléger les notations.
• On notera ${\hat{y}}_{\mathit{it}}^s$
  
  l'estimateur statique pour l'individu i à l'instant t. Celui-ci peut être fourni pour les répondants comme les non répondants.
  Cet estimateur ne dépend pas de la configuration de valeurs manquantes puisqu'il est déterminé indépendamment instant par instant. Cet estimateur statique dépend de la classe d'imputation k de l'individu non répondant, c'est-à-dire que les individus répondants de sa classe d'imputation seront considérés.
• On notera ${\hat{y}}_{\mathit{it},c}^{\gamma }$
  
  l'estimateur combiné pour l'individu i à l'instant t dans la configuration de valeurs manquantes c.
• Les classes d'imputation k seront indicées de 1 à K. Dans une optique de cohérence, elles seront les mêmes pour les prévisions statiques que pour les combinaisons. Les classes de non réponse (qui potentiellement peuvent être les mêmes) seront indicées de 1 à L.

In the following, we will use the following notations:

• Individuals are indexed by i. They are N in the population and n in the sample, ie the individuals whose data are collected.
• The timestamps are indexed by t ranging from 1 to T.
• For an individual i, it is noted w _i its design weight (constant over time and equal to the inverse of the probability of inclusion in the sample), _it is consumption at the instant t, _it has its presence indicator at time t (which is worth 1 if the data y _it is known and which is worth 0 if the data y _it is missing).
• We will note ${\widehat{\mathrm{there}}}_{\mathit{it}}^d$
  
  the dynamic estimator for the individual i at time t. This can be provided for both non-respondents and respondents.
  This estimator is dependent on the configuration of the set of missing values considered, so we are talking about ${\widehat{\mathrm{there}}}_{\mathit{it},\mathrm{vs}}^d:$
  
  the dynamic estimator in the case of a configuration of missing values c. The index c will sometimes be omitted to reduce the ratings.
• We will note ${\widehat{\mathrm{there}}}_{\mathit{it}}^s$
  
  the static estimator for the individual i at time t. This can be provided for both respondents and nonrespondents.
  This estimator does not depend on the configuration of missing values since it is determined independently instant by moment. This static estimator depends on the imputation class k of the nonrespondent individual, ie the responding individuals in its imputation class will be considered.
• We will note ${\widehat{\mathrm{there}}}_{\mathit{it},\mathrm{vs}}^{\gamma }$
  
  the combined estimator for the individual i at time t in the configuration of missing values c.
• The imputation classes k will be indexed from 1 to K. For the sake of consistency, they will be the same for static forecasts as for combinations. Nonresponse classes (which can potentially be the same) will be indexed from 1 to L.

First Embodiment: Linear Regression Combination

According to this embodiment, the combined estimator of the missing value will be a linear combination of one (or more) dynamic estimators and one (or more) static estimators.

To find the coefficients of the linear combination, one will use the coefficients of the linear regression of the true value on the values predicted by the different static and dynamic estimators, separately for each triplet (configuration of nonresponse, date, class of imputation ). This regression will include a constant, which will absorb the bias induced by the dynamic individual model at time t. The combination will be done separately for each imputation class.

The model can be the following: $$Y_i=\mathrm{at}+s{\widehat{\mathrm{there}}}_i^s+d{\widehat{\mathrm{there}}}_i^d+{\epsilon }_i$$

The model can be estimated by "Least Ordinary Squares" separately for each triplet (date, class, nonresponse configuration). The static estimator remains the same for all the configurations since it is calculated instant by moment without taking into account the rest of the curve, but the dynamic estimator changes since, according to the non-response configuration, the available data of the curve are not the same.

The combined estimator of a missing value for an individual i at time t will then be given by: $${\widehat{\mathrm{there}}}_{i,t}^{\gamma }=\mathrm{at}+s{\widehat{\mathrm{there}}}_{i,t}^s+d{\widehat{\mathrm{there}}}_{i,t}^d$$

The implementation of this method can be the following.

et au moins une estimation dynamique ${\hat{y}}_{\mathit{i},\mathit{t}}^d\mathrm{.}$

For an individual i not responding at time t, perform at least one static estimate ${\widehat{\mathrm{there}}}_{\mathit{i},\mathit{t}}^s$

and at least one dynamic estimate ${\widehat{\mathrm{there}}}_{\mathit{i},\mathit{t}}^d\mathrm{.}$

et au moins un estimateur dynamique ${\hat{y}}_{\mathit{iʹ},t}^d\mathrm{.}$

Pour les estimateurs statiques, on considère les individus i' répondants de la même classe d'imputation que l'individu i non répondant pour lequel on recherche l'estimateur combiné, et on récupère la même valeur prédite quelle que soit la configuration de non réponse. Pour les estimateurs dynamiques, on récupèrera une valeur prédite différente pour chacune des configurations de non réponse étudiées. Par exemple, on « cache » les vraies valeurs autour de l'instant t considéré pour se retrouver dans la configuration de valeur manquante étudiée, puis ensuite on estime la valeur à l'instant considéré seulement à partir des valeurs non cachées. On exploite ainsi les prévisions dynamiques obtenues en reproduisant sur les individus i' répondants de la même classe la même configuration de non réponse que l'individu non répondant i au même instant.For each individual responding at the same time t, compute at least one static estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s$

and at least one dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d\mathrm{.}$

For static estimators, individuals are considered to be respondents in the same imputation class as the nonresponding individual for whom the combined estimator is sought, and the same predicted value is retrieved regardless of the non-response configuration. . For dynamic estimators, a different predicted value will be obtained for each of the nonresponse configurations studied. For example, one "hides" the true values around the considered moment t to find oneself in the configuration of missing value studied, then one estimates the value at the instant considered only from the values not hidden. The dynamic forecasts obtained are thus exploited by reproducing on the individuals in the same class the same non-response configuration as the nonrespondent individual at the same time.

et ${\hat{y}}_{\mathit{iʹ},t}^d$

pour obtenir les coefficients a, s et d de combinaison. La régression linéaire entre des valeurs prédites et des valeurs réelles pour déterminer des coefficients de combinaison linéaire est décrite dans la publication de F.X. Diebold, "Serial correlation and the combination of forecasts", Journal of Business and Economic Statistics, 6, 105-111, 1988 .On the set of individuals i 'respondents at time t, separately for each triplet (instant, class, configuration), we regress the true value y _{i', t} at the given moment, (ie the value collected from 'individual i' responding at time t) to the values of the estimators ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s$

and ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d$

to obtain the coefficients a, s and d of combination. The linear regression between predicted and actual values to determine linear combination coefficients is described in the publication of FX Diebold, "Serial Correlation and the Combination of Forecasts," Journal of Business and Economic Statistics, 6, 105-111, 1988 .

et dynamique ${\hat{y}}_{\mathit{i},t}^d$

de chaque individu i non répondant à l'instant t.We can then assign a set of coefficients a, s, d appropriate to the static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$

of each individual i not responding at time t.

If it is not possible to find the coefficient set for a triplet (instant, class, configuration), for example if all the individuals of the same class are missing simultaneously, we use the estimated model on the duet (instant configuration). If it is still impossible, we can use the model on all instants and all classes for the configuration. If the dynamic estimator can never be calculated for a given date (for example, in the case of the first and / or last moment that can not be corrected by linear interpolation), we will simply use the static estimator.

Second Embodiment: Combination by Loss Function

According to this embodiment, the combined missing value estimator will also be a linear combination of one (or more) dynamic estimators and one (or more) static estimators.

This method is close to the previous one, since it is always a question of finding a linear combination of the static and dynamic estimators which is optimal. According to this second embodiment, however, we will choose an optimality criterion based on the performance of the estimators at the aggregated level and no longer individually because, in the context of a reconstitution of the flows, for example, the main objective will be above all to obtain the best possible estimate of the total and not the best individual prediction.

ait la meilleure performance possible au niveau collectif.For each instant t and each configuration c of no response, we will look for the combination parameter φ _{t, c} between 0 and 1 such that the combined forecast $${\widehat{\mathrm{there}}}_i^{\gamma }=\phi {\widehat{\mathrm{there}}}_i^s+\left(1-\phi \right){\widehat{\mathrm{there}}}_i^d$$

have the best possible performance at the collective level.

avec p̂_i, la probabilité de réponse estimée de l'individu à l'instant donné (t), avec w_i, le poids de sondage de l'individu, et
avec E_i, l'erreur entre l'estimateur ŷ_i',t et la valeur collectée y_i',t pour ledit individu répondant à l'instant donné.We calculate the loss L ( ŷ ) of each estimator, static and dynamic, as: $$\mathrm{The}\left(\widehat{\mathrm{there}}\right)={\displaystyle \underset{r}{\Sigma }}\frac{\left(1-{\hat{p}}_{\mathit{ee\text{'}}}\right)w_{\mathit{ee\text{'}}}}{{\hat{p}}_{\mathit{ee\text{'}}}}{E_{\mathit{ee\text{'}}}}^2,$$

with p _i , the estimated probability of response of the individual at the given instant (t), with w _i , the probing weight of the individual, and
with E _i , the error between the estimator ŷ _{i ', t} and the value collected y _{i', t} for said individual responding to the given instant.

The estimator is all the better if the loss is small, and moreover, the criterion takes into account the survey weights and the probabilities of non-response.

We then determine a combination parameter φ such as: $$\phi =\frac{\mathrm{The}\left({\widehat{\mathrm{there}}}^d\right)}{\mathrm{The}\left({\widehat{\mathrm{there}}}^d\right)+\mathrm{The}\left({\widehat{\mathrm{there}}}^s\right)}$$

This parameter expresses that the greater the loss of the dynamic estimator compared to the static estimator, the greater the weight of the static estimator.

pour l'individu donné i non répondant à l'instant donné t pourra alors être déterminé en appliquant le paramètre de combinaison ϕ à la combinaison linéaire des estimateurs statique ${\hat{y}}_{\mathit{i},t}^s$

et dynamique ${\hat{y}}_{\mathit{i},t}^d$

dans l'équation (2) définie ci-dessus : $${\hat{y}}_{\mathit{iʹ},t}^{\gamma }=\phi {\hat{y}}_{\mathit{i},t}^s+\left(1-\phi \right){\hat{y}}_{\mathit{i},t}^d\mathrm{.}$$

The combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

for the given individual i not responding at the given instant t can then be determined by applying the combination parameter φ to the linear combination of the static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$

in equation (2) defined above: $${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }=\phi {\widehat{\mathrm{there}}}_{\mathit{i},t}^s+\left(1-\phi \right){\widehat{\mathrm{there}}}_{\mathit{i},t}^d\mathrm{.}$$

The combined estimator obtained according to this embodiment will not be unbiased because the combination of estimators does not have a constant, the bias of the dynamic estimate is therefore not corrected. However, this bias will be attenuated by the linear combination, especially since, if it is important, one can think that the loss of the dynamic estimator will be important and therefore its low coefficient.

The first and second embodiments described above gave a symmetrical role to the static and dynamic estimators. The third and fourth embodiments described below provide for making the estimates sequentially rather than simultaneously.

Third Embodiment: Static Regression on Dynamic Prediction

We start from the observation that the dynamic estimator contains very rich information on the individual and provides a good forecast of consumption. The static information can then be used in addition to this dynamic estimate, allowing to refine it by taking into account the specificity of the moment and in particular to eliminate the instantaneous biases not captured by the dynamic models.

To do this, at each moment (and for each non-response configuration), we will model the link between the dynamic estimates and the real values of the individuals (that is to say, model the error made by the dynamic estimator ) to try to suppress its bias.

avec ε _i suivant un bruit blanc.We postulate a linear relation between the dynamic estimator and the true value for each individual i, by instant t, class and configuration of missing value. : $${\widehat{\mathrm{there}}}_{i,t}^{\gamma }=d{\widehat{\mathrm{there}}}_{i,t}^d+\mathrm{at}+{\epsilon }_i$$

with ε _i following a white noise.

This model could be made more complex by adding other explanatory variables than the dynamic prediction in the regression, for example the level of consumption over the previous period.

The implementation of this method can be the following.

et on collecte les vrais valeurs y_i',t . On peut alors déterminer par régression les coefficients d et a de la relation linéaire (5).For each individual responding at time t, in the same imputation class as the nonresponding individual for whom a combined estimator is sought, the dynamic estimators are computed in each missing value configuration. ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d$

and the true values y _{i ', t} are collected. The coefficients d and a of the linear relation (5) can then be determined by regression.

Then, moment by moment, the dynamic estimates of a non-responding individual are used as the input of the static estimator to determine the combined estimator according to the linear relationship (5) and coefficients determined by regression. Exception handling is the same as that described with reference to the first embodiment.

Fourth embodiment: The nearest neighbor on dynamic estimate

mais on ne cherche pas à modéliser paramétriquement cette relation, ce qui permet d'être plus robuste, en évitant les problèmes liés à de mauvaises spécifications.We now present a non-parametric variant of the third embodiment. It is always assumed that, for each missing value configuration, there is a class-by-class relationship between the true value y _i for an individual i and the dynamic estimator ${\widehat{\mathrm{there}}}_i^d$

but we are not trying to model parametrically this relationship, which allows to be more robust, avoiding problems related to poor specifications.

Instant by moment, when a consumption is missing for an individual i, we will replace it by that y _{', t} of the individual i' respondent of his class which resembles him the most on a given criterion, called nearest neighbor . Here, the proximity criterion will be the dynamic forecast for the nonresponse configuration studied.

le plus proche de l'estimateur dynamique ${\hat{y}}_{\mathit{i},t}^d$

de l'individu i non répondant considéré. L'estimateur combiné ${\hat{y}}_{\mathit{i},t}^{\gamma }$

pour l'individu i non répondant à l'instant donné t est alors l'estimateur statique de la valeur manquante fixée à la valeur collectée y_i',t du plus proche voisin à l'instant donné.We look for the individual respondent who, for the configuration c of the missing value considered, presents a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d$

closest to the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$

of the individual i nonrespondent considered. The combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

for the individual i not responding at the given instant t is then the static estimator of the missing value set to the collected value y _{i ', t} of the closest neighbor at the given moment.

The methods described above are not limiting. In particular, several static and / or dynamic estimators can be taken into account simultaneously when determining a combined estimator for a nonresponsive individual at a given time. In addition, other variables can be incorporated into the models as complements to dynamic and static estimators. The invention is particularly applicable in the context of missing data in time series collected by sampling.

Claims (10)

A method of determining a missing value among data collected for a plurality of individuals at a plurality of times, the missing value being associated with a given individual (i) not responsive to a given instant (t), the method comprising the steps of: - calculation of a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$

the missing value from values collected for the given individual (i) at preceding and following instants the given instant (t); - calculation of a static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

the missing value from values relating to individuals (i ') responding at the given moment (t); - determination of a combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

the missing value from the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d\mathrm{.}$

The method of claim 1, wherein the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$

is calculated for a given configuration (c) of the missing value, the configuration being a function of the length of the series of successive missing values and the location of the missing value in this series. Method according to one of claims 1 or 2, wherein the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

is calculated for a given imputation class (k) to which the given individual who is not responding at the given moment belongs. Method according to one of claims 1 to 3, wherein the determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\mathrm{there}}$

consists of a linear combination of static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d\mathrm{.}$

The method of claim 4, comprising the steps of: for each individual (i ') responding at the given instant (t), calculations of a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$

and a static estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s,$

and determining a combined estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }$

as the linear combination of said static and dynamic estimators: ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }=\mathrm{at}+s{\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s+d{\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d;$

determination of coefficients of the linear combination (a, s, d) from the individuals (i ') responding at the given instant (t), by linear regression of the collected values y _{i', t} at the given instant ( t) on said static estimators ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d;$

- determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

for the given individual (i) not responding at the given instant (t) by applying said coefficients (a, s, d) to the linear combination of the static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d\mathrm{.}$

The method of claim 4, comprising the steps of: for each individual (i ') responding at the given instant (t), calculations of a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$

and a static estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^s,$

and calculating a loss of said static estimator and a loss of said dynamic estimator, each loss L ( ŷ ) being calculated as: $$\mathrm{The}\left(\widehat{\mathrm{there}}\right)={\displaystyle \underset{r}{\Sigma }}\frac{\left(1-{\hat{p}}_{\mathit{ee\text{'}}}\right)w_{\mathit{ee\text{'}}}}{{\hat{p}}_{\mathit{ee\text{'}}}}{E_{\mathit{ee\text{'}}}}^2,$$

with p _i , the estimated probability of response of the individual at the given instant (t), with w _i , the probing weight of the individual, and
with E _i , the error between the estimator ŷ _{i ', t} and the value collected y _{i, t} for said individual at the given moment; determination of a combination parameter φ _c such as: $${\phi }_{\mathrm{vs}}=\frac{\mathrm{The}\left({{\widehat{\mathrm{there}}}_{\mathrm{vs}}}^d\right)}{\mathrm{The}\left({{\widehat{\mathrm{there}}}_{\mathrm{vs}}}^d\right)+\mathrm{The}\left({\widehat{\mathrm{there}}}^s\right)}$$

- determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\mathrm{there}}$

for the given individual (i) not responding at the given instant (t) by applying the combination parameter φ _c to the linear combination of static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d$

as following : $${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^{\gamma }={\phi }_{\mathrm{vs}}{\widehat{\mathrm{there}}}_{\mathit{i},t}^s+\left(1-{\phi }_{\mathrm{vs}}\right){\widehat{\mathrm{there}}}_{\mathit{i},t,\mathrm{vs}}^d\mathrm{.}$$

Method according to one of claims 1 to 3, wherein the determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\mathrm{there}}$

consists of a sequential combination of static estimators ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

and dynamic ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d\mathrm{.}$

The method of claim 7, comprising the steps of: for each individual (i ') responding at the given instant (t), calculating a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d;$

- determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

for the given individual (i) not responding at the given moment (t) as the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

the missing value calculated from the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$

the nonrespondent individual and an estimated relationship between dynamic estimators ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d$

and values collected y _{i ', t} on the individuals (i') respondents at the given moment (t). The method of claim 7, comprising the steps of: for each individual (i ') responding at the given instant (t), calculating a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$

- comparison of the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^d$

of the given individual (i) not responding at the given moment (t) with each dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$

individuals (i ') responding at the given instant (t), and determining a nearest neighbor as being the individual (i') responding at the given instant (t) having a dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t}^d$

closest to the dynamic estimator ${\widehat{\mathrm{there}}}_{\mathit{ee\text{'}},t,\mathrm{vs}}^d$

the given individual (i) not responding at the given moment (t); - determination of the combined estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^{\gamma }$

for the given individual (i) no responding at the given moment (t) as the static estimator ${\widehat{\mathrm{there}}}_{\mathit{i},t}^s$

the missing value set to the collected value y _{i ', t} of the nearest neighbor at the given instant (t). The method of any one of the preceding claims, wherein the collected data is power consumption data.

Images