JP2017207878A - Missing data estimation method, missing data estimation device, and missing data estimation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the reliability of an estimated value for missing data.SOLUTION: An arithmetic unit 12 detects, from multidimensional data 20, that a first value 23 corresponding to a set of a first attribute value 21a and a second attribute value 22a is missing. The arithmetic unit 12 calculates a first reliability index 17 with regard to a first estimation method 13 that calculates a first estimated value 15, using a second value 24 that corresponds to a set of a first attribute value 21b and the second attribute value 22a. The arithmetic unit 12 calculates a second reliability index 18 with regard to a second estimation method 14 that calculates a second estimated value 16, using a third value 25 that corresponds to a set of the first attribute value 21a and a second attribute value 22b. The arithmetic unit 12 complements an estimated value that is equivalent to the first value 23 using an estimation method selected on the basis of comparison of the first reliability index 17 with the second reliability index 18.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a missing data estimation method, a missing data estimation device, and a missing data estimation program.

  Large-scale data may be collected and analyzed using an information processing system. Large-scale data may be multidimensional data in which a set of values is classified by a plurality of classification axes. For example, it is conceivable to collect and analyze statistical values relating to a plurality of regions (for example, municipalities such as municipalities and prefectures). In that case, the collected statistical values are classified according to the classification axis such as year, region, and statistical type. A value is identified by selecting one attribute value from each classification axis. For example, a statistical value of 3.72 million is specified for a set of attribute values of year = 2015, region = Yokohama city, statistical type = population.

  Here, in the collected large-scale data, values corresponding to all combinations of attribute values are not necessarily prepared, and some values may be missing. The omission occurs when it is difficult to obtain the value from a person who knows the value, or when there is no person who knows the value due to reasons such as omission of investigation or disaster. If there is a problem in the analysis if some values are missing, it is conceivable to estimate the missing values from other values.

  For example, a missing value estimation method for estimating missing values in a relational database from other numerical values has been proposed. In the proposed missing value estimation method, another column having the same numerical value type as the column to which the missing value belongs is selected from a plurality of columns (columns) included in the relationship table. Whether or not the types of numerical values are the same is determined based on the similarity of the end words of column names (column names). Next, by regression analysis, an estimation formula indicating the relationship between the column to which the missing value belongs and the other selected column is generated using the numerical values of the rows other than the row (record) to which the missing value belongs. Then, the missing value is estimated by applying the generated estimation formula to the row to which the missing value belongs.

  Further, for example, a missing data estimation method for estimating the cost of a certain software development project based on the costs of similar software development projects performed in the past has been proposed. In addition, for example, the relationship between rows and columns is learned for each of a plurality of matrix data, and a unified model including parameters common to the plurality of matrix data and error terms that differ depending on the matrix data is generated. A method for predicting missing values has been proposed.

JP-A-7-85082 JP 2005-141508 A JP 2012-194741 A

  When estimating missing values in multidimensional data, there may be multiple estimation methods. In this case, the problem is how to estimate the missing value. If the estimation method is fixed to one in advance, the reliability of the estimated value may be greatly reduced depending on the missing pattern of values in the multidimensional data. For example, the reliability of an estimated value calculated by a specific estimation method differs between the case where population data of one region is missing over a plurality of years and the case where population data of a plurality of regions are missing in one year. .

  In one aspect, an object of the present invention is to provide a missing data estimation method, a missing data estimation device, and a missing data estimation program that can improve the reliability of an estimated value.

  In one aspect, a missing data estimation method executed by a computer has been proposed. In the missing data estimation method, a set of values is classified using a plurality of classification axes including a first classification axis based on a plurality of first attribute values and a second classification axis based on a plurality of second attribute values. It is detected from the multidimensional data that a first value corresponding to a set of one first attribute value and one second attribute value is missing. Regarding a first estimation method for calculating a first estimated value corresponding to the first value using a second value corresponding to a set of another first attribute value and one second attribute value, A first confidence index indicating the reliability of the first estimation method is calculated from the multidimensional data. Regarding a second estimation method for calculating a second estimated value corresponding to the first value using a third value corresponding to a set of one first attribute value and another second attribute value, A second reliability index indicating the reliability of the second estimation method is calculated from the multidimensional data. The estimated value corresponding to the missing first value is complemented using the estimation method selected based on the comparison between the first confidence index and the second confidence index.

  Moreover, in one aspect, a missing data estimation device having a storage unit and a calculation unit is provided. In one aspect, a missing data estimation program is provided.

  In one aspect, the reliability of estimated values for missing data is improved.

It is a figure which shows the example of the missing data estimation apparatus of 1st Embodiment. It is a figure which shows an example of the hardware of the information processing apparatus of 2nd Embodiment. It is a figure which shows an example of the real number data of 2nd Embodiment. It is a figure which shows an example of the function which the information processing apparatus of 2nd Embodiment has. It is a figure which shows an example (time series analysis) of the extraction data of 2nd Embodiment. It is a figure which shows an example (time series analysis) of the correlation type | formula of 2nd Embodiment. It is a figure which shows an example (area correlation analysis) of the extraction data of 2nd Embodiment. It is a figure which shows an example (area correlation analysis) of the correlation type | formula of 2nd Embodiment. It is a figure which shows an example (item correlation analysis) of the extraction data of 2nd Embodiment. It is a figure which shows an example (item correlation analysis) of the correlation type | formula of 2nd Embodiment. It is a figure which shows an example of the process data of 2nd Embodiment. It is a flowchart which shows an example of the process sequence of 2nd Embodiment.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
A first embodiment will be described.

FIG. 1 is a diagram illustrating an example of a missing data estimation apparatus according to the first embodiment.
The missing data estimation device 10 includes a storage unit 11 and a calculation unit 12.
The storage unit 11 stores multidimensional data 20. The storage unit 11 may be a volatile semiconductor memory such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory.

  The calculation unit 12 processes the multidimensional data 20 stored in the storage unit 11. The arithmetic unit 12 may be a processor such as a CPU (Central Processing Unit) or a DSP (Digital Signal Processor). The arithmetic unit 12 may include an electronic circuit for a specific purpose such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). The processor executes a program stored in a memory such as a RAM (or the storage unit 11). The program executed by the processor includes a missing data estimation program in which processing described below is described. A set of processors may be referred to as “multiprocessor” or simply “processor”.

  The multidimensional data 20 is data in which a set of values is classified using a plurality of classification axes. The plurality of classification axes includes a first classification axis 21 and a second classification axis 22. For the first classification axis 21, a plurality of first attribute values including the first attribute values 21a and 21b are used. Each value included in the set of values is associated with any one of a plurality of first attribute values. A plurality of second attribute values including second attribute values 22a and 22b are used for the second classification axis 22. Each value included in the set of values is associated with any one of a plurality of second attribute values.

  The multidimensional data 20 may be regional statistical data obtained by collecting statistical values for different time points, different regions, and different statistical types. The first classification axis 21 may be any one of a time point axis, a region axis, and a statistics type axis. The plurality of first attribute values may be any one of a plurality of time points, a plurality of regions, and a plurality of statistical types. The second classification axis 22 may be another one of the time axis, the regional axis, and the statistics type axis. The plurality of second attribute values may be another one of a plurality of time points, a plurality of regions, and a plurality of statistical types.

  The computing unit 12 detects from the multidimensional data 20 that the first value 23 is missing. The first value 23 is associated with the first attribute value 21 a for the first classification axis 21, and the second attribute value 22 a is associated with the second classification axis 22. In the multidimensional data 20, there may not be a value associated with the set of the first attribute value 21a and the second attribute value 22a. Further, a NULL value indicating that there is no correct value may be associated with the set of the first attribute value 21a and the second attribute value 22a.

  The computing unit 12 selects one of the estimation methods from among a plurality of estimation methods. The calculation unit 12 supplements the multidimensional data 20 with an estimated value corresponding to the missing first value 23 using the selected estimation method. At this time, the computing unit 12 calculates a reliability index for each of the plurality of estimation methods using the multidimensional data 20, and evaluates the plurality of estimation methods. For example, the calculation unit 12 compares the reliability indexes of a plurality of estimation methods, and selects the estimation method with the highest reliability index. The plurality of estimation methods include a first estimation method 13 and a second estimation method 14.

  The first estimation method 13 is a method for calculating the first estimated value 15 using the second value 24 included in the multidimensional data 20. The second value 24 is associated with the first attribute value 21 b different from the first value 23 for the first classification axis 21, and the second value 24 is the same as the first value 23 for the second classification axis 22. The attribute value 22a is associated. The calculation unit 12 calculates a first reliability index 17 indicating the reliability of the first estimation method 13 using the multidimensional data 20.

  The first estimation method 13 generates a first estimation formula by simple regression analysis using the first attribute value 21b and the second value 24, and applies the first attribute value 21a to the first estimation formula. Then, a method of calculating the first estimated value 15 may be used. The first reliability index 17 may be an index indicating the reliability of regression analysis of the first estimation formula, and may be a correlation coefficient, a determination coefficient, or the like.

  The second estimation method 14 is a method for calculating the second estimated value 16 using the third value 25 included in the multidimensional data 20. The third value 25 is associated with the same first attribute value 21 a as the first value 23 for the first classification axis 21, and the second value different from the first value 23 for the second classification axis 22. The attribute value 22b is associated. The calculation unit 12 calculates a second reliability index 18 indicating the reliability of the second estimation method 14 using the multidimensional data 20.

  The second estimation method 14 generates a second estimation formula by multiple regression analysis using the second value 24 and the fourth value (not shown), and adds the third value 25 to the second estimation formula. May be applied to calculate the second estimated value 16. The fourth attribute is associated with the first attribute value 21b that is the same as the second value 24 for the first classification axis 21, and the second attribute that is different from the second value 24 for the second classification axis 22. The value 22b is associated. The second confidence index 18 may be an index indicating the reliability of the regression analysis of the second estimation formula, and may be a correlation coefficient, a determination coefficient, or the like.

  If there are three or more classification axes in the multidimensional data 20, the first value 23, the second value 24, and the third value 25 have the same third attribute value for the third classification axis. It may be associated. When there are three or more classification axes, the calculation unit 12 may further calculate a third confidence index for the third estimation method and compare it with the first confidence index 17 and the second confidence index 18. . The third estimation method may be a method of calculating a third estimated value using a fifth value (not shown) included in the multidimensional data 20. The fifth attribute is associated with the first attribute value 21a that is the same as the first value 23 for the first classification axis 21, and the second attribute that is the same as the first value 23 for the second classification axis 22. The value 22a is associated. However, the fifth value is associated with a third attribute value different from the first value 23 for the third classification axis.

  According to the missing data estimation device 10 of the first embodiment, it is detected from the multidimensional data 20 that the first value 23 is missing. Then, the first confidence index 17 is calculated by applying the multidimensional data 20 to the first estimation method 13 for calculating the first estimated value 15 corresponding to the missing first value 23. Further, the second reliability index 18 is calculated by applying the multidimensional data 20 to the second estimation method 14 for calculating the second estimated value 16 corresponding to the missing first value 23. Then, the estimated value corresponding to the first value 23 is complemented using an estimation method selected based on the comparison between the first confidence index 17 and the second confidence index 18.

  Thereby, the reliability of the estimated value complemented improves. If the estimation method to be used is fixed to one in advance, depending on the missing pattern of the values of the multidimensional data 20, the reliability of the calculated estimated value may be greatly reduced. For example, when the number of populations in one region is missing over a plurality of years, the estimation formulas may have low reliability even if an estimation formula indicating the time-series change in the number of populations in that region is generated. In addition, when the number of populations in a plurality of regions is missing in one year, the reliability of the estimation formulas may be low even if an estimation formula indicating the correlation of the population numbers between regions is generated. On the other hand, by evaluating the reliability of a plurality of estimation methods and selecting an estimation method, an appropriate estimation method can be selected based on a missing pattern of the multidimensional data 20 or the like.

[Second Embodiment]
Next, a second embodiment will be described.
The information processing apparatus 100 according to the second embodiment collects statistical data such as population and the number of births from a plurality of regions, and analyzes and processes the statistical data so that comparison between regions is easy. As the region, for example, a basic local government (minimum unit of local government) such as a municipality is adopted. The collected statistical data is organized based on attributes such as year, region, and item. However, not all data is available, and some data may be missing. In that case, the information processing apparatus 100 estimates missing data using the collected data. This estimation method can be realized by hardware and functions of the information processing apparatus 100 described later.

[2-1. hardware]
First, the hardware of the information processing apparatus 100 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of hardware of the information processing apparatus according to the second embodiment.

  The information processing apparatus 100 includes a CPU 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a medium reader 106, and a communication interface 107. The CPU 101 is an example of the calculation unit 12 according to the first embodiment. The RAM 102 or the HDD 103 is an example of the storage unit 11 according to the first embodiment.

  The CPU 101 is a processor including an arithmetic circuit that executes program instructions. The CPU 101 loads at least a part of the program and data stored in the HDD 103 into the RAM 102 and executes the program.

  The CPU 101 may include a plurality of processor cores, the information processing apparatus 100 may include a plurality of processors, and the processes described below may be executed in parallel using a plurality of processors or processor cores. A set of processors (multiprocessor) may be called a “processor”.

  The RAM 102 is a volatile memory that temporarily stores programs executed by the CPU 101 and data used by the CPU 101 for calculation. Note that the information processing apparatus 100 may include a type of memory other than the RAM, or may include a plurality of memories.

  The HDD 103 is a nonvolatile storage device that stores software programs such as an OS (Operating System) and application software, and data. The information processing apparatus 100 may include other types of storage devices such as a flash memory and an SSD (Solid State Drive), and may include a plurality of nonvolatile storage devices.

  The image signal processing unit 104 outputs an image to the display 71 connected to the information processing apparatus 100 in accordance with a command from the CPU 101. As the display 71, a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD), a plasma display (PDP), an organic electro-luminescence (OEL) display, or the like can be used. .

The input signal processing unit 105 acquires an input signal from the input device 72 connected to the information processing apparatus 100 and outputs it to the CPU 101.
As the input device 72, a mouse, a touch panel, a touch pad, a pointing device such as a trackball, a keyboard, a remote controller, a button switch, or the like can be used. A plurality of types of input devices may be connected to the information processing apparatus 100. Note that at least one of the display 71 and the input device 72 may be formed integrally with the housing of the information processing apparatus 100.

  The medium reader 106 is a reading device that reads programs and data recorded on the recording medium 73. Examples of the recording medium 73 include a magnetic disk such as a flexible disk (FD) and an HDD, an optical disk such as a CD (Compact Disc) and a DVD (Digital Versatile Disc), a magneto-optical disk (MO), A semiconductor memory or the like can be used. For example, the medium reader 106 stores the program and data read from the recording medium 73 in the RAM 102 or the HDD 103.

  The communication interface 107 is an interface that is connected to the network 74 and communicates with other information processing apparatuses via the network 74. The communication interface 107 may be a wired communication interface connected to a communication device such as a switch via a cable, or may be a wireless communication interface connected to a base station via a wireless link.

  The hardware of the information processing apparatus 100 has been described above. Here, for convenience of explanation, an example in which one piece of hardware is used has been shown. However, a plurality of pieces of hardware connected via a communication cable or a network may be used.

[2-2. function]
Next, functions of the information processing apparatus 100 will be described. FIG. 3 is a diagram illustrating an example of real number data according to the second embodiment. The information processing apparatus 100 stores real number data 111a. The real number data 111a is a set of statistical data collected for a plurality of regions. The real number data 111a includes open data that is open to the public such as LOD (Linked Open Data). However, the real number data 111a may include unpublished data such as a result of a questionnaire conducted independently in a certain region. In the second embodiment, statistical data such as population and birth rate is used as the real number data 111a. However, the missing data estimation method described below can also be applied to other types of data. .

  The real number data 111a is three-dimensional data in which real numbers are arranged by three classification axes of year, region, and item. The attribute value of the year dimension is the number of years such as 2009, 2010, 2011, 2012, and the like. The attribute value of the regional dimension is a local government name such as the local government M1 or the local government M2. The attribute value of the item dimension is a statistical type name such as population, number of births, number of deaths, and the like. One real value can exist for a certain year, certain region, and certain item set. For example, information that the population of the municipality M1 in 2009 is 5000, the number of births of the municipality M2 in 2010 is 2100, and the number of deaths in the municipality M1 in 2011 is 80 is included in the real number data 111a.

  However, some real values may be missing from the real number data 111a due to reasons such as unexamined or uninvestigated. For example, the population of the municipality M1 in 2011 is missing in the real number data 111a. The missing real value (missing value) is expressed as, for example, a NULL value. However, a predetermined symbol indicating a missing value or a predetermined numerical value may be used. Further, when the real number data 111a includes an abnormal value that is clearly erroneous, the information processing apparatus 100 may treat the abnormal value as a missing value.

  When there is a missing value in the real number data 111a, the information processing apparatus 100 estimates a real value corresponding to the missing value from surrounding real values. At this time, there are three estimation methods. The first estimation method is a method of analyzing data in the same region and the same item as the missing value in time series (time series analysis). The second estimation method is a method (regional correlation analysis) in which data of the same item as the missing value is compared and analyzed between different regions. The third estimation method is a method (item correlation analysis) in which data in the same region as the missing value is compared and analyzed between different items. The information processing apparatus 100 employs the one with the highest estimation accuracy among the three estimation methods for each missing value.

  FIG. 4 is a diagram illustrating an example of functions of the information processing apparatus according to the second embodiment. As illustrated in FIG. 4, the information processing apparatus 100 includes a storage unit 111, a data acquisition unit 112, a time series analysis unit 113, a regional correlation analysis unit 114, an item correlation analysis unit 115, and a missing value complement unit 116. And a data processing unit 117.

  The function of the storage unit 111 can be realized by using the RAM 102, the HDD 103, the medium reader 106, the recording medium 73, and the like described above. The functions of the data acquisition unit 112 can be realized using functions such as the CPU 101 and the communication interface 107 described above. The functions of the time series analysis unit 113, the regional correlation analysis unit 114, the item correlation analysis unit 115, the missing value complementing unit 116, and the data processing unit 117 can be realized using the CPU 101 described above.

The storage unit 111 stores real number data 111a, correlation formula data 111b, and processed data 111c.
The correlation formula data 111b includes correlation formulas generated by time series analysis, regional correlation analysis, and item correlation analysis. The correlation formula generated by the time series analysis is a regression formula generated by the single regression analysis, and is a calculation formula for calculating the estimated value of the missing value from the year to which the missing value belongs. The correlation formula generated by the regional correlation analysis is a regression formula generated by the multiple regression analysis, and is a calculation formula for calculating the estimated value of the missing value from the data of other regions in the year to which the missing value belongs. The correlation formula generated by the item correlation analysis is a regression formula generated by the multiple regression analysis, and is a calculation formula for calculating the estimated value of the missing value from the data of other items in the year to which the missing value belongs. Further, the correlation formula data 111b includes a determination coefficient for each of the generated correlation formulas.

  The processed data 111c is secondary data generated based on the real number data 111a and the correlation equation data 111b. For example, the number of births / population = birth ratio is calculated from the population and the number of births in a certain region in a certain year. In generating the processed data 111c, if a real value to be used exists in the real number data 111a, the real value is used, whereas if the real value to be used is a missing value, an estimated value is used.

  The data acquisition unit 112 acquires the real number data 111 a and stores the acquired real number data 111 a in the storage unit 111. For example, the data acquisition unit 112 acquires open data published on the network as the real number data 111a. Further, for example, the data acquisition unit 112 receives provision of unique data other than open data from a computer used by a local government. If the acquired real number data 111a is not organized by year, region, or item, the data acquisition unit 112 analyzes the structure of the acquired real number data 111a and organizes it by year, region, or item. May be stored in the storage unit 111.

  The time series analysis unit 113 generates a correlation equation by time series analysis when there is data loss in the real number data 111a. For example, the time series analysis unit 113 performs single regression analysis using existing data, and calculates a coefficient and a determination coefficient for determining a regression line. Then, the time series analysis unit 113 stores the coefficient for determining the regression line and the determination coefficient in the storage unit 111 as the correlation equation data 111b. An example of time series analysis will be described later.

  When there is a missing data in the real number data 111a, the regional correlation analysis unit 114 generates a correlation formula by regional correlation analysis. For example, the regional correlation analysis unit 114 performs multiple regression analysis using existing data, and calculates a coefficient and a determination coefficient for determining a regression line. Then, the regional correlation analysis unit 114 stores the coefficient for determining the regression line and the determination coefficient in the storage unit 111 as the correlation equation data 111b. An example of regional correlation analysis will be described later.

  The item correlation analysis unit 115 generates a correlation equation by item correlation analysis when there is data loss in the real number data 111a. For example, the item correlation analysis unit 115 performs multiple regression analysis using existing data, and calculates a coefficient and a determination coefficient for determining a regression line. Then, the item correlation analysis unit 115 stores the coefficient for determining the regression line and the determination coefficient in the storage unit 111 as the correlation equation data 111b. An example of item correlation analysis will be described later.

  The missing value complementing unit 116 detects missing data in the real number data 111a when executing the process using the real number data 111a. When there is missing data, the missing value complementing unit 116 selects a correlation formula to be used for estimation of the missing value from the three correlation formulas based on the determination coefficient included in the correlation formula data 111b.

  At this time, the missing value complementing unit 116 notifies the time series analyzing unit 113, the regional correlation analyzing unit 114, and the item correlation analyzing unit 115 of information on the region or item to which the missing value belongs. In response to this notification, the time series analysis unit 113, the regional correlation analysis unit 114, and the item correlation analysis unit 115 calculate a correlation equation and a determination coefficient that can be used for estimation of missing values.

  For example, when the population value in the municipality M1 in 2011 is missing, the missing value complementing unit 116 refers to the determination coefficient regarding the population of the municipality M1 calculated by the time series analysis. Further, the missing value complementing unit 116 refers to the determination coefficient regarding the population of the local government M1 calculated by the regional correlation analysis. In addition, the missing value complementing unit 116 refers to the determination coefficient regarding the population of the local government M1 calculated by the item correlation analysis. Then, the missing value complementing unit 116 selects a correlation expression corresponding to the largest determination coefficient among the three determination coefficients.

  The missing value complementing unit 116 calculates an estimated value of the missing value using the selected correlation equation and the real number data 111a, and stores the calculated estimated value in the storage unit 111 as substitute data for the missing value. The missing value complementing unit 116 may embed the calculated estimated value in the processed data 111c. The time series analysis unit 113, the regional correlation analysis unit 114, the item correlation analysis unit 115, and the missing value complementing unit 116 execute the above-described processing for each missing value included in the real number data 111a.

  The data processing unit 117 processes the real value included in the real number data 111a to generate the processed data 111c. For example, the data processing unit 117 calculates a processing value useful for comparison between regions from two or more real values included in the real number data 111a and a predetermined calculation formula.

  At this time, when the real value to be used is a missing value, the data processing unit 117 calculates the processed value using the estimated value calculated by the missing value complementing unit 116. The data processing unit 117 may insert only the calculated processing value into the processing data 111c, or may insert a real value or the like used for calculation of the processing value into the processing data 111c. In the latter case, the data processing unit 117 may insert the estimated value calculated by the missing value complementing unit 116 into the processed data 111c, or insert the correlation equation used to calculate the estimated value into the processed data 111c. Also good.

  The data processing unit 117 may output the generated processed data 111c. For example, the data processing unit 117 displays the processed data 111 c on the display 71. For example, the data processing unit 117 transmits the processed data 111c to another information processing apparatus.

Next, time series analysis, regional correlation analysis, and item correlation analysis will be described.
(Time series analysis)
When calculating a correlation formula for missing values of a certain year, a certain region, and a certain item, the time series analysis unit 113 extracts data of a plurality of years for the same region and the same item as the missing value from the real number data 111a. . FIG. 5 is a diagram illustrating an example (time series analysis) of the extracted data according to the second embodiment. The extracted data 111d illustrated in FIG. 5 indicates the transition of the population in the local government M1. Data may be lost when data is collected every other year, such as in the national census, or when there is a period during which data cannot be collected due to factors such as a large-scale disaster. In the example of FIG. 5, there is a lack of data in the 2011 population of the municipality M1. Therefore, data on the population of other years of the municipality M1 is extracted as the extracted data 111d.

  The correlation formula by the time series analysis is generated using the extracted data 111d. For example, a correlation equation as shown in FIG. 6 can be used as a correlation equation based on time series analysis. FIG. 6 is a diagram illustrating an example (time series analysis) of a correlation equation according to the second embodiment.

  For example, when the population is represented as P and the year is represented as Y, the regression line intercept (c11) and the coefficient q1 for the variable Y are determined from the extracted data 111d by simple regression analysis using P as an explanatory variable and Y as an explanatory variable. . That is, a correlation equation P = q1 × Y + c11 is generated. Similarly, when the number of births is missing, if the number of births is B, the intercept (c12) of the regression line and the coefficient q2 for the variable B are determined by using B as the explained variable. That is, a correlation equation B = q2 × Y + c12 is generated. Although the linear form is exemplified here as the correlation formula, the correlation formula may be a nonlinear formula.

  The time series analysis unit 113 may adjust the correlation formula through dialogue with the user. For example, the time series analysis unit 113 generates a correlation expression of the order designated by the user and displays it on the display 71. When the user gives an instruction to change the order by looking at the generated correlation equation, the time series analysis unit 113 regenerates the correlation equation with the changed order. In addition, when the range of years used for generating the correlation formula is specified by the user, the time series analysis unit 113 limits the correlation formula to data in the specified range of years, such as excluding data that is too old. It may be generated.

When the regression analysis is performed as described above, a determination coefficient for evaluating the residual between the regression equation and the extracted data 111d is obtained. The determination coefficient corresponds to the square of the correlation coefficient R and may be expressed as R ² . For example, if each real value included in the extracted data 111d is y _i , the average of the real values is y ^* , and the estimated value corresponding to each real value calculated using the correlation equation is f _i , the determination coefficient is 1 -Sum (y _i -f _i ) ² / sum (y _i -y ^* ) ² This is the residual sum of squares divided by the sum of squares of the difference from the mean of the real value and subtracted from it. However, as an evaluation index, a correlation coefficient, a degree of freedom adjusted determination coefficient, a degree of freedom adjusted correlation coefficient, or the like may be used.

The correlation formula data 111b stores the correlation formula and the determination coefficient generated by the time series analysis in association with the analysis type = “time series”, the region, and the item. For example, in the correlation formula data 111b, the correlation formula = “q1 × Y + c11” and the determination coefficient (R ² ) = “are associated with the analysis type =“ time series ”, the region =“ M1 ”, and the item =“ population ”. 0.964 "is stored.

(Regional data)
When calculating a correlation formula for missing values of a certain year, a certain region, and a certain item, the regional correlation analysis unit 114 extracts a plurality of regions and a plurality of years of data regarding the same item as the missing value from the real number data 111a. To do. FIG. 7 is a diagram illustrating an example (regional correlation analysis) of extracted data according to the second embodiment. The extracted data 111e illustrated in FIG. 7 indicates the transition of population regarding each of the municipalities M1, M2,..., M9. In the example of FIG. 7, there is a lack of data in the 2011 population of the municipality M1. Therefore, data about a plurality of regions and a population of a plurality of years is extracted as the extracted data 111e. In the second embodiment, the basic municipality is used as a regional unit, but a prefecture, electoral district, school district, or the like may be used as a regional unit. Hereinafter, the population of the municipality Mi is expressed as mi.

  The correlation formula based on the regional correlation analysis is generated using the extracted data 111e. For example, a correlation formula as shown in FIG. 8 can be used as a correlation formula based on regional correlation analysis. FIG. 8 is a diagram illustrating an example of a correlation formula (regional correlation analysis) according to the second embodiment. The correlation formula relating to the population generated by the regional correlation analysis is obtained by multiple regression analysis in which the population of a certain region in the same year is the explained variable and the population of the other region is the explanatory variable.

  For example, by multiple regression analysis with the population m1 of the municipality M1 as the explanatory variable and the population m2 of other municipalities M2,..., M9 as the explanatory variable, the regression line intercept (c21) and the variable m2,. , M9 are determined as coefficients r11,. That is, the correlation formula for estimating the population m1 of the municipality M1 is determined. However, the example of the correlation equation shown in FIG. 8 further includes a variable Y indicating year as an explanatory variable, and a coefficient r19 for the variable Y. That is, a correlation equation of m1 = r11 × m2 +... + R18 × m9 + r19 × Y + c21 is generated. Correlation equations can be determined in the same way for other municipalities. Note that the variable Y may not be included in the explanatory variable. Although the linear form is exemplified here as the correlation formula, the correlation formula may be a non-linear formula.

  The regional correlation analysis unit 114 may adjust the correlation formula through dialogue with the user. For example, the regional correlation analysis unit 114 generates a correlation equation of the order specified by the user and displays it on the display 71. When the user instructs to change the order by looking at the generated correlation formula, the regional correlation analysis unit 114 regenerates the correlation formula with the changed order. In addition, when the range of years used for generating the correlation formula is specified by the user, the regional correlation analysis unit 114 limits the correlation formula to the data of the specified year range, such as excluding data that is too old. It may be generated.

  In the above description, the correlation between the local governments M1 to M9 is analyzed. However, when a region range for analyzing the correlation is designated by the user, the region correlation analysis unit 114 converts the region range data into the designated region range data. The correlation equation may be generated in a limited manner. For example, the region correlation analysis unit 114 may have other regions that are adjacent to the region to which the missing value belongs, other regions that exist within a predetermined distance from the region to which the missing value belongs, and that have the same economic characteristics as the region to which the missing value belongs. The correlation may be analyzed in a limited area. Even if there is no instruction from the user, the regional correlation analysis unit 114 may limit the range of the region in which the correlation is analyzed based on the above criteria. In addition, the regional correlation analysis unit 114 may regenerate the correlation formula while changing the range of the region for which the correlation is analyzed through a dialog with the user.

When the regression analysis is performed as described above, a determination coefficient for evaluating the residual between the regression equation and the extracted data 111e is obtained. In the correlation formula data 111b, the correlation type generated by the regional correlation analysis and the determination coefficient are stored in association with the analysis type = “regional correlation”, the region, and the item. For example, in the correlation formula data 111b, correlation type = “r11 × m2 +... + R18 × m9 + r19 × Y + c21” and a determination coefficient are associated with analysis type = “regional correlation”, region = “M1”, and item = “population”. (R ² ) = “0.465” is stored. However, as an evaluation index, a correlation coefficient, a degree of freedom adjusted determination coefficient, a degree of freedom adjusted correlation coefficient, or the like may be used.

(Data by item)
The item correlation analysis unit 115 extracts a plurality of items and data of a plurality of years for the same region as the missing value from the real number data 111a when calculating a correlation formula for the missing value of a certain year / region / item. To do. FIG. 9 is a diagram illustrating an example of extracted data (item correlation analysis) according to the second embodiment. The extracted data 111f illustrated in FIG. 9 shows temporal changes for each item such as population, number of births, number of deaths, and the like. In the example of FIG. 9, there is a missing data in the 2011 population of the municipality M1. Therefore, data about a plurality of years and a plurality of items of the local government M1 is extracted as the extracted data 111f.

  The correlation formula by the item correlation analysis is generated using the extracted data 111f. For example, a correlation equation as shown in FIG. 10 can be used as a correlation equation based on item correlation analysis. FIG. 10 is a diagram illustrating an example (item correlation analysis) of a correlation equation according to the second embodiment. The correlation formula relating to the municipality M1 generated by the item correlation analysis is obtained by multiple regression analysis in which a value related to a certain item is an explained variable and a value related to another item is an explanatory variable.

  For example, the regression line intercept (c31) is obtained by multiple regression analysis using the value of the population P of the municipality M1 as an explanatory variable, and other items of the municipality M1 as explanatory variables such as the number of births B of the municipality M1 and the number of deaths D of the municipality M1. ) And the coefficients s11, s12,... For the variables B, D,. That is, the correlation formula for estimating the value of the population P of the municipality M1 is determined. However, the example of the correlation equation shown in FIG. 10 further includes a variable Y indicating the year as an explanatory variable, and a coefficient s19 for the variable Y. That is, a correlation formula of P = s11 × B + s12 × D +... + S19 × Y + c31 is generated. Correlation equations can be similarly determined for other items. Note that the variable Y may not be included in the explanatory variable. Although the linear form is exemplified here as the correlation formula, the correlation formula may be a non-linear formula.

  The item correlation analysis unit 115 may adjust the correlation formula through dialogue with the user. For example, the item correlation analysis unit 115 generates a correlation equation of the order specified by the user and displays it on the display 71. When the user instructs to change the order by looking at the generated correlation formula, the item correlation analysis unit 115 regenerates the correlation formula with the changed order. In addition, when the year range used for generating the correlation formula is specified by the user, the item correlation analysis unit 115 limits the correlation formula to data in the specified year range, such as excluding data that is too old. It may be generated.

  In the above description, the correlation between the population P, the number of births B, and the number of deaths D is analyzed. When the range of items whose correlation is to be analyzed is designated by the user, the item correlation analysis unit 115 is designated. The correlation formula may be generated by limiting to the data of the item range. For example, the item correlation analysis unit 115 may analyze the correlation by limiting to items that are strongly related to the item to which the missing value belongs. The item correlation analysis unit 115 may limit the range of items for which the correlation is analyzed based on a predetermined criterion without an instruction from the user. In addition, the item correlation analysis unit 115 may regenerate the correlation equation while changing the range of the items for which the correlation is analyzed through interaction with the user.

When the regression analysis is performed as described above, a determination coefficient for evaluating the residual between the regression equation and the extracted data 111f is obtained. In the correlation formula data 111b, the correlation type generated by the item correlation analysis and the determination coefficient (R ² ) are stored in association with the analysis type = “item correlation”, the region, and the item. For example, in the correlation formula data 111b, correlation type = “s11 × B + s12 × D +... + S19 × Y + c31” and determination coefficient are associated with analysis type = “item correlation”, region = “M1”, and item = “population”. (R ² ) = “0.712” is stored. However, as an evaluation index, a correlation coefficient, a degree of freedom adjusted determination coefficient, a degree of freedom adjusted correlation coefficient, or the like may be used.

  When three correlation equations that can calculate an estimated value for a certain missing value are generated, the missing value complementing unit 116 selects the correlation equation having the largest determination coefficient from the three correlation equations, and uses the selected correlation equation. Calculate an estimate. The data processing unit 117 generates the processed data 111c using the estimated value calculated by the missing value complementing unit 116 or the estimation formula selected by the missing value complementing unit 116. FIG. 11 is a diagram illustrating an example of the machining data according to the second embodiment.

  As an example, the processed data 111c includes birth ratios of a plurality of regions in 2011. The birth ratio of 2011 in a certain region can be calculated by dividing the number of births B in 2011 in the region by the population P in 2011 in the region. The real number data 111a includes the number of births B = “1900” of the municipality M2 in 2011 and the population P = “220,000” of the municipality M2 in 2011. Therefore, the data processing unit 117 acquires the real numbers of the birth number B and the population P from the real number data 111a. For example, the data processing unit 117 inserts the acquired real value into the processed data 111c. The data processing unit 117 calculates the birth ratio = “0.86%” from the acquired real value, and inserts the calculated birth ratio into the processed data 111c.

  The real number data 111a includes the number of births B = “8” of the local government M1 in 2011. Therefore, the data processing unit 117 acquires the real value of the birth number B from the real number data 111a. However, the real number data 111a lacks the population P of the municipality M1 in 2011. Therefore, for example, the data processing unit 117 acquires the correlation formula = “q1 × Y + c11” for estimating the population P from the missing value complementing unit 116, and inserts the correlation formula into the processed data 111c instead of the real value. Then, the data processing unit 117 calculates “8 / (q1 × Y + c11)” as an expression for obtaining the birth ratio, and inserts the calculated expression into the processed data 111c. Regarding the processing value expressed using the correlation equation, the data processing unit 117 may calculate a specific value at an arbitrary timing such as when the processing data 111c is displayed.

  Alternatively, the data processing unit 117 acquires the estimated value population P = “4850” from the missing value complementing unit 116, and inserts the estimated value into the processed data 111c instead of the real value. Then, the data processing unit 117 calculates the birth ratio = “0.16%” from the real value of the number of births B and the estimated value of the population P, and inserts the calculated birth ratio into the processed data 111c. At this time, the data processing unit 117 may add information indicating that the birth ratio is calculated using the estimated value to the processed data 111c.

[2-3. Process flow]
Next, a flow of processing executed by the information processing apparatus 100 when using the real number data 111a will be described with reference to FIG. FIG. 12 is a flowchart illustrating an example of a processing procedure according to the second embodiment.

  (S101) The missing value complementing unit 116 selects one attribute combination ({year, region, item}) that can specify one value (real value or NULL value) included in the real number data 111a. At this time, the missing value complementing unit 116 selects one unselected group from the possible {year, region, item} pairs.

  (S102) The missing value complementing unit 116 determines whether there is a real value of the set selected in S101. If the value corresponding to the selected set is a real value, the process proceeds to S108. On the other hand, if the value corresponding to the selected set is a NULL value, the process proceeds to S103. That is, when the real value corresponding to the selected set is missing, the missing value complementing unit 116 uses the time-series analyzing unit 113, the regional correlation analyzing unit 114, and the item correlation analysis for the missing part information (selected set). The unit 115 is notified, and the process proceeds to S103.

  (S103) The time series analysis unit 113 generates a correlation equation that can estimate the missing value based on the information of the missing part notified from the missing value complement unit 116 by time series analysis. Calculate the coefficient of determination.

  For example, when the population value of the municipality M1 in 2011 is missing, the time series analysis unit 113 limits the area to “M1”, the item to “population”, and the year from the real number data 111a. Extraction data 111d that is not limited is extracted. The time series analysis unit 113 calculates a correlation formula from the extracted data 111d by single regression analysis, and stores the coefficient for the variable of the correlation formula in the correlation formula data 111b. Further, the time series analysis unit 113 stores the determination coefficient obtained in the regression analysis in the correlation formula data 111b.

As an example, the time series analysis unit 113 obtains coefficients q1 and c11 of the correlation equation as shown in FIG. 6 by linear regression analysis using “population” of the municipality M1 as an explained variable and “year” as an explanatory variable. In addition, by this linear regression analysis, the time series analysis unit 113 obtains a determination coefficient (R ² ) as shown in FIG. When the correlation equation is a nonlinear equation, if nonlinear regression analysis is performed, a set of coefficients indicating a regression curve and a determination coefficient for evaluating a residual between a real value and an estimated value are obtained.

However, if the correlation equation that can calculate the correlation value corresponding to the selected missing value has already been generated by the time series analysis, S103 is skipped.
(S104) The regional correlation analysis unit 114 generates a correlation expression that can estimate the missing value based on the information of the missing part notified from the missing value complementing unit 116 by the regional correlation analysis. Calculate the coefficient of determination.

  For example, when the population value of the municipality M1 in 2011 is missing, the regional correlation analysis unit 114 extracts the real number data 111a without limiting the area, limiting the item to “population”, and not limiting the year. Data 111e is extracted. The regional correlation analysis unit 114 calculates a correlation formula from the extracted data 111e by multiple regression analysis, and stores the coefficient for the variable of the correlation formula in the correlation formula data 111b. In addition, the regional correlation analysis unit 114 stores the determination coefficient obtained in the regression analysis in the correlation formula data 111b.

As an example, the regional correlation analysis unit 114 performs correlation as shown in FIG. 8 by linear regression analysis using “population” of the municipality M1 as an explained variable and “population” and “year” of the municipalities M2 to M9 as explanatory variables. The coefficients r11 to r19 and c21 of the equation are obtained. By this linear regression analysis, the regional correlation analysis unit 114 obtains a determination coefficient (R ² ) as shown in FIG. When the correlation equation is a nonlinear equation, if nonlinear regression analysis is performed, a set of coefficients indicating a regression curve and a determination coefficient for evaluating a residual between a real value and an estimated value are obtained.

However, if a correlation equation that can calculate a correlation value corresponding to the selected missing value has already been generated by the regional correlation analysis, S104 is skipped.
(S105) The item correlation analysis unit 115 generates a correlation expression that can estimate the missing value based on the information of the missing part notified from the missing value complementing unit 116 by the item correlation analysis, and Calculate the coefficient of determination.

  For example, when the value of the population of the municipality M1 in 2011 is missing, the item correlation analysis unit 115 extracts the real number data 111a by limiting the area to “M1”, without limiting the item, and without limiting the year. Data 111f is extracted. The item correlation analysis unit 115 calculates a correlation formula from the extracted data 111f by multiple regression analysis, and stores the coefficient for the variable of the correlation formula in the correlation formula data 111b. Further, the item correlation analysis unit 115 stores the determination coefficient obtained in the regression analysis in the correlation formula data 111b.

As an example, the item correlation analysis unit 115 performs linear regression analysis using “population” of the municipality M1 as an explanatory variable, items such as “number of births” and “deaths” of the local government M1, and “year” as explanatory variables. The coefficients s11 to s19 and c31 of the correlation equation as shown in FIG. 10 are obtained. By this linear regression analysis, the item correlation analysis unit 115 obtains a determination coefficient (R ² ) as shown in FIG. When the correlation equation is a nonlinear equation, if nonlinear regression analysis is performed, a set of coefficients indicating a regression curve and a determination coefficient for evaluating a residual between a real value and an estimated value are obtained.

However, if a correlation equation that can calculate a correlation value corresponding to the selected missing value has already been generated by the item correlation analysis, S105 is skipped.
Note that the execution order of the processes of S103, S104, and S105 may be changed.

  (S106) The missing value complementing unit 116 compares the determination coefficients calculated in the processes from S103 to S105, and specifies the maximum determination coefficient. That is, the missing value complementing unit 116 is a correlation formula that best explains the real value of the real number data 111a among the correlation formulas calculated by the time series analysis unit 113, the regional correlation analysis unit 114, and the item correlation analysis unit 115. The correlation equation corresponding to the largest coefficient of determination is selected.

  (S107) The missing value complementing unit 116 calculates an estimated value (substitute value) of the missing value using the correlation equation corresponding to the determination coefficient identified in S106, and stores it in the storage unit 111 as substitute data for the missing value. . The missing value complementing unit 116 may insert the calculated estimated value (alternative value) into the processed data 111c.

  (S108) The missing value complementing unit 116 determines whether or not all sets of {year, region, item} have been selected. When all the groups have been selected, the series of processes shown in FIG. On the other hand, if there is an unselected pair, the process proceeds to S101. That is, when alternative data is obtained for all missing values included in the real number data 111a, or when the real number data 111a does not include a missing value, the series of processing illustrated in FIG. 12 ends.

The flow of processing executed by the information processing apparatus 100 has been described above.
As described above, the information processing apparatus 100 has a mechanism for obtaining a correlation equation for estimating a missing value by three different estimation methods of time series analysis, regional correlation analysis, and item correlation analysis, and is evaluated to have high estimation accuracy. Select the correlation formula to be used. Therefore, even if there are many missing data for a specific year, region, and item, and it is difficult to obtain a highly accurate correlation formula only with a specific estimation method, the correlation formula obtained from another estimation method is used. Missing values can be estimated.

  For example, when the value of a certain item in a certain local government is missing continuously for several years, it may be difficult to generate a highly accurate correlation equation by time series analysis. In this case, a highly accurate correlation equation may be generated from regional correlation analysis or item correlation analysis. In addition, when the values of a plurality of local governments are missing for a certain item in a certain year, it may be difficult to generate a highly accurate correlation equation in the regional correlation analysis. In this case, a highly accurate correlation formula may be generated from time series analysis or item correlation analysis. In addition, when the values of a plurality of items are missing for a certain municipality in a certain year, it may be difficult to generate a highly accurate correlation equation by item correlation analysis. In this case, a highly accurate correlation equation may be generated from time series analysis or regional correlation analysis.

DESCRIPTION OF SYMBOLS 10 Missing data estimation apparatus 11 Memory | storage part 12 Operation part 13 1st estimation method 14 2nd estimation method 15 1st estimated value 16 2nd estimated value 17 1st reliability index 18 2nd reliability index 20 Multidimensional Data 21 First classification axis 21a, 21b First attribute value 22 Second classification axis 22a, 22b Second attribute value 23 First value 24 Second value 25 Third value

Claims (5)

A missing data estimation method executed by a computer,
From multidimensional data in which a set of values is classified using a plurality of classification axes including a first classification axis based on a plurality of first attribute values and a second classification axis based on a plurality of second attribute values, 1 Detecting that a first value corresponding to a set of two first attribute values and one second attribute value is missing,
A first estimation method for calculating a first estimated value corresponding to the first value by using a second value corresponding to a set of another first attribute value and the one second attribute value A first confidence index indicating the reliability of the first estimation method is calculated from the multidimensional data,
A second estimation method for calculating a second estimated value corresponding to the first value by using a third value corresponding to a set of the one first attribute value and another second attribute value A second confidence index indicating the reliability of the second estimation method is calculated from the multidimensional data,
Using an estimation method selected based on a comparison between the first confidence index and the second confidence index, supplementing the estimated value corresponding to the missing first value;
Missing data estimation method. The first estimation method generates a first estimation formula by simple regression analysis using the other first attribute value and the second value, and the first estimation formula includes the first estimation formula. A method of calculating the first estimated value by applying an attribute value of 1;
The second estimation method uses a fourth value corresponding to a set of the second value, the other first attribute value, and the other second attribute value, by a multiple regression analysis. 2 is generated, and the second estimated value is calculated by applying the third value to the second estimated expression.
The missing data estimation method according to claim 1. The plurality of first attribute values is one of a plurality of time points, a plurality of regions, and a plurality of statistical types, and the plurality of second attribute values are the plurality of time points, the plurality of regions, and Another one of the plurality of statistical types,
The missing data estimation method according to claim 1. Stores multidimensional data in which a set of values is classified using a plurality of classification axes including a first classification axis based on a plurality of first attribute values and a second classification axis based on a plurality of second attribute values. A storage unit;
Detecting from the multi-dimensional data that a first value corresponding to a set of one first attribute value and one second attribute value is missing,
A first estimation method for calculating a first estimated value corresponding to the first value by using a second value corresponding to a set of another first attribute value and the one second attribute value A first confidence index indicating the reliability of the first estimation method is calculated from the multidimensional data,
A second estimation method for calculating a second estimated value corresponding to the first value by using a third value corresponding to a set of the one first attribute value and another second attribute value A second confidence index indicating the reliability of the second estimation method is calculated from the multidimensional data,
A calculation unit that supplements an estimated value corresponding to the missing first value using an estimation method selected based on a comparison between the first confidence index and the second confidence index;
A missing data estimation device. On the computer,
From multidimensional data in which a set of values is classified using a plurality of classification axes including a first classification axis based on a plurality of first attribute values and a second classification axis based on a plurality of second attribute values, 1 Detecting that a first value corresponding to a set of two first attribute values and one second attribute value is missing,
A first estimation method for calculating a first estimated value corresponding to the first value by using a second value corresponding to a set of another first attribute value and the one second attribute value A first confidence index indicating the reliability of the first estimation method is calculated from the multidimensional data,
A second estimation method for calculating a second estimated value corresponding to the first value by using a third value corresponding to a set of the one first attribute value and another second attribute value A second confidence index indicating the reliability of the second estimation method is calculated from the multidimensional data,
Using an estimation method selected based on a comparison between the first confidence index and the second confidence index, supplementing the estimated value corresponding to the missing first value;
Missing data estimation program that executes processing.

Images