JP6075279B2 - Data analysis apparatus, method and program - Google Patents

Description

The present invention relates to a data analysis apparatus , method, and program for analyzing collected data using a statistical model that assumes latent variables.

  In recent years, a large amount of data such as social network service (SNS) usage history, Internet shopping purchase history, location history using mobile terminal location information, purchase history accumulated by using prepaid cards, etc. Has become easy to collect.

  On the other hand, many of these data are freely set by users and service providers, and when analyzing them for another purpose, not all data suitable for the purpose is necessarily prepared. For this reason, in order to perform analysis suitable for the purpose, the problem of data loss must be solved.

  One method for efficiently collecting the target data is a questionnaire. The questionnaire is effective when conducting social surveys to capture the actual situation of people's consciousness and behavior. There are two types of social surveys: statistical social surveys aimed at obtaining a large picture of society by taking a large amount of data, such as the national census, and case-based social surveys such as interviews with small groups and participation observations. Can be broadly divided.

  Among these, statistical social survey methods include the interview method, tome method, mail method, collective method, telephone method, and electronic method (what are called online surveys, web surveys, etc.). This is a method of collecting the responses of the target person to the question items (questionnaire). In particular, in the electronic law, since all people in an environment connected to the Internet can register, it is possible to collect information on people of various attributes such as age, occupation, and residence. In addition, since there are usually many companies with tens of thousands of members, it is possible to easily collect data of thousands even if the user attribute conditions are narrowed down. However, in either method, it is difficult to obtain answers from all of the questions.

  As a method of obtaining a highly accurate answer by reducing the number of questions and reducing the respondent load as much as possible, there is a selection answer that determines whether or not to answer the corresponding question according to the intention of the respondent. This is used when you do not want to answer the question being asked, or when you have multiple options for a single question. As a result, the collected response data is deficient.

So far, the missing mechanism of why the variable is missing even if the data is missing has been considered in the following three ways since Rubin's research (see Non-Patent Document 1, for example). (See Non-Patent Document 2, for example).
(1) Missing Completely At Random (MCAR)
Which values are missing is completely random.
(2) Missing At Random (MAR)
Which values are missing may depend on the data, but not on missing values.
(3) Non-random defects Not Missing at Random (NMAR)
Whether it is missing depends on the value of the missing value itself and other variables that have not been observed.
The data analysis method differs depending on which of these can be assumed.

Assuming that the loss mechanism is MAR or MCAR, the full information maximum likelihood method can be applied as a parameter estimation method for statistical models. Specific examples of the algorithm include a pseudo-Newton method and an EM algorithm that includes both latent variables and missing values in complete data (see, for example, Non-Patent Document 3).
By the way, many statistical models assume latent variables. Especially in the analysis of questionnaire data, there are models including various latent variables, such as a factor analysis model that finds hidden factors, a structural equation modeling that generalizes them, and a hidden Markov model that can model changes in time.

  The latent variable means a variable that is not observed. For example, assuming that there are test score data for five subjects: country, number, company, science, and English, a two-factor model is applied. At this time, two factors (for example, a science score and a literary score) are latent variables. This is because science scores and humanities scores cannot be observed.

In contrast, missing values or missing values mean that variables that should be observable cannot be observed. For example, it is assumed that there is an item “annual income” in the questionnaire data. Some people do not want to answer their annual income, so it is often left blank. In this way, it means that data cannot be observed for some reason even though there should actually be data.
That is, the difference between missing values and latent variables is that missing values can be observed originally, but latent variables are never observed.

First, an EM algorithm for estimating a statistical model including latent variables using missing data will be described. Assume that the p-dimensional observation variable is X = (X1,..., Xp) ^T and the m-dimensional latent variable vector is F = (F1,..., Fm) ^T. Now, it is assumed that the q-dimensional parameter θ = (θ1,..., Θq) ^T is estimated by the maximum likelihood method. When the observation variable is missing, if the observation data is described as X [n] and the missing value as X_ [n] for the nth observation, the complete data is Xn = [X [n], X_ [n]. ]. At this time, in the EM algorithm, it is necessary to calculate the conditional expected value E [logf (xn, fn) | x [n]] with the observation data of the complete log likelihood being given. If the data follow an independent equidistribution, the conditional expectation is

Given in. Here, f (xn, fn) is the simultaneous distribution of the observation data and the latent variable, and N is the sample size.

  However, when there is a large amount of defects, the cost of calculating this conditional expected value often increases. For example, as a technique often used in a model including a latent variable, there is a factor analysis model in which an observed variable is represented by a linear combination of latent variables. This factor analysis model equation is shown in equation (1) in FIG.

In the factor analysis model, it is assumed that a linear relationship X−μ = ΛF + ε holds between the observed variable vector and the latent variable vector (equation (2) in FIG. 16). Where μ = (μ1,..., Μp) ^T is a p-dimensional average vector, Λ = (λij) is a p × m factor loading matrix, and ε = (ε1,..., Εp) ^T is a p-dimensional unique factor vector. (Formula (3) in FIG. 16).

In contrast to the orthogonal factor model (Equation (4) in FIG. 17) assuming normality for the common factor vector and the unique factor vector, the observed variable vector follows a multivariate normal distribution, and its covariance matrix is cov [X] = ΛΛ. It is given by ^T + Ψ (equation (5) in FIG. 17). However, ψ = diag (ψ1,..., Ψp) represents the unique variance (formula (4) in FIG. 17). Now, when complete data x1,..., XN are given, it is assumed that the model is estimated by the maximum likelihood method. Then, as shown in FIG. 18, the log likelihood function is

Given in. Here, Σ = ΛΛ ^T + ψ.

  Next, let us consider the case where the observed variable is missing. FIG. 19 shows the model estimation. As shown in the equation (7) of FIG. 19, the data is divided into observed values and missing values, and the expected values and covariance matrices of the observed variables are E (X [n]) = μ [n], Cov (X [ n]) = Σ [n], the likelihood under the MAR is as shown in FIG.

Can be described.

At this time, the log likelihood function is as shown in FIG.
It is expressed.

In the past, we have addressed the problem of factor analysis for several hundred samples with 20-30% deficiencies. To solve the missing value problem of factor analysis under that condition, the following method has been used as an algorithm for calculating the maximum likelihood estimate when MAR is assumed.
(A) A method of calculating a complete information maximum likelihood estimated value by a pseudo-Newton method in which the likelihood of an observation variable is accumulated for each observation and a parameter that maximizes the value is obtained.
(B) A method using an EM algorithm in which both common factors and missing observation variables are included in complete data (for example, see Non-Patent Document 4).
“Complete data” refers to data when data that is not observed, such as missing values and latent variables, is temporarily acquired.

  Of the above two methods, the Newton method of (A) is calculated by the method shown in FIG. 20, but when the number of observation variables is large, the number of parameters increases and the calculation becomes slow and unstable. It is known. For example, the calculation time when a defect is randomly generated, the defect rate is 90%, and the number of samples is 20,000 is calculated using the existing statistical tool called OpenMx. The operating frequency of the CPU is 2.3 GHz and the memory capacity However, it took more than a day using a personal computer with a 16GB general-purpose OS (Operating System). Also, it greatly depends on the initial value, and the parameter value diverges for a certain initial value.

  On the other hand, the EM algorithm of (B) includes latent variables (common factors) and missing values that cannot be directly observed as shown in FIG. 21 in complete data. At this time, the complete log likelihood function is given by equation (11) shown in FIG. Here, in order to calculate the expected value of the complete log likelihood, it is sufficient if the expression (12) of FIG. 21 can be calculated, but the calculation is significantly slowed when the number of missing data is large.

Actually, the calculation order of the matrix operation required for one iteration is O (p ² N) as shown in the equation (13) described in FIG. Here, O (X) means a constant multiple of x. Therefore, when both p and N are large, the calculation amount becomes large. For example, in order to estimate the parameters by the EM algorithm for the previous data, the calculation time of O (100 ² × 20,000) is required to calculate the conditional expected value for the posterior distribution of the formula Sxx. Is required for several hours.

  The “missing rate” represents the ratio between the number of elements actually observed when the data is expressed in a matrix and the number of elements when all the data is observed. When the defect rate is “1”, all data is missing, and when it is “0”, all data is observed.

  As described above, these methods were established in the 70s and 80s when the sample size was not so large. Like the data collected on the current web, “the sample size is huge, A statistical model analysis method has not been established for collected data with the characteristic of “a large number and a large number of defects”. Note that the upper limit of the sample size that can be said to be not so large is about 500, the lower limit of the sample size that can be said to be huge is about 5000, the lower limit that can be said to have many observation variables is about 50, The approximate lower limit for a large number of defects is about 70%.

Rubin, D. B. “Inference and Missing Data”, Biometrika, Vol. 63, No. 3, pp.581-592, 1976. Takahiro Hoshino, “Statistical Science of Survey Data—Causal Reasoning / Selection Bias / Data Fusion”, pp.27-28, Iwanami Shoten, 2009. Dempster, A. P., Laird, N. M., and Rubin, D. B., “Maximum likelihood from incomplete data via the EM algorithm”, Journal of the Royal Statistical Society, Series B (Methodological), pp. 1-38, 1977. M. Jamshidian and R. I. Jennrich, “An EM Algorithm for ML Factor Analysis with Missing Data”, Journal of the Royal Statistical Society, Series B (Methodological), Vol. 59, No. 3, pp. 569-587, 1997.

  As described above, a method for calculating the full information maximum likelihood estimate by the pseudo Newton method, which is a conventional algorithm for calculating the maximum likelihood estimate when MAR is assumed for the missing value problem of factor analysis, When the number of observation variables is large, the number of parameters increases, so that the calculation becomes slow and unstable. In addition, the method using the EM algorithm that includes both common factors and missing observation variables in the complete data includes latent variables (common factors) and missing values that cannot be observed directly in the complete data, so the number of missing data is large. There is a problem that the calculation is sometimes extremely slow.

The present invention has been made paying attention to the above circumstances, and the object thereof is to enable high-speed parameter estimation in a statistical model assuming latent variables when the loss rate is high and there are many observation variables. A data analysis apparatus , method, and program are provided.

In order to achieve the above object, the present invention comprises the following various aspects.
(1) In a statistical model that assumes latent variables when performing statistical analysis processing on observation data stored in the storage medium and means or process for receiving the observation data observed and storing it in the storage medium ; An analysis means or process for estimating parameters included in the statistical model by constructing likelihood using only the observed data and latent variables without including missing data that has not been observed ;
Said analysis means or process comprises:
The dimension of the observed variable is p, the number of factors is m, the p-dimensional observed variable vector is X = (x1, ..., xp) ^T , the p-dimensional average vector is μ = (μ1, ..., μp) ^T , and the m-dimensional latent variable The vector is F, the p × m factor loading matrix is Λ = (λij) = (λ1,..., Λp) ^T , the p-dimensional unique factor vector is ε, the sample size is N, and iobs (n) is n (= 1,. , N) In the factor analysis model represented by X−μ = ΛF + ε, where i is the subscript i of the variable xi observed in the th sample,
Accepting input of each initial value of the p-dimensional mean vector μ, the p × m factor loading matrix Λ, and Ψ = diag (ψ1,...
Using the EM algorithm, the complete log likelihood constructed based only on the observed variable xi is
Is given as a conditional expectation value E of the full log likelihood, using a matrix M representing the presence or absence of deficiency,
E [Fn | xn] = M ^-1 Λ ^T Ψ ^-1 xn,
^{E [FnFn T | xn] =} M -1 + E [Fn | xn] E [Fn | xn] T
And the matrix M = (mij) is
M = Λ ^T Ψ ^-1 Λ + Im
Calculated by
In addition, a penalty term is added to maximize the logarithmic likelihood.
After the calculation process of the conditional expected value E and the maximization process of the complete log likelihood added with the penalty term, it is determined whether the parameter has converged,
When it has been determined that it has not converged, the process of calculating the conditional expected value E and the process of maximizing the full log likelihood added with the penalty term are repeated until a preset number of iterations is reached . Is.

(2) In a statistical model that assumes latent variables when performing statistical analysis processing on observation data stored in the storage medium and means or process for receiving the observation data and storing it in the storage medium, Analyzing means or process for estimating parameters included in the statistical model by constructing likelihood using only the observed data and latent variables without including missing data that has not been observed; Threshold determination means or process for determining whether observation data equal to or greater than the threshold value is stored by comparing the number of stored observation data samples with a set threshold value, and the number of variables used to estimate the parameter By referring to a table in which the minimum number of samples necessary for estimation of the parameter is stored in advance for the combination of the required error and the defect rate. , Corresponding to the combination of the number of variables in the observation data stored in the storage medium, the loss rate obtained from the observation data stored in the storage medium and the determined missing data, and the specified required error A relationship determination unit or process for searching the number of samples and setting the number of samples as the threshold of the threshold determination unit or process, wherein the analysis unit or process is equal to or greater than a threshold by the threshold determination unit or process . If the observation data is determined to have been stored, to the observed data stored in the storage medium is obtained so as to perform a process for estimating the parameters.

  According to each aspect of the present invention, since the parameter estimation method that maximizes the conditional expected value of only the observed data and the latent variable is employed instead of maximizing the conditional expected value, it is deficient. Instead of estimating all the data iteratively in the algorithm, it is only necessary to estimate the latent variables. Therefore, it is possible to perform parameter estimation in a statistical model assuming a latent variable when the loss rate is high and there are many observed variables at high speed.

The figure which shows the structure of the whole system containing the data analyzer which concerns on 1st Embodiment of this invention. The block diagram which shows the function structure of the data analyzer which concerns on 1st Embodiment of this invention. The flowchart which shows operation | movement of a service provider terminal among a series of operation | movement from collection of data to transmission of analysis data. The flowchart which shows the transmission operation | movement of the questionnaire response data by a user terminal, and the reception operation | movement of the questionnaire response data by an analysis server among a series of operation | movement from data collection to transmission of analysis data. The flowchart which shows the procedure and content of the analysis process by an analysis server among a series of operation | movement from collection of data to transmission of analysis data. The figure which shows 0-1 matrix showing the presence or absence of a defect | deletion. The flowchart which shows the process sequence and process content by the high-speed factor analysis part of the data analyzer shown in FIG. The figure for demonstrating the effect by the data analyzer shown in FIG. The block diagram which shows the function structure of the data analyzer which concerns on 2nd Embodiment of this invention. The figure which shows an example of the data table which shows the relationship of the defect rate, the number of variables, an error, and the number of samples with which the data analysis apparatus shown in FIG. 9 is provided. The block diagram which shows the function structure of the data analyzer which concerns on 3rd Embodiment of this invention. The block diagram which shows the function structure of the data analyzer which concerns on 4th Embodiment of this invention. The block diagram which shows the function structure of the data analyzer which concerns on 5th Embodiment of this invention. The block diagram which shows the function structure of the data analyzer which concerns on 6th Embodiment of this invention. The figure for demonstrating factor analysis. The figure used for explanation of a factor analysis model. The figure for demonstrating the first half part of the estimation method of the model by a maximum likelihood method. The figure for demonstrating the second half part of the estimation method of the model by maximum likelihood method. The figure for demonstrating the estimation method of the model by a maximum likelihood method when there exists a defect | deletion. The figure for demonstrating the calculation method by a Newton method. The figure for demonstrating the existing EM algorithm.

Embodiments according to the present invention will be described below with reference to the drawings.
[First Embodiment]
In the first embodiment of the present invention, when estimating a parameter in a statistical model assuming a latent variable, the missing data not observed is not included in the complete data, and the complete likelihood is calculated using only the observed data. By configuring, the calculation amount of the matrix operation is greatly reduced to achieve high-speed calculation, and this parameter estimation method is applied to factor analysis.

(Constitution)
FIG. 1 is a schematic configuration diagram of a system including an analysis server which is a first embodiment of a data analysis apparatus according to the present invention. As shown in the figure, the system of the first embodiment is configured such that a plurality of user terminals UT1 to UTn, a service provider terminal SP, and an analysis server SVa are communicably connected via a communication network NW.

  Each of the user terminals UT1 to UTn includes a personal computer, a tablet terminal, a smart phone, or a mobile phone used by a plurality of users to be observed, and includes a web browser and a mailer. The service provider terminal SP is composed of a personal computer used by the administrator of the service provider, and includes a Web browser and a mailer as with the user terminals UT1 to UTn. The communication network includes, for example, an IP (Internet Protocol) network and an access network for accessing the IP network.

Incidentally, the analysis server SVa is composed of a server computer, and is configured as follows. FIG. 2 is a block diagram showing the functional configuration.
That is, the analysis server SVa includes a communication interface unit 1, a control unit 2a, and a storage unit 3a. The communication interface unit 1 performs data communication with the user terminals UT1 to UTn and the service provider terminal SP according to a communication protocol defined by the communication network NW.

  The storage unit 3a includes a nonvolatile memory such as an HDD (Hard Disc Drive) or SSD (Solid State Drive) that can be written and read as needed as a storage medium, and is necessary for realizing the first embodiment. As a storage unit, a collected data storage unit 31 and an analysis data storage unit 32 are provided.

  The collected data storage unit 31 is used to accumulate questionnaire response data collected from the user terminals UT1 to UTn. In addition to questionnaire response data, each user's place of stay history, purchase history, communication history, and the like may be stored. The analysis data storage unit 32 is used for storing analysis data obtained by an analysis process described later.

  The control unit 2 includes a CPU (Central Processing Unit), and as a control and processing function necessary for realizing the first embodiment, a collected data management unit 21, a sample number threshold determination unit 22, and a fast factor analysis And an analysis data display control unit 24. All of these functions are realized by causing the CPU to execute a program stored in a program memory (not shown).

  The collected data management unit 21 transmits the questionnaire data including the survey items sent from the service provider terminal SP from the communication interface unit 1 to the user terminals UT1 to UTn to be collected and displays them, and the user terminals UT1 to UT1. The questionnaire response data returned from the UTn is received via the communication interface unit 1, and the received collected data is stored in the collected data storage unit 31.

  The sample number threshold determination unit 22 performs a process of determining whether or not answer data for a preset number of people has been accumulated in the collected data storage unit 31. The fast factor analysis unit 23 reads the response data from the collected data storage unit 31 and does not include missing data that has not been observed in the complete data in the statistical model assuming a latent variable for the read response data. The factor analysis process is performed using a method of constructing the full likelihood using only the observed data. And the process which stores the data showing the analysis result in the analysis data memory | storage part 32 is performed.

  The analysis data display control unit 24 reads the analysis data from the analysis data storage unit 32 in response to a request from the service provider terminal SP, and transmits the analysis data from the communication interface unit 1 to the requesting service provider terminal SP. A process of transmitting data from the communication interface unit 1 to the user terminals UT1 to UTn set in advance as distribution destinations is performed.

(Operation)
Next, the operation of the system including the analysis server SVa configured as described above will be described. FIGS. 3 to 5 are flowcharts showing a series of operation procedures from the creation of questionnaire data to collection of the response data, analysis of the response data, and transmission of data representing the analysis result.

(1) Creation and Distribution of Questionnaire First, in the service provider terminal SP, in step S11 shown in FIG. 3, questionnaire data is created according to the input operation of the service provider administrator. As the questionnaire data, data including arbitrary or optional answer items with many variables is created. Specifically, the number of item types and choice types is set to a large value of 100 or more, and items that can be selected and selected are included. The created questionnaire data is sent to the analysis server SVa.

  When the analysis server SVa receives the questionnaire data by the communication interface unit 1, under the control of the collection data management unit 21, the analysis server SVa collects user terminals UT <b> 1 to UT <b> 1 that are stored in advance in a user information storage unit (not shown). The questionnaire data is distributed to UTn. As a delivery method, e-mail is used, for example.

(2) Collection of Answer Data for Questionnaire When a user inputs an answer to the distributed questionnaire data in the user terminals UT1 to UTn, an answer for returning the inputted answer in step S21 shown in FIG. Data is created. At this time, the user does not have to answer items that he does not want to answer. When the user performs a transmission operation, in step S22, the created answer data is transmitted from the user terminals UT1 to UTn to the analysis server SVa that is the transmission source.

  Whenever the response data is returned from the user terminals UT1 to UTn, the analysis server SVa receives the response data via the communication interface unit 1 in step S23, and the received response is received. Data is stored in the collected data storage unit 31. At this time, the collected data is stored in association with, for example, a user ID and a questionnaire ID for identifying the questionnaire.

(3) Analysis of Collected Data The analysis server SVa first sets the collected data stored in the collected data storage unit 31 to a preset sample number threshold in step S31 shown in FIG. It is determined whether it has been reached. When it is confirmed in step S32 that the collected data is equal to or greater than the threshold value, the fast factor analysis unit 23 is started in step S33, and the following calculation is executed in the fast factor analysis unit 23 thereafter. .

In other words, in the statistical model that assumes latent variables as described above, parameter estimation is performed by constructing complete likelihood using only observed data, not including missing data in missing data. I do. Here, an EM algorithm for estimating parameters of a statistical model including latent variables using missing data will be described as an example. Assume that the p-dimensional observation variable is X = (X1,..., Xp) ^T and the m-dimensional latent variable vector is F = (F1,..., Fm) ^T.

  Now, consider estimating the parameter θ by the maximum likelihood method. When the observation variable is missing, if the observation data is described as X [n] and the missing value as X_ [n] for the nth observation, the complete data is Xn = [X [n], X_ [n]. ]. At this time, in the EM algorithm, it is necessary to calculate the conditional expected value E [logf (xn, fn) | x [n]] with the observation data of the complete log likelihood being given. If the data follow an independent equidistribution, the conditional expectation is

Given in. Here, f (xn, fn) is the simultaneous distribution of the observation data and the latent variable, and N is the sample size.

Next, the full log likelihood is
Calculated by

Also, the expected value of the complete log likelihood function is
E [Fn | xn] = M ^-1 Λ ^T Ψ ^-1 xn,
^{E [FnFn T | xn] =} M -1 + E [Fn | xn] E [Fn | xn] T
Is calculated by However, M = Λ ^T Ψ ⁻¹ Λ + Im.
Also, for this calculation
O (p ₀ N)
Need. Here, p ₀ represents the average number of variables that are not missing.

  FIG. 7 is a flowchart showing the procedure and processing contents of the process for calculating the expected value of the complete log likelihood and the complete log likelihood function. In the figure, the fast factor analysis unit 23 initially sets i = 1 in step S41, and then accepts inputs of μ0 ^, Λ0 ^, and Ψ0 ^ as initial values in step S42. Next, after incrementing the value of i in step S43, the E step is executed in step S44. In this E step, the conditional expected value fn ^ of the common factor is calculated. Subsequently, the M step is executed in step S45. In this M step, the process of maximizing the complete log likelihood is performed.

  When the processing of the E step and M step is completed, it is determined whether or not the parameter has converged in step S46. If the parameter has converged, the data is stored in the analysis data storage unit 32 in order to display the solution in step S47. After that, the process ends. On the other hand, if the parameter has not converged, the number of iterations is determined in step S48, and if i has not reached the upper limit value, the process returns to step S43 to increment the value of i, and then the steps S44 and S45 described above are performed. Execute the process. On the other hand, when the number of iterations reaches the upper limit value, a message indicating that convergence is not performed is stored in the analysis data storage unit 32 in step S49.

The calculation method described above is a method in which only variables (common factors) that cannot be directly observed in the existing EM algorithm shown in FIG. 20 are regarded as latent variables. In this case, the complete log likelihood is constructed based only on the observed variables. For this reason, when calculating conditional expectation values with complete log-likelihood data given, the size of the matrix to be calculated is reduced unless missing values are included in the complete data, and therefore matrix operations are also performed. Less.
Incidentally, the calculation order of matrix operations required for one iteration required O (p ² N) in the conventional EM algorithm, but as described above when the EM algorithm based on this embodiment is used. O (p ₀ N). Thereby, the calculation speed is greatly improved.

(4) Transmission of Analysis Data Data representing the analysis result obtained by the factor analysis process is stored in the analysis data storage unit 32 in association with the questionnaire ID. When the analysis processing by the high-speed factor analysis unit 23 is completed, the analysis server SVa subsequently sends the analysis data, that is, the analysis value and error value of the factor from the analysis data storage unit 32 in step S34. read out. Then, the analysis value and error value of the read factor are transmitted from the communication interface unit 1 to the requesting service provider terminal SP. When the distribution destination of the analysis data is set in advance, the analysis data is transmitted from the communication interface unit 1 toward the user terminals UT1 to UTn.

(Effects of the first embodiment)
As described above in detail, the first embodiment employs a parameter estimation method that maximizes the conditional expected value of only the observed data and the latent variable, instead of maximizing the conditional expected value. . For this reason, it is only necessary to estimate only the latent variables without estimating all the missing data by iterative calculation in the algorithm. Therefore, even if there are many defects, it is possible to calculate at high speed. Here, the lower limit of missing values that can be calculated at high speed is about 50%.

  FIG. 6 shows a matrix M = (mij) of 0-1 so that the data matrix is X = (xij) and represents the presence or absence of a defect. “0” indicates that xij is not observed, and “1” indicates that xij is observed. In this embodiment, since “missing data” does not include missing values, parameter estimation is performed using only the “1” portion of the matrix in FIG. 4. For this reason, in a situation where i is large and “0” is large, the time required for calculating the expected value to be substituted for “0” is not required, and the calculation speed can be increased accordingly. Incidentally, in the conventional method, since “missing data” includes missing values, parameters are estimated while estimating missing values x_ [n] corresponding to “0”. Therefore, when i is large and “0” is large, the amount of calculation becomes extremely large, and a very long time is required for the calculation process.

  In the first embodiment, by performing the factor analysis process by applying the above algorithm, the calculation speed can be increased as compared with the case where the conventional EM algorithm is used. Specifically, under the conditions of 90% deficit, about 50 times the number of respondents, 90 variables, and 5,000 samples, based on the pseudo Newton method described earlier as (A), (B ) Can be calculated at about two orders of magnitude faster than when the conventional EM algorithm described as) is used.

  FIG. 8 shows a change in the calculation speed improvement ratio with respect to the number of defects when the algorithm in the first embodiment is used in comparison with the conventional EM algorithm and the pseudo Newton method. As is apparent from the figure, in the algorithm according to the first embodiment, the speed improvement ratio increases as the number of defects increases.

[Second Embodiment]
In the second embodiment of the present invention, when information indicating how much error analysis result the factor analysis result desired by the service provider administrator wants to derive is set, it is necessary from the missing rate and the number of variables. The analysis server returns the correct number of samples.

(Constitution)
FIG. 9 is a block diagram showing a functional configuration of an analysis server SVb which is the second embodiment of the data analysis apparatus according to the present invention. In the figure, the same parts as those in FIG.

  In addition to the collected data storage unit 31 and the analysis data storage unit 32, the storage unit 3b is provided with a setting error storage unit 33. The setting error storage unit 33 is used to store an error required for the analysis result desired by the service provider.

  In addition to the collected data management unit 21, the sample number threshold determination unit 22, the fast factor analysis unit 23, and the analysis data display control unit 24, the control unit 2b includes a required error setting unit 25, a defect rate determination unit 26, A variable number confirmation unit 27 and a relationship determination unit 28 are further provided.

  The request error setting unit 25 receives information representing a request error transmitted from the service provider terminal SP via the communication interface unit 1 and stores the received information representing the request error in the setting error storage unit 33. I do.

  The missing rate determination unit 26 performs a process of determining how much data is missing for each attribute in the collected data stored in the collected data storage unit 31. The variable number confirmation unit 27 performs a process of confirming how many variables are used in the factor analysis in the collected data stored in the collected data storage unit 31.

  The relationship determination unit 28 includes a relationship determination data table. This relationship determination table stores the minimum number of samples necessary for factor analysis, which is set in advance for all combinations of the number of variables, required error, and loss rate. An example is shown in FIG. Then, by referring to this relationship determination table, the number of variables confirmed by the variable number confirmation unit 27, the loss rate determined by the loss rate determination unit 26, and the setting error storage unit 33 are stored. The minimum required number of samples corresponding to the combination of the required errors that have been searched is searched, and a process of giving the searched minimum required number of samples to the sample number threshold determination unit 22 is performed.

(Operation)
Next, the collected data analysis operation by the analysis server SVb configured as described above will be described.
In the service provider terminal SP, it is assumed that the manager inputs the attribute that the factor analysis is desired to be subjected to the factor analysis and the required error information indicating the error range within which the result is desired. Then, the input request error information is transmitted from the service provider terminal SP to the analysis server SVb.

  In the analysis server SVb, when the request error information transmitted from the service provider server terminal SP is received by the communication network unit 1, the request error setting unit 25 stores the received request error information in the setting error storage unit 33. To do.

  Subsequently, the analysis server SVb uses the loss rate determination unit 26 to determine how much deficiency occurs in the corresponding attribute in the collected data stored in the collected data storage unit 31 based on the questionnaire format data created by the service provider. Find out what you are doing. Further, the variable number confirmation unit 27 counts the number of variables corresponding to the corresponding attribute in the collected data accumulated in the collected data storage unit 31.

  Next, the analysis server SVb counts the required error value stored in the setting error storage unit 33, the missing rate determined by the missing rate determination unit 26, and the variable number confirmation unit 27 by the relationship determination unit 28. The relationship determination data table is searched based on the number of variables thus obtained, and the minimum number of samples necessary for factor analysis corresponding to the required error value, missing rate, and number of variables is read from the table. Then, the read sample number is set as a threshold value in the sample number threshold value determination unit 22.

  Therefore, when the collected data analysis process is performed, the sample number threshold determination unit 22 sets the number of samples of the collected data accumulated in the collected data storage unit 31 to the number of samples set in the sample number threshold determination unit 22. Compared to the value. If the number of samples of the collected data is equal to or greater than the set value of the number of samples, a request for execution of factor analysis processing on the collected data is sent to the fast factor analysis unit 23. The processing described in the first embodiment is executed.

  On the other hand, when the number of collected data samples is less than the set number of samples, the sample number threshold value determination unit 22 is stored in the collected data storage unit 31 from the minimum necessary number of samples. The value obtained by subtracting the number of samples of collected data is obtained. Then, information indicating how many samples are required is transmitted from the collected data management unit 21 to the service provider terminal SP.

(Effect of 2nd Embodiment)
As described above, according to the second embodiment, the value of the required error stored in the setting error storage unit 33, the loss rate determined by the loss rate determination unit 26, and the variable number check unit 27 are counted. The minimum number of samples required to perform the factor analysis within the error requested by the service provider can be read from the data table and set in the sample number threshold determination unit 22 based on the number of variables that have been set. . Therefore, an optimal threshold value can always be set in the sample number threshold value determination unit 22.

[Third Embodiment]
In the third embodiment of the present invention, a high-speed structural equation modeling processing unit 231 is provided in place of the high-speed factor analysis unit 23 in the second embodiment.

  FIG. 11 is a block diagram showing a functional configuration of an analysis server SVc which is the third embodiment of the data analysis apparatus according to the present invention. In the figure, the same parts as those in FIG.

  The control unit 2 c includes a high-speed structural equation modeling processing unit 231 instead of the high-speed factor analysis unit 23. The high-speed structural equation modeling processing unit 231 executes structural equation modeling processing using an EM algorithm that forms complete likelihood using only observed data, without including missing data in the complete data. To do. That is, the structural equation modeling process is executed by applying a parameter estimation method that maximizes the conditional expected value of only observed data and latent variables without maximizing the conditional expected value.

  The major difference between exploratory factor analysis and structural equation modeling is that exploratory factor analysis is a way to find invisible factors from observed data, whereas structural equation modeling assumes complex relationships between factors and observed variables. This is a method for verifying whether the assumed model is correct.

As a model often used in structural equation modeling, there is a LISREL model (LInear Structural RELations model). This LISREL model is
y = (y1, ..., yp) ^T
x = (x1,..., xq) ^T
As an observation variable, and
η = (η1, ..., ηm) ^T
ξ = (ξ1, ..., ξt) ^T
Is a latent variable,
y = Λyη + δ ^y
x = Λxξ + δ ^x
η = Bη + Γξ + δ
It is expressed as still,
η to N (0, Ωη) δ ^{y to} N (0, Σy)
ξ to N (0, Ωξ) δ ^{x to} N (0, Σx)
It is.

Suppose that the parameters included in this model are estimated by the EM algorithm. Extract the observed variables from yn and xn and calculate the complete log-likelihood function based only on those variables.
With this configuration, it is possible to speed up the calculation by about two digits. That is, a variable observed from yn and xn is extracted, and Σy, Λy, Σx, and Λx corresponding to the variable are used.

[Fourth Embodiment]
In the fourth embodiment of the present invention, a fast penalty type maximum likelihood processing unit 232 is provided in place of the fast factor analysis unit 23 in the second embodiment.

  FIG. 12 is a block diagram showing a functional configuration of an analysis server SVd which is the fourth embodiment of the data analysis apparatus according to the present invention. In the figure, the same parts as those in FIG.

  The control unit 2 d includes a fast penalty type maximum likelihood processing unit 232 instead of the fast factor analysis unit 23. The high-speed penalty formula maximum likelihood processing unit 232 applies a parameter estimation method that uses the full log likelihood used in the maximum likelihood method as it is for the complete log likelihood of the EM algorithm and maximizes it by adding a penalty term. Analyzing collected data.

  With such a configuration, in the analysis processing of the collected data, the fast penalty type maximum likelihood processing unit 232 uses the complete log likelihood used in the maximum likelihood method as the complete log likelihood of the EM algorithm as it is, and further penal terms To maximize the process. The ordinary maximum likelihood method often cannot be analyzed when the dimension of collected data is larger than the sample size. However, by using the penalized maximum likelihood method, it is possible to analyze when the dimension of the collected data is larger than the sample size. Furthermore, even when there are a large amount of deficiencies, as in the first embodiment, it is only necessary to estimate the latent variables without estimating all the deficient data by iteratively calculating in the algorithm. It can be calculated at high speed.

[Fifth Embodiment]
In the fifth embodiment of the present invention, a high-speed principal component analysis unit 233 is provided instead of the high-speed factor analysis unit 23 in the second embodiment.

  FIG. 13 is a block diagram showing a functional configuration of an analysis server SVe which is the fifth embodiment of the data analysis apparatus according to the present invention. In the figure, the same parts as those in FIG.

  The control unit 2 e includes a high-speed principal component analysis unit 233 instead of the high-speed factor analysis unit 23. The fast principal component analysis unit 233 does not include missing data that has not been observed in the complete data, but uses the EM algorithm that constructs the complete likelihood using only the observed data. Perform analysis.

  Principal component analysis is used as a method of compressing high-dimensional data to low dimensions. However, principal component analysis normally cannot fill in missing values because it does not include a probability structure. However, in the fifth embodiment, as described above, a method of calculating a principal component analysis including a probabilistic structure by an EM algorithm is adopted, and this method is further expanded to employ an EM algorithm that does not supplement missing values in the case of a large number of defects. To do. In this way, even if there are a large number of deficiencies, it is only necessary to estimate only the latent variables instead of estimating all the deficient data by iteratively calculating in the algorithm. Principal component analysis can be performed.

[Sixth Embodiment]
In the sixth embodiment of the present invention, a fast factor regression model processing unit 234 is provided in place of the fast factor analysis unit 23 in the second embodiment.

FIG. 14 is a block diagram showing a functional configuration of an analysis server SVf which is a sixth embodiment of the data analysis apparatus according to the present invention. In the figure, the same parts as those in FIG.
The control unit 2 f includes a fast factor regression model processing unit 234 instead of the fast factor analysis unit 23. The fast factor regression model processing unit 234 applies a parameter estimation method for constructing a complete likelihood using only the observed data to the factor regression model without including the missing data not observed in the complete data. In this way, similar explanatory variables are collectively estimated as one factor.

  This factor regression model can be estimated by the EM algorithm, but it takes a long time to apply the normal EM algorithm when there are a large number of missing explanatory variables. Therefore, the parameters can be estimated at high speed by using an EM algorithm that does not fill in missing values as in this embodiment. This method is particularly useful when there are a large number of explanatory variables and a large number of explanatory variables are missing.

[Other Embodiments]
In each of the above-described embodiments, the case where a series of analysis processes from collection / accumulation of collected data to transmission of analysis results are all executed by the analysis server has been described as an example, but this example process is divided into a plurality of processes. The plurality of divided processes may be distributedly processed by a plurality of servers.

  In addition, the configuration of the analysis server, the analysis processing procedure, the processing content, and the like can be variously modified and implemented without departing from the gist of the present invention.

  In short, the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in each embodiment. Furthermore, you may combine suitably the component covering different embodiment.

  SV ... analysis server, SP ... service provider terminal, UT1 to UTn ... user terminal, NW ... communication network, 1 ... communication interface unit, 2a, 2b, 2c, 2d, 2e, 2f ... control unit, 3a, 3b ... storage unit , 21 ... Collected data management unit, 22 ... Sample number threshold determination unit, 23 ... Fast factor analysis unit, 24 ... Analysis data display control unit, 25 ... Required error setting unit, 26 ... Loss rate determination unit, 27 ... Number of variables Confirmation unit 28 ... Relationship determination unit 31 ... Collected data storage unit 32 ... Analysis data storage unit 33 ... Setting error storage unit 231 ... High-speed structural equation modeling processing unit 232 ... Maximum likelihood processing unit with high-speed penalty 233: fast principal component analysis unit, 234: fast factor regression model processing unit.

Claims (5)

It means for storing in the storage medium receives the observed observed data,
When statistical analysis processing is performed on observation data stored in the storage medium, a statistical model that assumes latent variables does not include missing data that has not been observed, and uses only the observation data and latent variables. An analysis means for estimating a parameter included in the statistical model by configuring likelihood , and
The analysis means includes
The dimension of the observed variable is p, the number of factors is m, the p-dimensional observed variable vector is X = (x1, ..., xp) ^T , the p-dimensional average vector is μ = (μ1, ..., μp) ^T , and the m-dimensional latent variable The vector is F, the p × m factor loading matrix is Λ = (λij) = (λ1,..., Λp) ^T , the p-dimensional unique factor vector is ε, the sample size is N, and iobs (n) is n (= 1,. , N) In the factor analysis model represented by X−μ = ΛF + ε, where i is the subscript i of the variable xi observed in the th sample,
Accepting input of each initial value of the p-dimensional mean vector μ, the p × m factor loading matrix Λ, and Ψ = diag (ψ1,...
Using the EM algorithm, the complete log likelihood constructed based only on the observed variable xi is
Is given as a conditional expectation value E of the full log likelihood, using a matrix M representing the presence or absence of deficiency,
E [Fn | xn] = M ^-1 Λ ^T Ψ ^-1 xn,
^{E [FnFn T | xn] =} M -1 + E [Fn | xn] E [Fn | xn] T
And the matrix M = (mij) is
M = Λ ^T Ψ ^-1 Λ + Im
Calculated by
In addition, a penalty term is added to maximize the logarithmic likelihood.
After the calculation process of the conditional expected value E and the maximization process of the complete log likelihood added with the penalty term, it is determined whether the parameter has converged,
When it is determined that the value has not converged, the process of calculating the conditional expected value E and the process of maximizing the complete log likelihood added with the penalty term are repeated until a preset number of iterations is reached. Data analysis device. It means for storing in the storage medium receives the observed observed data,
When statistical analysis processing is performed on observation data stored in the storage medium, a statistical model that assumes latent variables does not include missing data that has not been observed, and uses only the observation data and latent variables. Analyzing means for estimating parameters included in the statistical model by configuring likelihood ;
Threshold determination means for determining whether observation data equal to or greater than the threshold is stored by comparing the number of samples of observation data stored in the storage medium with a set threshold;
A table in which the minimum number of samples necessary for the parameter estimation is stored in advance for the combination of the number of variables used for the parameter estimation, the required error, and the loss rate, and by referring to this table, The sample corresponding to a combination of the number of variables in the observation data stored in the storage medium, the missing rate obtained from the observation data stored in the storage medium and the determined missing data, and the specified required error A relationship determination unit that searches for a number and sets the number of samples as the threshold of the threshold determination unit ;
The analysis unit executes a process of estimating the parameter with respect to the observation data stored in the storage medium when the threshold determination unit determines that observation data equal to or greater than the threshold is stored. Data analysis equipment. A data analysis method executed by a data analysis device including a computer and a storage medium,
A process of storing in the storage medium receiving the observed observed data,
When statistical analysis processing is performed on observation data stored in the storage medium, a statistical model that assumes latent variables does not include missing data that has not been observed, and uses only the observation data and latent variables. An analysis process for estimating a parameter included in the statistical model by configuring a likelihood , and
In the analysis process,
The dimension of the observed variable is p, the number of factors is m, the p-dimensional observed variable vector is X = (x1, ..., xp) ^T , the p-dimensional average vector is μ = (μ1, ..., μp) ^T , and the m-dimensional latent variable The vector is F, the p × m factor loading matrix is Λ = (λij) = (λ1,..., Λp) ^T , the p-dimensional unique factor vector is ε, the sample size is N, and iobs (n) is n (= 1,. , N) In the factor analysis model represented by X−μ = ΛF + ε, where i is the subscript i of the variable xi observed in the th sample,
Accepting input of each initial value of the p-dimensional mean vector μ, the p × m factor loading matrix Λ, and Ψ = diag (ψ1,...
Using the EM algorithm, the complete log likelihood constructed based only on the observed variable xi is
Is given as a conditional expectation value E of the full log likelihood, using a matrix M representing the presence or absence of deficiency,
E [Fn | xn] = M ^-1 Λ ^T Ψ ^-1 xn,
^{E [FnFn T | xn] =} M -1 + E [Fn | xn] E [Fn | xn] T
And the matrix M = (mij) is
M = Λ ^T Ψ ^-1 Λ + Im
Calculated by
In addition, a penalty term is added to maximize the logarithmic likelihood.
After the calculation process of the conditional expected value E and the maximization process of the complete log likelihood added with the penalty term, it is determined whether the parameter has converged,
If it is determined that it has not converged, the calculation process of the conditional expected value E and the maximization process of the full log likelihood added with the penalty term are repeated until a preset number of iterations is reached. > A data analysis method characterized by A data analysis method executed by a data analysis device including a computer and a storage medium,
A process of storing in the storage medium receiving the observed observed data,
When statistical analysis processing is performed on observation data stored in the storage medium, a statistical model that assumes latent variables does not include missing data that has not been observed, and uses only the observation data and latent variables. An analysis process for estimating parameters included in the statistical model by configuring likelihood ;
A threshold determination step of determining whether observation data equal to or greater than the threshold is stored by comparing the number of observation data samples stored in the storage medium with a set threshold;
By referring to a table in which the minimum number of samples necessary for parameter estimation is stored in advance for the combination of the number of variables used for parameter estimation, the required error, and the defect rate, the parameter is stored in the storage medium. The number of variables in the observed data, the missing rate determined from the observed data stored in the storage medium and the determined missing data, and the number of samples corresponding to the specified required error are searched, A relationship determination step of setting the number of samples as the threshold in the threshold determination step ,
In the analysis process, when it is determined by the threshold determination process that observation data equal to or higher than the threshold is stored, a process for estimating the parameter is performed on the observation data stored in the storage medium, How to analyze data.   The program for making a computer perform the process of each process of Claim 3 or 4.