JP2010039756A - Independence test device, data analysis device, and independence test program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To accurately test the independence of an item in sample data even in the case of the small number of samples of sample data. <P>SOLUTION: Resampling from sample data is performed more than once with the number of pieces of resample data different by size, which is equal to or smaller than the number of pieces of sample data, for each piece of resample data, and a test statistic for testing the independence of two items in the sample data is calculated on the basis of resampled sample data, and a test statistic in the number of pieces of data larger than the number of pieces of the sample data is estimated on the basis of an approximate curve determined by a plurality of calculated test statistics per each piece of resampling data, and the estimated test statistic is compared with a threshold to determine the independence of two items. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an independence determination device, a data analysis device, and an independence test program.

  In statistics, knowing whether certain variables have interdependencies is a basic and important analysis. For example, there are many examples such as the relationship between disease symptoms and efficacy, age, or gender in the medical field, and the relationship between the occurrence of defects in production plants and various environmental factors. As a method for examining the independence between variables, a method called a statistical hypothesis test method has been developed, and a chi-square test, a likelihood ratio test (G test), and the like are known (for example, Non-Patent Document 1). reference). With these conventional statistical hypothesis testing methods, the probability distribution function that outputs data can be approximated by a normal distribution by the central limit theorem, assuming that a lot of data is obtained, and the statistic is derived from the approximation. Since it is assumed that it can be expressed by a square distribution, when only a small number of data can be obtained, the accuracy of approximation by the normal distribution deteriorates, so that the reliability of the threshold value is lowered, and therefore the reliability of the hypothesis test result is lowered.

Further, as a technique for calculating a small amount of data with high accuracy, a bootstrap hypothesis testing method is known (for example, see Non-Patent Document 2). When bootstrap hypothesis testing is performed, there is an effect of improving the accuracy of the statistics in the obtained data, but the threshold accuracy of the chi-square distribution for calculating the threshold for comparison is low because the reliability of the approximation accuracy is low. The reliability remains low, and the reliability of hypothesis testing with a small number of data is still low.
“Department of Statistics of Natural Science”, Faculty of Liberal Arts, University of Tokyo, University of Tokyo Press, 1992 “Science Frontier 11 in Statistical Science I”, Iwanami Shoten, 2003, p. 56

  The present invention provides an independence test apparatus, a data analysis apparatus, and an independence test program that can accurately test the independence of items in sample data even when the number of sample data is small. Objective.

  In order to achieve the above object, the independence test apparatus according to the present invention sets a plurality of resample numbers whose sizes are not more than the sample number of a plurality of sample data having a plurality of items each consisting of an event corresponding to a random variable. And extracting means for extracting the sample data of the number of resamples from the plurality of sample data for each set number of resamples, and the sample data extracted by the extraction means based on the sample data extracted by the extraction means To test the independence of two items among a plurality of items or a conditional independence test amount for testing the independence of the two items on the condition of one or more items other than the two items A calculation for calculating an independence test amount based on a G test amount or a chi-square test amount for measuring a value corresponding to the degree of correlation between the two items for each resample number. Means and the calculating means An estimation means for estimating an independence test quantity in a larger number of samples than the number of samples based on an approximated curve determined from the independence test quantity for each resample number issued; and an independence test quantity estimated by the estimation means And determining means for determining that the two items are independent when the value is equal to or less than a predetermined value.

  Further, the calculation means may calculate the independence test amount by converting the G test amount or the chi-square test amount into a mutual information amount or a conditional mutual information amount.

  Further, the extraction means performs a plurality of extractions for one resample number, and the calculation means calculates a plurality of independence test amounts for each resample number, and the plurality of independent test quantities. The independence test amount for each resample number can be calculated from the average value of the sex test amount.

  In addition, the extraction unit performs a plurality of extractions for one resample number, the calculation unit calculates a plurality of independence test amounts for each resample number, and the estimation unit includes: The independence test amount in the number of samples larger than the number of samples can be estimated based on an approximate curve determined by a weighted least square method based on the variance of the plurality of independence test amounts.

  The data analysis apparatus according to the present invention relates to the independence test apparatus and items determined to be not independent by the determination means, and constitutes a Bayesian network or a Markov network for a plurality of items of the sample data. And a configuration means.

  Further, the independence test program according to the present invention sets a plurality of resample numbers having different sizes below the sample number of a plurality of sample data having a plurality of items each consisting of an event corresponding to a random variable. Extraction means for extracting the sample data of the number of resamples from the plurality of sample data for each number of resamples, and a plurality of items of sample data extracted by the extraction means based on the sample data extracted by the extraction means The independence of two items in or the independence to test a conditional independence test amount for testing the independence of the two items subject to one or more items other than the two items A calculation means for calculating an independence test quantity based on a G test quantity or a chi-square test quantity for measuring a value according to the degree of correlation between the two items for each resample number; Said calculating means Based on an approximate curve determined from the calculated independence test amount for each resample number, an estimation means for estimating an independence test amount in a sample number larger than the sample number, and an independence test amount estimated by the estimation means Is a program for causing the two items to function as determination means for determining that the two items are independent when the value is equal to or less than a predetermined value.

  As described above, according to the independence test apparatus according to claim 1 and the independence test program according to claim 6, the independence of items in sample data can be increased even when the number of sample data is small. The effect that the test can be performed with high accuracy is obtained.

  In addition, according to the independence test apparatus described in claims 2 to 4, it is possible to obtain an effect that the test amount for testing the independence can be accurately estimated.

  Further, according to the data analysis apparatus of the fifth aspect, it is possible to obtain an effect that the data analysis can be performed with high accuracy based on the independence between items determined with high accuracy.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Hereinafter, a case will be described in which the independence test apparatus of the present invention is applied to a data analysis apparatus that analyzes the relationship between items by analyzing sample data to form a Bayesian network.

  As shown in FIG. 1, the data analysis apparatus 10 according to the present embodiment includes an input device 12 such as operation keys, a keyboard, a mouse, and a touch panel for inputting various settings and conditions, and a configured Bayesian network. A display device 14 such as a display for visualizing and displaying, and a computer 16 for executing data analysis processing are provided.

  The computer 16 includes a CPU 24 that controls the entire data analysis apparatus 10, a ROM 26 that stores various programs such as a data analysis program and an independence test program described later, a RAM 28 that temporarily stores data as a work area, It includes an HDD (hard disk) 30 as storage means for storing information, a network I / F (interface) unit 32 for connecting to a network, an I / O (input / output) port 34, and a bus connecting these. It is configured. The input device 12 and the display device 14 are connected to the I / O port 34.

  When the computer 16 is described with functional blocks divided for each function realization means determined based on hardware and software, as shown in FIG. 2, a data extraction unit 36 for resampling from sample data, and a data extraction unit Based on the sample data extracted in 36, a test amount calculation unit 38 that calculates a test amount for testing independence, and the number of sample data data based on the test amount calculated by the test amount calculation unit 38 An estimator 40 for estimating a test amount for a larger number of data; a determination unit 42 for determining whether two items in the sample data are independent based on the test amount estimated by the estimator 40; Based on the determination result of 42, the sample data items are represented by a configuration including a network configuration unit 44 that configures a Bayesian network. Door can be.

  Here, regarding the principle of the independence test of the present embodiment, n sample data holding values (events) corresponding to random variables for a plurality of items (X, Y, Z...) Are given. A case where the independence between the item X and the item Y is tested will be described.

First, natural numbers m ₁ , m ₂ ,..., _{M i ,.} Next, a hypothesis that “item X and item Y are independent under item Z” is made, m ₁ sample data is resampled from the given n sample data, and based on the hypothesis Based on the obtained theoretical value and the actual measurement value obtained from the sample data, an independence test amount is calculated as a metric representing the degree of conditional correlation between the item X and the item Y. Similarly, m ₂ ..., _{M i} ,..., _{Mi max} sample data are resampled from n sample data, and a test amount is calculated for each.

As shown in FIG. 3, the calculated test amount is plotted on the coordinate system with the number of sampling data on the horizontal axis and the test amount on the vertical axis (• mark). Approximate curve is obtained from the test amount calculated for the number of resampling data of m ₁ , m ₂ ,..., m _{i ,.} Estimate the test amount at N (x mark). Independence is determined by comparing the estimated test amount with a threshold value. As the verification amount, it is desirable to use a G statistic or a chi-square statistic that can be easily converted into mutual information or conditional mutual information. And if these test statistics are below the threshold of the chi-square distribution calculated using the value (5%) normally used as the significance of the chi-square distribution, the hypothesis is correct (item X, item Y and Is determined to be independent), and if it is greater than the threshold, it is determined that the hypothesis is incorrect (item X and item Y are not independent).

  Next, the processing routine of the data analysis program in the present embodiment will be described with reference to FIG. This section describes the case of configuring a Bayesian network to learn a person's profile, preferences, and preferences, determine whether new email is important to the individual, and build a system that informs the person of the result. .

  In step 100, sample data is acquired. The sample data may be input from the input device 12, stored in a storage device externally connected via a network, or stored in advance in the HDD 30 of the computer 16. Here, it is assumed that 500 pieces of sample data having item X, item Y, and item Z are acquired.

  In the sample data, as shown in FIG. 5, the item X is an item (conference) regarding whether or not the word “conference” exists in the text of the e-mail. Event x1 = “present”, event x2 = “none” ". Item Y is an item (frequently) regarding whether or not frequent electronic mail has been exchanged with the sender of the electronic mail in the past, and event y1 = “present” and event y2 = “none”. The item Z is an item (important) regarding whether or not the new arrival e-mail is important, and event z1 = “important” and event z2 = “not important”. The items in parentheses are item names that directly represent the contents of each item.

  Here, if the random variable of the event x is P (x), the events x1 and x2 of the item X are P (x1) = 1/2 and P (x2) = 1 / 2. When based on the actually measured value of the resampled sample data, P (x1) = number of x1 / number of resampled data, P (x2) = number of x2 / number of resampled data. The same applies to item Y and item Z. Thus, the sample data has a plurality of items consisting of events corresponding to random variables.

  Next, in step 102, an item for analyzing the correlation between items is determined by testing the independence among the items in the sample data. Here, in order to test conditional independence, two items and one item as a condition are determined. This determination may be performed according to a predetermined rule for any item in order to analyze the correlation for all combinations of items of the sample data, or based on a selection signal input from the input device 12 by user selection. You may go.

  Here, in determining whether new mail is important or not, there is a correlation between whether or not there is a word “meeting” in the text and whether or not the sender has frequently exchanged mail in the past. For the purpose of analyzing whether or not there is, item X and item Y are determined as items for testing independence, and item Z is determined as a condition item.

  Next, in step 104, the independence test process described later is executed to test the independence of the items determined in step 102.

  Next, in step 106, a network between items is constructed based on the independence test result. If the conditional independence hypothesis is rejected as a test result, it means that there is a correlation between items, so it is determined that there is an edge constituting the Bayesian network. Further, when the conditional independence hypothesis is adopted, it means that there is no correlation between items, so that it is determined that there is no edge constituting the Bayesian network. Based on this determination, a Bayesian network is configured. For example, if the hypothesis that “the item X and the item Y are independent under the condition of the item Z” is not rejected, the item X and the item Y are independent under the condition of the item Z. There will be no edge between Y, and if X and Z and all conditional independence hypotheses are rejected between Y and Z, there will be an edge between X and Z and between Y and Z. Will exist. At this stage, the edge is in a state of an undirected edge that is not yet a directed edge, but an orientation rule that determines the direction of a directed edge in consideration of conditional independence in a Bayesian network (for example, “C. Meek,“ Causal According to “Inference and Casual Expansion with Background Knowledge”, “Conference on Uncertainty in Artificial Intelligence, 1995”), a Bayesian network as shown in FIG. 6 is configured.

  Next, in step 108, it is determined whether or not to end the analysis. When determining items according to a predetermined rule, it is determined whether or not the analysis has been completed for all combinations of items. When an item is determined by the user's selection, a selection screen for determining whether or not to analyze another item is displayed, and the determination is made based on a selection signal input by the user. When the analysis is to be terminated, the process proceeds to step 110, where the configured Bayesian network is visualized and displayed on the display device 14, and the process is terminated. If the analysis is not terminated, the process returns to step 102 and the analysis process is repeated for another item. The configured Bayesian network data may be output to an external device connected via the network.

  Next, the processing routine of the independence test process executed in step 104 of the data analysis process (FIG. 4) will be described with reference to FIG.

In step 200, the number of resampling data indicating the number of data at the time of resampling from the sample data m _i, and the sampling times jmax indicating whether to perform many times resampling for each resampled data number m _i is determined. This determination may be made according to a predetermined rule based on the number n of sample data, or may be made by input from the user. In the case of determining according to a predetermined rule, for example, m ₁ = n × 0.2, m ₂ = n × 0.4, m ₃ = n × 0.5, m ₄ = n × 0.6, m ₅ = n × 0.7, m ₆ = n × 0.8, m ₇ = n × 0.9, j = 10 times. Here, since the number n of the sample data is 500, resampling the data number _{m i} is determined 100/200/250/300/350/400/450 and.

Next, in step 202, "1" is set to the i and j, then in step 204, it performs the j th resampling resampling data number m _i. Here, since i and j are “1”, the first resampling is performed with the number of resampling data m ₁ (100).

Next, in step 206, a test amount for testing the independence is calculated based on the resampled sample data. Here, the mutual information amount of the formula (4) is used as the test amount by the conversion relationship (formula (3)) of the G test amount represented by the following formula (1) with the mutual information amount represented by the following formula (2). the value of χ-square distribution is also used divided by the similarly 2m _i. When the χ square test amount is used, the χ square test amount itself is an approximation of the G test amount, so that it can be converted into the mutual information by the same formula. Since the mutual information is a metric showing the correlation calculated from the probability value, it can be seen that if a sufficient number of data is obtained, it converges to a certain value by the law of large numbers (for example, see Non-Patent Document 1). . Therefore the following equation (4) converges to a certain value once the resampled data number m _i by sufficient number of data can be obtained.

Note that O is the observation frequency, and E is the expected frequency under the hypothesis, which is the frequency when assumed to be independent or conditional independent.

Note that z with an arrow (also expressed as “z ^* ”) represents a combined event of at least one item (event). Further, p (x, y, z ^* ) is the joint probability of the event x, y, z ^* , and p (x, y | z ^* ) is the event x, y under the condition that the event z ^* occurs. , P (x | z ^* ) is the marginal probability of event x under the condition that event z ^* occurs, and p (y | z ^* ) is the condition under which event z ^* occurs Is the marginal probability of event y. If the item X and the item Y are independent, the equation (5) is established, so that the mutual information amount is “0”, and the value increases as the correlation between the item X and the item Y increases.

The χ square test amount χ ² is

It is represented by Here, O and E are the same as defined in the G test amount.

For example, it is assumed that sample data with 100 resamplings is totaled and a totaling result as shown in FIG. 8 is obtained. From this total result, p (x, y, z) = 0.35, p (x) for event x = x1 (present) and event y = y1 (present) under the condition of event z = z1 (important). x, y | z) = 0.35, p (x | z) = 0.45, and p (y | z) = 0.45. Similarly, under the condition of event z = z1 (important), if x = x1 and y = y2, if x = x2 and y = y1, if x = x2 and y = y2, and so on, Under the condition z = z2 (not important), when x = x1 and y = y1, when x = x1 and y = y2, when x = x2 and y = y1, x = x2 and y = y2 Substituting the sum in the above case into the above equation (1), the test amount is calculated.

Next, in step 208, it is determined whether or not resampling for the determined number of samplings is completed by determining whether or not j = jmax. Here, it is determined to perform resampling 10 times for _one resampling data number m ₁ , and since it is still the first time, the result is negative and the process proceeds to step 210.

  In step 210, j is incremented by 1, and the process returns to step 204. The process is repeated until resampling for the determined number of sampling times (10 times) is completed. If j = jmax, the routine proceeds to step 212.

In step 212, calculates the average test weight MI _i and the variance values for one resampling data number m _i for j times the resampling data number m _i of the j-test weight MI _ij calculated by resampling . The calculated average test amount MI _i and its variance are temporarily stored in a predetermined storage area.

Next, at step 214, by determining whether a i = imax, it is determined whether to exit the resampling for all of the determined resampling data number m _i. Here, resampling the data number _{m i} is 100/200/250/300/350/400/450 of 7 but have been determined, yet because not only performed resampling at 100 resampling data, negative Then, the process proceeds to step 216.

In step 216, back increments i by one to step 204, for all of the determined resampling data number m _i repeats the processing until the resampling is finished. If i = imax, the process proceeds to step 218.

In step 218, as shown in FIG. 9, the horizontal axis resampling data number, test weight on the vertical axis on the coordinate system taking the value in terms of mutual information in terms of the average test each resampled data number m _i The quantity MI _i is plotted (square mark). Note that the error bar indicated at each plotted point indicates the distribution of the test amount MI _ij calculated for each sampling frequency j. Based on these points, the approximate curve 50 is calculated by the least square method or the weighted least square method by variance.

Next, in step 220, the number of data sufficiently larger than the number n of sample data is determined as the number of extrapolated data N. The number N of extrapolated data may be determined based on the number n of sample data, the number of resampled data m _i , the number of sampling times j, or the like, or may be determined by input from the user. Good. Based on the calculated approximate curve, it calculates the test statistic in the determined extrapolated data number N as the estimated test weight MI _N.

Next, in step 222, a threshold distribution 52 is calculated by converting the chi-square distribution with a significance level of 95% into mutual information based on the degree of freedom of the item to be tested for independence. th is calculated based on the threshold distribution 52. When the extrapolated data number N and 2000, as shown in FIG. 9, the estimated test weight MI _N and threshold value th is calculated.

Next, in step 224, the estimated test statistic MI _N, it is determined whether or not greater than the threshold th, when the estimated test statistic MI _N is greater than the threshold th, the process proceeds to step 226, the test result "rejection" ( outputs not independent) conditionally, when the estimated test statistic MI _N is less than the threshold value th, the test result outputs "adoption" (independently a is conditional), the process returns.

  As described above, the sample data obtained is resampled multiple times with different numbers of resampling data, and data sufficiently larger than the number of sample data based on the test amount calculated for each resampling. Since the test quantity in number is estimated, independence can be tested accurately even when the number of sample data is small, and by constructing a network between items using this test result, highly accurate data analysis is possible. It can be carried out.

  In the present embodiment, the case of testing conditional independence has been described. However, it is also possible to test for independence without conditions for two items without setting a condition item.

  In the present embodiment, the case where a Bayesian network is configured using the result of the independence test as data analysis has been described. However, if the network is configured based on correlation between two variables or conditional correlation. For example, a Markov network may be configured.

  In the present embodiment, the approximate curve is calculated from the average verification amount for each resampling. However, the verification amount for each resampling is plotted as it is (for example, the error bar for each point in FIG. 9). Range), an approximated curve may be calculated by a weighted least square method based on the variance of the test amount for each number of resampling data.

  In this embodiment, the G test amount is used as the test amount. However, a χ square test amount or another statistical independent test amount that can be converted into the mutual information amount may be used.

  Further, in the present embodiment, the case where the threshold value for comparison with the estimated test amount is calculated based on the χ square distribution is described, but the present invention is not limited to this, and an appropriate threshold value is set for each extrapolated data number. May be set.

  Further, in the present embodiment, the case where the number of items of sample data is three has been described, but sample data including more items may be used. Since the number of independence tests increases exponentially as the number of items increases, the PC algorithm (eg, “Computation, Causation, & Discover” edited by C. Glymour & F. Cooper, AAAI Press / MIT Press, 1999) Bayesian network construction algorithms based on efficient conditional independence tests such as

It is the schematic which shows the structure of the data analysis apparatus 10 which concerns on this Embodiment. It is a block diagram which shows the functional structure of the data analysis apparatus 10 which concerns on this Embodiment. It is a diagram for demonstrating the principle of the independence test of this Embodiment. It is a flowchart which shows the processing routine of the data analysis program in this Embodiment. It is a figure which shows an example of sample data. It is a figure which shows an example of the Bayesian network comprised by data analysis. It is a flowchart which shows the processing routine of the independence test | inspection program in this Embodiment. It is a table | surface which shows an example of a total result. It is a figure for demonstrating calculation of an estimated test amount and a threshold value.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Data analysis apparatus 12 Input apparatus 14 Display apparatus 16 Computer 36 Data extraction part 38 Test amount calculation part 40 Estimation part 42 Determination part 44 Network structure part

Claims (6)

A plurality of resample numbers having different sizes below the sample number of a plurality of sample data having a plurality of items each corresponding to a random variable are set, and the sample data of the resample number is set for each set resample number Extraction means for extracting from a plurality of sample data;
Based on the sample data extracted by the extraction means, an independence test amount for testing the independence of two items among the plurality of items of the sample data extracted by the extraction means or 1 other than the two items A conditional independence test quantity for testing the independence of the two items on condition of two or more items, and a G test quantity for measuring a value according to the degree of correlation between the two items Or a calculation means for calculating an independence test amount based on the chi-square test amount for each resample number;
Based on an approximate curve determined from the independence test amount for each resample number calculated by the calculation unit, estimation means for estimating an independence test amount in a sample number larger than the sample number;
A determination unit that determines that the two items are independent when the independence test amount estimated by the estimation unit is equal to or less than a predetermined value;
Independence tester including.   The independence test apparatus according to claim 1, wherein the calculation means calculates the independence test amount by converting the G test amount or the chi-square test amount into a mutual information amount or a conditional mutual information amount. The extraction means performs a plurality of extractions for one resample number,
The calculation means calculates a plurality of independence test amounts for the number of extractions for each resample number, and calculates an independence test amount for each resample number from an average value of the plurality of independence test amounts.
The independence test | inspection apparatus of Claim 1 or Claim 2. The extraction means performs a plurality of extractions for one resample number,
The calculation means calculates a plurality of independence test amounts for the number of extractions for each resample number,
The estimation means estimates an independence test amount in a sample number larger than the sample number based on an approximate curve determined by a weighted least square method based on a variance of the plurality of independence test amounts.
The independence test | inspection apparatus of Claim 1 or Claim 2. The independence test apparatus according to any one of claims 1 to 4,
Configuration means for associating items determined to be independent by the determination means and configuring a Bayesian network or a Markov network for a plurality of items of the sample data;
Data analysis device including Computer
A plurality of resample numbers having different sizes below the sample number of a plurality of sample data having a plurality of items each corresponding to a random variable are set, and the sample data of the resample number is set for each set resample number Extraction means for extracting from a plurality of sample data;
Based on the sample data extracted by the extraction means, an independence test amount for testing the independence of two items among the plurality of items of the sample data extracted by the extraction means or 1 other than the two items A conditional independence test quantity for testing the independence of the two items on condition of two or more items, and a G test quantity for measuring a value according to the degree of correlation between the two items Or a calculation means for calculating an independence test amount based on the chi-square test amount for each resample number;
Based on an approximate curve determined from the independence test amount for each resample number calculated by the calculation unit, estimation means for estimating an independence test amount in a sample number larger than the sample number;
A determination unit that determines that the two items are independent when the independence test amount estimated by the estimation unit is equal to or less than a predetermined value;
Independence test program to make it function.

Images