JP5925044B2 - Different number counting method, apparatus, and program - Google Patents

Description

  The present invention relates to a technique for estimating the number of different data groups having an enormous number of elements.

  In information processing, it is required to process a data group having an enormous number of elements efficiently and in a short time. In many cases, a data group used for information processing includes a plurality of elements of the same type. There is a technique for improving the efficiency of data group processing by utilizing the fact that the same type of elements are included in the data group. That is, the number of types of elements included in the data group, that is, the number of different types may be effectively used for information processing. The number of different types of data or data items of a finite data set is called the number of different data or data items.

  For example, in data sorting in a database in which an enormous amount of data is accumulated, it is known that the number of different elements included in the database is extracted and the sorting process can be accelerated by using the different numbers. Further, when storing data in the database, the entire data amount can be compressed by recording the type and frequency of the elements instead of storing all the data in individual areas. Even in that case, it is required to count the number of types of elements in advance.

  Further, if the number of users who have accessed a certain site can be extracted from an access log in which a huge access history to the site is accumulated, it can be used for access analysis. In that case, the number of different access source addresses included in the access log indicates the number of users. In a large-scale Internet site, in order to protect the server from various external attacks and access loads exceeding the allowable range, it is necessary to constantly monitor the access status from the outside. Measuring the number of unique access hosts from a large number of access logs is part of that. That is, the number of different host identification information such as IP addresses is counted from the access log data. In a large site, the number of accesses per day exceeds one billion, and a large amount of computer resources are required to grasp the statistics of all cases at all times.

  In creating translation software and dictionary databases, the number of words is extracted from a huge corpus and used to construct the database. In that case, the number of different corpora is the number of words.

  In addition, in address management, the number of surnames of residents can be extracted from the accumulated information of residents and used for management of residents data. In that case, the number of different surnames in the resident information indicates the number of surnames.

  As described above, counting the number of different data or data items is very useful in data processing. However, if the data group is enormous, it is different and enormous amount of processing and storage capacity are required for counting. Therefore, a technique for efficiently counting the number of different data groups is required.

  In response to such a requirement, Patent Document 1 discloses a technique for estimating the number of different data groups with high accuracy. The technique disclosed in Patent Document 1 attempts to estimate the number of differences with high accuracy by using a counting method suitable for counting small different numbers and a counting method suitable for counting large different numbers.

JP 2008-59293 A

  The technique of Patent Document 1 assumes that different numbers of various data groups cannot be counted with high accuracy by a single different number counting method, and estimates the different numbers of the same data group by two counting methods. . However, redundant processing is performed for that purpose, and processing efficiency is not good.

  An object of the present invention is to provide a technique for efficiently estimating the number of different data groups.

The different number counting method according to one aspect of the present invention is a different number counting method for counting the number of different populations of data groups, wherein the matrix generation means includes a size N of the population, and the population. Based on the extraction rate r for extracting a sample from the sample, the frequency difference number vector indicating the number of differences for each frequency of the population and the sample extracted from the population at the extraction rate, which is obtained by the equation (C-1) Approximation obtained by integrating and approximating the operation for obtaining the sum of discrete values in equation (C-2) using each element of the binomial probability matrix P that defines the relationship with the frequency vector different in frequency A step of obtaining a logarithmic compression binomial probability matrix LP which is a compression matrix obtained by compressing the binomial probability matrix P in a row direction and a column direction according to an equation, and generating an inverse matrix LP ⁻¹ of the logarithm compression binomial probability matrix LP. Unlike the number calculation An estimated logarithmic compression frequency difference number of the population is obtained by multiplying the logarithm compression frequency difference number vector obtained by compressing the frequency difference number vector of the sample extracted from the population and the inverse matrix LP ^−1. calculating a vector, has a step of calculating a sum of the estimated logarithmic compression power varies the number of vector elements of the population, as an estimate of the number of different of the population, the.

  According to the present invention, since the number of different populations can be estimated using a matrix calculated in advance and a sample extracted from the population, the number can be efficiently estimated without analyzing all the elements. be able to.

It is a block diagram of a different number counting device by this embodiment. It is a flowchart which shows operation | movement of the number counting apparatus which differs in this embodiment. It is a block diagram which shows an example of the data analysis apparatus 20 incorporating the number counting apparatus 10 different from this embodiment. It is the figure which showed an example of the access log typically. It is the figure which showed an example of the access analysis result data typically. It is a figure which shows a mode that a sample is extracted from a population. It is a diagram for explaining f- frequency different number U _f. It is the figure which showed the principle of the approximation by a binomial probability matrix with the toy model. It is a figure which shows a mode that a binomial probability matrix is compressed. It is a figure which shows the mode of the distribution of the element in a binomial probability matrix. It is a figure which shows a mode that a frequency vector differs in frequency. It is the figure which showed the principle of the approximation by a logarithm compression binomial probability matrix. It is a figure which shows the result of having changed the extraction rate in the estimation experiment 1, and comparing the estimation precision. It is a figure which shows the result of having changed the variable and the extraction rate in the estimation experiment 1, and comparing the estimation precision. FIG. 15 is a graph showing the true value in the result of FIG. 14. It is a figure which shows the estimation experiment result in the estimation experiment 2. FIG. It is a figure which shows a step function and its approximated curve.

  A basic embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of a different number counting device according to this embodiment. The different number counting device 10 of this embodiment is a device that counts the number of different populations composed of a huge data group.

  The population is, for example, an enormous amount of data stored in a database or a set based on the value of each item when the data is composed of a plurality of items. The number of data stored in the database is the size of the population. Since the population may contain data having the same value, the number of different populations is equal to or smaller than the size of the population.

  The different number counting apparatus 10 of the present embodiment uses an approximation for counting different numbers. The approximation method estimates the number of different populations from a sample group extracted from the population at a predetermined extraction rate. The extraction rate is given in advance.

  Here, a data vector such as a population and a sample group extracted from the population is defined as a frequency vector with different frequencies. The frequency difference number vector is a vector indicating the number of differences for each frequency. The frequency indicates how many pieces of data have the same value. Therefore, it can be said that the number vector different in frequency indicates how many types of data have the same value.

  Here, the frequency vector of the population is unknown, but the frequency vector of the sample is unknown.

  Referring to FIG. 1, the different number counting apparatus 10 includes a matrix generation unit 11 and a different number calculation unit 12.

  Based on the size of the population and the extraction rate for extracting a sample from the population, the matrix generation unit 11 extracts the frequency difference number vector indicating the number of differences for each frequency of the population and the extraction rate from the population. A matrix is generated that defines the relationship with the frequency-variable number vector indicating the number of different samples for each frequency by binomial probabilities. This matrix will be described in detail later in stages.

  The different number calculation unit 12 calculates an estimated value of the number of different populations based on the matrix generated by the matrix generation unit 11 and the sample extracted from the population at the extraction rate.

  According to this embodiment, since the number of different populations can be estimated using a matrix calculated in advance and a sample extracted from the population, the number can be efficiently estimated without analyzing all the elements. can do.

  Hereinafter, more specific operations in each unit will be described.

  The above matrix will be referred to as a logarithmic compression binomial probability matrix in the sense that the binomial probability matrix is logarithmically compressed. The binomial probability matrix refers to a matrix in which probabilities are arranged as elements so that the product of the population frequency vector and the number vector differs from the sample frequency vector. The logarithmic compression binomial probability matrix is a compression matrix obtained by compressing the binomial probability matrix in the row direction and the column direction.

  Then, the different number calculation unit 12 compresses the frequency vector of the sample extracted from the population, obtains a logarithm compression frequency difference number vector of the sample, calculates the logarithm compression frequency difference number vector of the sample, and logarithm compression A number vector different from the estimated logarithmic compression degree of the population is calculated by the product of the binomial probability matrix and the inverse matrix. Furthermore, the different number calculation unit 12 sets the sum of the elements of the logarithmic compression degree different number vector estimated for the population as an estimated value of the number of different populations. The logarithmically compressed frequency difference number vector is a vector obtained by compressing the frequency difference number vector so as to fit the logarithm compression binary probability matrix.

  In this way, using a logarithm-compressed binary probability matrix that compresses the binomial probability matrix indicating the relationship between the frequency vector of the population and the frequency vector of the sample group in the row and column directions, Since the number is calculated, the number can be calculated efficiently with a small amount of processing.

  At that time, the matrix generation unit 11 increases the number of rows in the binomial probability matrix to be combined into one element of the logarithm compression binomial probability matrix as the row number increases, and similarly, one element of the logarithm compression binomial probability matrix The number of columns in the binomial probability matrix summarized in (1) is compressed so as to increase as the column number increases. The binomial probability matrix, which changes slowly as the row number and column number increase, is combined into one element in large units as the row number and column number increase, greatly reducing the amount of processing while suppressing deterioration in accuracy. Is possible.

  More preferably, the matrix generation unit 11 increases the number of rows in the binomial probability matrix combined into one element of the logarithmic compression binomial probability matrix exponentially as the row number increases, and the logarithm compression binomial probability matrix It is preferable to compress the number of columns in the binomial probability matrix to be integrated into a single element so as to increase exponentially as the column number increases. For example, the compression may be performed by dividing the element with a power of 2 as a break. Since the binomial probability matrix whose logarithmic change gradually increases as the row number or column number increases, it is combined into one element in an exponentially large unit as the row number or column number increases. It is possible to greatly reduce the processing amount while suppressing.

  A basic operation flow of the number counting apparatus according to this embodiment will be described with reference to a flowchart.

  FIG. 2 is a flowchart showing the operation of the number counting device, which is different from the present embodiment. Referring to FIG. 2, the different number counting apparatus 10 first generates a matrix based on the binomial probability as described above by the matrix generation unit 11 (step 101). Subsequently, the different number counting device 10 calculates the number of different populations based on the matrix and the sample.

  Next, a specific calculation process performed by the number counting device according to the present embodiment will be described.

  First, how to obtain a logarithmically compressed binomial probability matrix will be described.

  A frequency vector U of a population of size N is expressed by equation (1), and a frequency vector V of a population composed of M samples is expressed by equation (2).

When a source is extracted from a population at a predetermined extraction rate r, an Nth-order matrix P whose element is the probability that k elements having the same n in the population are extracted as samples is a binomial probability matrix It is. This binomial probability matrix is a matrix that represents the occurrence probability for each frequency of the sample for each frequency of the population. The binomial probability matrix can be expressed by equation (3). If i <j, then P _ij = 0.

  These U, V, and P are in the relationship of Expression (4). When V is given, U can be estimated by Expression (5).

If the population size N is enormous as it is, the binomial probability matrix P becomes enormous and it is difficult to obtain the inverse matrix P- ¹ . Therefore, a logarithmically compressed binomial probability matrix LP obtained by compressing the binomial probability matrix P by logarithmization is introduced. The logarithm-compressed binomial probability matrix LP adds the probability values to the binomial probability matrix P by dividing the columns in the column direction (power of 2) and delimiting the rows in the row direction (power of 2 ÷ extraction rate). It is a matrix that averages the probability values. The operation for compressing the binomial probability matrix into a logarithmic compression binary probability matrix may be an operation for obtaining a weighted average in the row direction and obtaining a total value in the column direction. What is necessary is just to set the weight of each element suitably. Or you may give a uniform weight to all the elements.

  The elements in the binomial probability matrix that are grouped and grouped into one are grouped in a group with a power of 2 as a break, and the number increases exponentially as it proceeds in the row and column directions.

  The logarithmic compression binomial probability matrix LP can be expressed by Equation (6).

  Subsequently, approximate calculation of the logarithmically compressed binomial probability matrix will be described.

LP _ij shown in Expression (6) includes a weighted sum of binomial probabilities.

If n∈ {1, 2,...}, r∈ (0,1) and X _{r, n} is a random variable according to the binomial distribution Bin (n, r), the weighted sum of binomial probabilities is It can be expressed as in formula (A-1).

Here, the weight f (n | β, n ₁ , n ₂ ) has a parameter β ≧ 1, and the zeta distribution cut on the set of natural numbers {n ₁ , n ₁ +1,..., N ₂ }. Is a probability function. That is, the weight f (n | β, n ₁ , n ₂ ) can be expressed as in equation (A-2).

  In this case, the normalization constant is represented by the formula (A-3).

  The approximate calculation for calculating the logarithm-compressed binomial probability matrix in the present embodiment at high speed is an integral approximation of the sum of the weighted binomial probabilities shown in the above formula (A-1) with high accuracy. Unlike the calculation of the sum of discrete values, the value of integration is determined by the values of the two end points, so that the amount of calculation is reduced.

  The arithmetic process for calculating the sum includes a process for obtaining the sum of the vertical direction (n direction) and the horizontal direction (k direction) of the matrix, and integral approximation suitable for each direction is applied.

  The matrix generation unit 11 applies an approximation method suitable for each of the horizontal direction (direction n) and the vertical direction (direction k). In the horizontal direction, an approximation using the well-known central limit theorem and continuous correction is applied, and in the vertical direction, the step function included in the equation (A-1) is approximated by a continuous curve and converted to integral. An approximate expression is applied. In the approximate expression, a continuous curve that approximates the step function is expressed by parameter display. The approximate expression includes the line integral of the continuous curve.

(Approximation in the horizontal direction (direction of k))

  The above formula (A-1) can be modified as the following formulas (A-4) and (A-5).

Note that, when the equation (A-4) is approximated by the equation (A-5), the probability distribution of the binomial random variable X _{r, n} is the average nr, variance nr ( 1-r) can be approximated by a normal distribution. This is the application of the central limit theorem.

  Further, the lower end is shifted by −0.5 and the upper end is shifted by +0.5 in the integration interval of the formula (A-5). This is called continuous correction. By using continuous correction, the approximation accuracy can be improved when a binomial distribution (binary probability distribution) is approximated by a normal distribution.

(Approximation in the vertical direction (direction of n))

  The above formula (A-5) can be further transformed into formula (A-6).

Here, χ (c ₁ , c ₂ ] (x) is a defining function on the section (c ₁ , c ₂ ] shown in the equation (A-7).

  Further, the parameter t is introduced to define the expressions (A-8) and (A-9).

  The parts of the formula (A-10) and the formula (A-11) in the formula (A-6) are sums of definition functions, that is, step functions.

  This step function is a typical discontinuous function. If this step function is calculated as it is, Equation (A-6) calculates the sum of a huge number of terms. Therefore, in the present embodiment, these formulas (A-10) and (A-11) are approximated by curves.

For example, when the behavior of the equation (A-10) is examined, b (n ₂ ) <b (n ₂ +1) <... <B (n ₁ ) is a jump point, and each jump point b (n) is n. It can be seen that it is a step function with a jump amount of ^-β . FIG. 17 is a diagram showing a step function and its approximate curve. In FIG. 17, the entire expression (A-10) is y, and b (n) in the expression (A-10) is x. The step function of the formula (A11) can also be expressed in the same manner as the formula (A-10).

  Therefore, if a parameter-displayed curve that matches the behavior of the step function is used, an appropriate approximation to the step function can be given. An example of approximation by such a parameter display curve is shown in Expression (A-12).

  An approximate curve according to this equation (A-12) is shown in FIG. It can be seen from FIG. 17 that the approximation is satisfactorily performed. A good approximation can be similarly applied to formula (A-11).

As for the formula (A-3) constituting the weight by the cut zeta distribution, approximation by integration is performed as shown in the formula (A-13).

  Expression (A-12) with respect to Expression (A-10), approximation expression with respect to Expression (A-11), and approximation of the weight of the cut zeta distribution (A-13) are expressed as Expression (A-6). substitute. At this time, since the approximate curve is a parameter display, the calculation is performed as a line integral. As a result, an expression (partially extracted from Expression (A-14)) that is the basis of approximate calculation is obtained.

For this formula (A-14), n ₁ = [2 ⁱ⁻² ] r ⁻¹ +1, n ₂ = 2 ⁱ⁻¹ r ⁻¹ , k ₁ = [2 ^j−2 ] +1, k ₂ = When 2 ^j−1 is substituted, an approximate value of the weighted sum of binomial probabilities in Equation (6) can be calculated. By dividing the obtained approximate value by (2 ^i-1 − [2 ⁱ⁻² ]) r ⁻¹ , the approximate value of LP _{ij in} Expression (6) is obtained.

  The approximate calculation of the logarithm compression binomial probability matrix as described above is possible.

  Further, the number vector V different in frequency is also compressed by logarithmization in the same manner as the logarithmic compression binary probability matrix LP. The compressed frequency difference number vector (logarithm compression frequency difference number vector) LV can be expressed by equation (7). A sample obtained by dividing the number vector of the sample with different frequency vectors (power of 2) and summing the values is called a number vector with different logarithmic compression frequency of the sample. A frequency vector of a population having a sample with an extraction rate r is referred to as a power vector of the logarithm compression frequency difference of the population, and the sum of values obtained by dividing the number vector (power of 2 ÷ extraction rate).

Then, the estimated value of U can be obtained by Expression (8) using the inverse matrix LP ⁻¹ of LP.

Through the arithmetic processing as described above, the matrix generation unit 11 generates the inverse matrix LP ⁻¹ of the logarithmic compression binomial probability matrix LP shown in the equation (6), and the different number calculation unit 12 uses the equation (8) to calculate the mother matrix. The estimated value of the number vector U with different frequency of the group is calculated.

  Here, in order to facilitate understanding of the calculation and its concept, a description is given of obtaining the binomial probability matrix P and compressing it to generate the logarithmically compressed binomial probability matrix LP. However, actually, the matrix generation unit 11 does not obtain the binomial probability matrix P once, but directly generates the logarithm compression binomial probability matrix LP.

  FIG. 3 is a block diagram illustrating an example of the data analysis apparatus 20 in which the number counting apparatus 10 according to the present embodiment is incorporated. Referring to FIG. 3, the data analysis apparatus 20 includes a storage unit 21, a size calculation unit 22, a sample extraction unit 23, and a data analysis unit 24 in addition to the same matrix generation unit 11 and different number calculation unit 12 as those in FIG. 1. Have.

  The storage unit 21 stores a population data group. As an example, a data group of access logs in which access data for a certain server is accumulated is defined as a population. FIG. 4 is a diagram schematically illustrating an example of an access log. This access log stores access data to a certain server in time series. In the access log, various items including an access number (No.), an originating IP address, and a user ID are stored for each access. The access number is a serial number assigned to each access in the order in which access to the server occurs. The source IP address is the IP address of the access source, and is used for identifying the user of the access source. The usage time is the time during which the use of browsing the site by the access has been continued.

  The size calculation unit 22 calculates the size of the population data group accumulated in the storage unit 21, that is, the original number, and notifies the matrix generation unit 11 and the sample extraction unit 23. For example, the size calculation unit 22 has a log counter, and counts the size of the data group by counting up the log counter each time information on one access is stored in the storage unit 21. Also good. Alternatively, the size calculation unit 22 may calculate the size of the access log from the size of the effective area of the access log in the storage unit 21 when information on the size of the population data group is needed. .

Sampling unit 23, the size of the population that has been notified by the size calculating unit 22, and based on the a given extraction rate, and calculates the number of samples to be extracted from the population, the sample of the number Extracted from the data group of the population in the storage unit 21. For example, if the size of the population is multiplied by the extraction rate, the number of samples to be extracted from the population can be obtained. The extracted sample is notified to the number calculation unit 12 differently.

The matrix generation unit 11 generates an inverse matrix LP ⁻¹ of the logarithmic compression binomial probability matrix LP based on the given extraction rate and the size notified from the size calculation unit 22, and the difference calculation unit 12 Notice. Specifically, the number of rows and the number of columns of the binomial probability matrix P are determined by the notified size. For example, the minimum number of rows and the number of columns that are not less than the notified size and that complete the compression unit in logarithmic compression may be used. By Equation (3), each element of the binomial probability matrix P is determined from the extraction rate. Furthermore, the logarithm compression binomial probability matrix LP is determined from each element of the binomial probability matrix P by Expression (6).

The difference number calculation unit 12 calculates an estimated value of the difference number U of the population based on the inverse matrix LP ^-1 notified from the matrix generation unit 11 and the sample group extracted by the sample extraction unit 23. By analyzing the sample group, the number vector V can be calculated differently. According to Expression (7), the number vector V different in frequency can be logarithmically compressed to obtain a number vector LV different in logarithmic compression frequency. Then, the estimated value of the number U of different populations can be calculated from the logarithm-compressed binomial probability matrix LP and the inverse matrix LP ^{−1 of} the logarithmic-compressed binomial probability matrix LV by Expression (8). The calculated estimated value of the number U of different populations is notified to the data analysis unit 24. For example, the different number calculation unit 12 records the estimated value of the calculated different number U of the population in the storage unit 21, and the data analysis unit 24 reads the estimated value of the different number U from the storage unit 21.

  The data analysis unit 24 analyzes the data group accumulated in the storage unit 21 using the estimated value of the different number calculated by the different number calculation unit 12. For example, the sorting process can be speeded up by using different numbers in sorting of data groups. For example, the data analysis unit 24 may generate access analysis result data from, for example, the access log illustrated in FIG. FIG. 5 is a diagram schematically illustrating an example of access analysis result data. The access analysis result data includes data of items such as the user ID, the number of accesses, and the average usage time for each user. The number of different users indicates the number of users, and it is necessary to secure storage areas for the number of users. For example, the data analysis unit 24 may secure storage areas for the number of users in advance and calculate and record data for each item for each user.

  Next, more specific examples in the above-described embodiment will be described.

  Unlike the present embodiment, the basic configuration of the number counting apparatus is the same as that shown in FIG. 1, and the basic operation is the same as that shown in FIG. Hereinafter, each detailed process in a present Example is demonstrated.

  FIG. 6 is a diagram illustrating how a sample is extracted from a population. Here, the sample extraction rate is 50%. Elements written in large letters in the figure are extracted as samples.

Frequency is large listen lever, the number of different in different number and the sample in the population is almost unchanged. In FIG. 6A, the number of different populations and the number of different samples are both 4, which are not changed. On the other hand, when the frequency is small, the number of differences in the sample decreases from the number of differences in the population. In FIG. 6B, the number of different populations is 4, and the number of different samples is 2.

  The probability that k elements are extracted from the element of frequency n can be represented by the binomial probability of Expression (9).

Here, the different number in which the frequency of the element is _f will be referred to as f−frequency different number U _f .

FIG. 7 is a diagram for explaining the number U _f different from f−frequency. Referring to FIG. 7, since only the frequency of A is 10, the number U ₁₀ = 1 differs from 10−frequency. Further, since the frequency of B and C is 3, the number U ₃ = 2 is different from the 3-frequency. Since the frequencies of D, E, and F are 2, the number U ₂ = 3, which is different from the 2-frequency. Since the frequencies of G, H, I, J, and K are 1, 5 is different and the number U ₁ = 5. The total number of differences is 11 because it is the sum of all f-frequency differences.

  The frequency difference number vector described above in Expression (1), Expression (2), and the like is a vector having f-frequency difference numbers as elements for each frequency.

The expected value V _k ^ of the k-frequency difference number of the sample can be expressed as in Expression (10) using the n-frequency difference number U _n of the population and the binomial probability of Expression (9). it can.

The frequency difference number vector U = (U ₁ ,..., U _N ) of the population of size N and the sample frequency difference number vector V = (V ₁ ,..., V _N ) are as described above. Are shown in equations (1) and (2), respectively.

  Here, when the binomial probability matrix P, which is an Nth-order matrix having binomial probabilities as elements, is used as the expression (3), the expression (4) is established. Therefore, the estimated value U ^ of the number U of different samples is obtained by the equation (5).

  FIG. 8 is a diagram showing the principle of approximation by a binomial probability matrix using a toy model.

As described above, according to the embodiments and examples of the present invention, the frequency difference vector of the population is approximately calculated from the frequency vector of the sample using the binomial probability matrix. The binomial probability matrix are those not subjected to compression.

  Referring to FIG. 8, a sample is extracted from a population of size 222 at an extraction rate of 50%.

  Looking at the frequency vector of the population, 1-frequency difference number = 121, 2-frequency difference number = 27, 3-frequency difference number = 10, 4-frequency difference number = 3. , 5-difference number = 1. The total number of differences in the entire population is 162. In practice, however, the number of different populations is unknown.

Looking at the frequency difference number vector of the actually extracted sample, 1-frequency difference number = 87, 2-frequency difference number = 10, 3-frequency difference number = 3, 4-frequency difference number = 0, and the number is different from 5−number = 0. The total number of samples is 100. Each different number of this specimen is known.

  The estimated value of the frequency difference vector of the population calculated as the product of the frequency vector of this sample and the inverse matrix of the binomial probability matrix (uncompressed) is 1−frequency difference number = 152 And 2-frequency difference number = 4, 3-frequency difference number = 24, 4-frequency difference number = 0, and 5-frequency difference number = 0. The estimated value of the number of differences in the entire population is 180, and it can be seen that the number of differences can be estimated within a certain amount of error.

  FIG. 9 is a diagram illustrating how the binomial probability matrix is compressed.

  The right side of FIG. 9 shows a part of the uncompressed binomial probability matrix, and the left side shows a logarithmically compressed binary probability matrix obtained by compressing the binomial probability matrix.

  The binomial probability matrix on the right is a matrix in which binomial probabilities are arranged as elements so that the product of the population frequency vector and the number vector differs from the sample frequency vector. A logarithm-compressed binary probability matrix on the left side is obtained by compressing the binomial probability matrix in the row direction and the column direction.

  A plurality of elements of the binomial probability matrix are combined into one element in the logarithm-compressed binomial probability example. A state in which a plurality of elements of the binomial probability matrix are combined into one element of the logarithmic compression binomial probability matrix is indicated by a broken line arrow in FIG.

  The number of rows in the binomial probability matrix combined into one element of the logarithmic compression binomial probability matrix increases exponentially as the row number increases, and is combined into one element of the logarithmic compression binomial probability matrix. The number of columns in is compressed so as to increase exponentially as the column number increases. As the number of elements to be collected increases, the effect of reducing the amount of processing increases.

  FIG. 10 is a diagram illustrating the distribution of elements in the binomial probability matrix. In FIG. 10, the change in the value of the element indicated by the binomial probability is indicated by a mountain-shaped curve. From this, it can be seen that, in the binomial probability matrix, the logarithmic change gradually decreases as the row number or column number increases. As described above, a binomial probability matrix whose logarithmically changes gradually as the row number or the column number increases, in this embodiment, one element in an exponentially large unit as the row number or the column number increases. Therefore, it is possible to greatly reduce the processing amount while suppressing deterioration in accuracy.

  FIG. 11 is a diagram showing a state in which a number vector is compressed differently.

  Referring to FIG. 11, two elements of frequencies 3 to 4 in the population frequency difference number vector are compressed into one element in the population logarithm compression frequency difference number vector. The compression here is an operation of adding the number different from 3-frequency and the number different from 4-frequency into one element.

Similarly, four elements of frequencies 5 to 8 in the number vector different in frequency of the sample are compressed into one element in the number vector different in logarithmic compression frequency of the sample . The compression is an operation of adding the number of 5- frequency differences to the number of 8-frequency differences into one element.

  FIG. 12 is a diagram showing the principle of approximation using a logarithmic compression binomial probability matrix.

  In the present embodiment, the frequency vector of the population frequency difference is approximately calculated from the frequency vector of the sample frequency using the compressed binomial probability matrix. In this example, the extraction rate is 1/256. Further, unlike FIG. 8, in this example, the compression of the number vector differs from the matrix and the frequency.

  Referring to FIG. 12, a sample is extracted from a population of size = 7524682 at an extraction rate of 1/256.

  Looking at the logarithm compression frequency difference number vector of the population, 256-frequency difference number = 1708, 512-frequency difference number = 454, 1024-frequency difference number = 64, 2048-frequency difference number = 21. It is. The total number of differences in the entire population is 2248. In practice, however, the number of different populations is unknown.

Looking at the logarithm compression frequency difference number vector of the actually extracted sample, 1-frequency difference number = 480, 2-frequency difference number = 213, 4-frequency difference number = 134, 8-frequency The difference number = 34. The total number of samples is 863 . Each different number of this specimen is known.

And the frequency depends number vector of the sample, is calculated as the product of the inverse matrix of binomial probability matrix (having been subjected to the compression), the estimated value of the frequency differs number vector of the population, 1-degree different number = 1345 And 2-frequency difference number = 309 , 3-frequency difference number = 132 , and 4-frequency difference number = 3 . The estimated value of the number of differences in the entire population is 1790, and it can be seen that the number of differences can be estimated within a certain amount of error.

Hereinafter, it will be shown by experimental results how the estimation accuracy of different numbers is different in the number counting using data.
(Estimation experiment 1)

  An experimental result of the estimation accuracy of the number of differences in the purchase transaction data recorded in the product purchase transaction will be described.

  As a population, 10 million purchase history data having 10 kinds of data items were used. That is, the purchase transaction data includes various variables such as a transaction date, a transaction store, and a transaction amount. Here, ten variables (data items) are selected. Moreover, calculation was performed about 1/4, 1/16, 1/64, 1/256, and 1/1024 as extraction rates.

  FIG. 13 is a diagram illustrating a result of comparing the estimation accuracy by changing the extraction rate in the estimation experiment 1. As a variable, one variable v5 is fixedly used. The true value indicates an estimated value for the true value.

  As can be seen from FIG. 13, when the extraction rate decreases, the estimation accuracy decreases. However, if the extraction rate is about 1/64, an estimation accuracy of 65% is obtained.

FIG. 14 is a diagram illustrating a result of comparing estimation accuracy by changing a variable and an extraction rate in the estimation experiment 1. FIG. 15 is a graph showing the true value in the result of FIG. In FIG. 15, a plot in which the estimation error is within 35% is surrounded by a broken line. As can be seen from this, it can be seen that the estimation error is within 35% at the extraction rate of 1/65 or more in 8 of the 10 variables used in the experiment.
(Estimation experiment 2)

  The experimental results of estimating the number of surnames of households nationwide from the surname data of households with a sampling rate of 1/16 will be described.

  According to the national census, the number of households nationwide is about 50 million. Last name data of 3123979 households was used as a sample with an extraction rate of 1/16. However, random sampling is not performed.

  FIG. 16 is a diagram illustrating an estimation experiment result in the estimation experiment 2. The number of different populations calculated from samples with 70506 different numbers is 131265.

  There are various theories of surnames from dozens of thousands to 300,000. The breadth of this width is greatly affected by the difference between the new and old characters. Therefore, when the last name using the old letter and the last letter using the new letter are added together, it may be the smaller value in the width.

  In this estimation experiment, the surnames using the old letters are added to the surnames using the new letters. Therefore, it can be said that the estimated value of 131265 matches well with the number of surnames of several ten thousand.

  The embodiments and examples of the present invention described above are examples for explaining the present invention, and are not intended to limit the scope of the present invention to only those embodiments or examples. Those skilled in the art can implement the present invention in various other modes without departing from the gist of the present invention.

  In addition, the devices of the above-described embodiments and examples can be realized by causing a computer to execute a software program that defines the processing procedure of each unit constituting the device. In that case, notification of data between the respective units may be performed via a data storage area such as a memory or a register. The data to be notified may be stored in the data storage area in the sending process, and the data may be read from the data storage area in the receiving process.

DESCRIPTION OF SYMBOLS 10 ... Different number counting apparatus, 11 ... Matrix generation part, 12 ... Different number calculation part, 20 ... Data analysis apparatus, 21 ... Memory | storage part, 22 ... Size calculation part, 23 ... Sample extraction part, 24 ... Data analysis part

Claims (7)

A different number counting method for counting the number of different populations of data groups,
Frequency indicating the number of differences for each frequency of the population, which is obtained by the formula (C-1) based on the size N of the population and the extraction rate r for extracting a sample from the population. Using each element of the binomial probability matrix P that defines the relationship between the different number vector and the frequency different number vector indicating the number of different samples for each frequency extracted from the population at the extraction rate r, the expression ( A logarithm compression binomial probability matrix LP which is a compression matrix obtained by compressing the binomial probability matrix P in the row direction and the column direction by an approximate expression obtained by integral approximation of the operation for obtaining the sum of discrete values in C-2), Generating an inverse matrix LP ⁻¹ of a logarithmically compressed binomial probability matrix LP;
A different number calculation means calculates the logarithmic compression of the population by the product of the logarithm compression frequency difference number vector obtained by compressing the frequency difference number vector of the sample extracted from the population and the inverse matrix LP- ^1. Calculating a frequency difference number vector, and calculating a sum of elements of the logarithm compression frequency difference number vector estimated for the population as an estimated value of the number of different numbers of the population.

The approximate expression is obtained by approximating the step function in the direction of n included in Expression (C-3) representing the weighted sum of binomial probabilities included in Expression (C-2) with a continuous curve. The different number counting method according to claim 1, which is an expression converted into an integral.

  The difference number counting method according to claim 2, wherein the approximate expression is an expression in which the continuous curve that approximates the step function is expressed by parameter display.   The method according to claim 3, wherein the approximate expression is an expression including a line integral of the continuous curve. The approximate expression is the sum of discrete values in the k direction included in the expression (C-3) representing the weighted sum of binomial probabilities included in the expression (C-2), and the central limit theorem. 5. The different number counting method according to any one of claims 1 to 4, which is an expression approximated by correction and converted into integral.

A different number counting device for counting the number of different populations of data groups,
Based on the size N of the population and the extraction rate r for extracting a sample from the population, the frequency difference number vector indicating the number of differences for each frequency of the population, which is obtained by Expression (C-1), Using each element of the binomial probability matrix P that defines the relationship with the frequency difference number vector indicating the number of differences for each frequency of the samples extracted from the population at the extraction rate r, in Equation (C-2) A logarithm compression binomial probability matrix LP, which is a compression matrix obtained by compressing the binomial probability matrix P in the row direction and the column direction, is obtained by an approximate expression obtained by integral approximation of an operation for obtaining a sum of discrete values, and the logarithm compression binomial probability Matrix generating means for generating an inverse matrix LP ⁻¹ of the matrix LP;
A logarithm compression frequency difference number vector obtained by compressing the frequency difference number vector of the sample extracted from the population and a product of the inverse matrix LP ⁻¹ to calculate an estimated logarithmic compression frequency difference number vector of the population And a different number calculating means for calculating a sum of the elements of the estimated logarithmic compression frequency different number vector of the population as an estimated value of the different number of the population.

A different number counting program for counting the number of different population consisting of data groups,
Computer
Based on the size N of the population and the extraction rate r for extracting a sample from the population, the frequency difference number vector indicating the number of differences for each frequency of the population, which is obtained by Expression (C-1), Using each element of the binomial probability matrix P that defines the relationship with the frequency difference number vector indicating the number of differences for each frequency of the samples extracted from the population at the extraction rate r, in Equation (C-2) A logarithm compression binomial probability matrix LP, which is a compression matrix obtained by compressing the binomial probability matrix P in the row direction and the column direction, is obtained by an approximate expression obtained by integral approximation of an operation for obtaining a sum of discrete values, and the logarithm compression binomial probability Matrix generating means for generating an inverse matrix LP ⁻¹ of the matrix LP;
A logarithm compression frequency difference number vector obtained by compressing the frequency difference number vector of the sample extracted from the population and a product of the inverse matrix LP ⁻¹ to calculate an estimated logarithmic compression frequency difference number vector of the population And a different number counting program for functioning as a different number calculating means for calculating a sum of elements of different number vectors of logarithmic compression degrees estimated for the population as an estimated value of the different number of the population.

Images