JP5925043B2 - Different number counting method, apparatus, and program - Google Patents

Description

  The present invention relates to a technique for estimating the number of different data groups having an enormous number of elements.

  In information processing, it is required to process a data group having an enormous number of elements efficiently and in a short time. In many cases, a data group used for information processing includes a plurality of elements of the same type. There is a technique for improving the efficiency of data group processing by utilizing the fact that the same type of elements are included in the data group. That is, the number of types of elements included in the data group, that is, the number of different types may be used for information processing. The number of different types of data or data items of a finite data set is called the number of different data or data items.

  For example, in data sorting in a database in which an enormous amount of data is accumulated, it is known that the number of different elements included in the database is extracted and the sorting process can be accelerated by using the different numbers. Further, when storing data in the database, the entire data amount can be compressed by recording the type and frequency of the elements instead of storing all the data in individual areas. Even in that case, it is required to count the number of types of elements in advance. For the values of data items in a finite data set, the appearance frequency is called the frequency of the values.

  Further, if the number of users who have accessed a certain site can be extracted from an access log in which a huge access history to the site is accumulated, it can be used for access analysis. In that case, the number of different access source addresses included in the access log indicates the number of users. In a large-scale Internet site, in order to protect the server from various external attacks and access loads exceeding the allowable range, it is necessary to constantly monitor the access status from the outside. Measuring the number of unique access hosts from a large number of access logs is part of that. That is, the number of different host identification information such as IP addresses is counted from the access log data. In a large site, the number of accesses per day exceeds one billion, and a large amount of computer resources are required to grasp the statistics of all cases at all times.

  In creating translation software and dictionary databases, the number of words is extracted from a huge corpus and used to construct the database. In that case, the number of different corpora is the number of words.

  In addition, in address management, the number of surnames of residents can be extracted from the accumulated information of residents and used for management of residents data. In that case, the number of different surnames in the resident information indicates the number of surnames.

  As described above, counting the number of different data or data items is very useful in data processing. However, if the data group is enormous, it is different and enormous amount of processing and storage capacity are required for counting. Therefore, a technique for efficiently counting the number of different data groups is required.

  In response to such a requirement, Patent Document 1 discloses a technique for estimating the number of different data groups with high accuracy. The technique disclosed in Patent Document 1 attempts to estimate the number of differences with high accuracy by using a counting method suitable for counting small different numbers and a counting method suitable for counting large different numbers.

JP 2008-59293 A

  The technique of Patent Document 1 assumes that different numbers of various data groups cannot be counted with high accuracy by a single different number counting method, and estimates the different numbers of the same data group by two counting methods. . However, redundant processing is performed for that purpose, and processing efficiency is not good.

  An object of the present invention is to provide a technique for efficiently estimating the number of different data groups.

  A different number counting method according to an aspect of the present invention is a different number counting method for counting a different number of populations composed of data groups, wherein the matrix generation means includes a size of the population, and the population. Based on the sampling rate at which the sample is extracted, a frequency vector indicating the number of differences for each frequency of the population, and a frequency indicating the number of differences for each frequency of samples extracted from the population at the extraction rate A step of generating a matrix defining a relationship with a different number vector, and a different number calculating means based on the matrix generated by the matrix generating means and the sample extracted from the population at the extraction rate And calculating an estimated value of the number of different populations.

  According to the present invention, since the number of different populations can be estimated using a matrix calculated in advance and a sample extracted from the population, the number can be efficiently estimated without analyzing all the elements. be able to.

It is a block diagram of a different number counting device by this embodiment. It is a flowchart which shows operation | movement of the number counting apparatus which differs in this embodiment. It is a block diagram which shows an example of the data analysis apparatus 20 incorporating the number counting apparatus 10 different from this embodiment. It is the figure which showed an example of the access log typically. It is the figure which showed an example of the access analysis result data typically. It is a figure which shows a mode that a sample is extracted from a population. It is a diagram for explaining f- frequency different number U _f. It is the figure which showed the principle of the approximation by a binomial probability matrix with the toy model. It is a figure which shows a mode that a binomial probability matrix is compressed. It is a figure which shows the mode of the distribution of the element in a binomial probability matrix. It is a figure which shows a mode that a frequency vector differs in frequency. It is the figure which showed the principle of the approximation by a logarithm compression binomial probability matrix. It is a figure which shows the result of having changed the extraction rate in the estimation experiment 1, and comparing the estimation precision. It is a figure which shows the result of having changed the variable and the extraction rate in the estimation experiment 1, and comparing the estimation precision. FIG. 15 is a graph showing the true value in the result of FIG. 14. It is a figure which shows the estimation experiment result in the estimation experiment 2. FIG.

  A basic embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram of a different number counting device according to this embodiment. The different number counting device 10 of this embodiment is a device that counts the number of different populations composed of a huge data group.

  The population is, for example, an enormous amount of data stored in a database or a set based on the value of each item when the data is composed of a plurality of items. The number of data stored in the database is the size of the population. Since the population may contain data having the same value, the number of different populations is equal to or smaller than the size of the population.

  The different number counting apparatus 10 of the present embodiment uses an approximation for counting different numbers. The approximation method estimates the number of different populations from a sample group extracted from the population at a predetermined extraction rate. The extraction rate is given in advance.

  Here, a data vector such as a population and a sample group extracted from the population is defined as a frequency vector with different frequencies. The frequency difference number vector is a vector indicating the number of differences for each frequency. The frequency indicates how many pieces of data have the same value. Therefore, it can be said that the number vector different in frequency indicates how many types of data have the same value.

  Here, the frequency vector of the population is unknown, but the frequency vector of the sample is unknown.

  Referring to FIG. 1, the different number counting apparatus 10 includes a matrix generation unit 11 and a different number calculation unit 12.

  Based on the size of the population and the extraction rate for extracting a sample from the population, the matrix generation unit 11 extracts the frequency difference number vector indicating the number of differences for each frequency of the population and the extraction rate from the population. A matrix is generated that defines the relationship with the frequency-variable number vector indicating the number of different samples for each frequency by binomial probabilities. This matrix will be described in detail later in stages.

  The different number calculation unit 12 calculates an estimated value of the number of different populations based on the matrix generated by the matrix generation unit 11 and the sample extracted from the population at the extraction rate.

  According to this embodiment, since the number of different populations can be estimated using a matrix calculated in advance and a sample extracted from the population, the number can be efficiently estimated without analyzing all the elements. can do.

  Hereinafter, more specific operations in each unit will be described.

  The above matrix will be referred to as a logarithmic compression binomial probability matrix in the sense that the binomial probability matrix is logarithmically compressed. The binomial probability matrix refers to a matrix in which probabilities are arranged as elements so that the product of the population frequency vector and the number vector differs from the sample frequency vector. The logarithmic compression binomial probability matrix is a compression matrix obtained by compressing the binomial probability matrix in the row direction and the column direction.

  Then, the different number calculation unit 12 compresses the frequency vector of the sample extracted from the population, obtains a logarithm compression frequency difference number vector, obtains the logarithm compression frequency difference number vector of the sample, and the logarithm compression A number vector different from the estimated logarithmic compression degree of the population is calculated by the product of the binomial probability matrix and the inverse matrix. Furthermore, the different number calculation unit 12 sets the sum of the elements of the logarithmic compression degree different number vector estimated for the population as an estimated value of the number of different populations. The logarithmically compressed frequency difference number vector is a vector obtained by compressing the frequency difference number vector so as to fit the logarithm compression binary probability matrix.

  In this way, using a logarithm-compressed binary probability matrix that compresses the binomial probability matrix indicating the relationship between the frequency vector of the population and the frequency vector of the sample group in the row and column directions, Since the number is calculated, the number can be calculated efficiently with a small amount of processing.

  At that time, the matrix generation unit 11 increases the number of rows in the binomial probability matrix to be combined into one element of the logarithm compression binomial probability matrix as the row number increases, and similarly, one element of the logarithm compression binomial probability matrix The number of columns in the binomial probability matrix summarized in (1) is compressed so as to increase as the column number increases. The binomial probability matrix, which changes slowly as the row number and column number increase, is combined into one element in large units as the row number and column number increase, greatly reducing the amount of processing while suppressing deterioration in accuracy. Is possible.

  More preferably, the matrix generation unit 11 increases the number of rows in the binomial probability matrix combined into one element of the logarithmic compression binomial probability matrix exponentially as the row number increases, and the logarithm compression binomial probability matrix It is preferable to compress the number of columns in the binomial probability matrix to be integrated into a single element so as to increase exponentially as the column number increases. For example, the compression may be performed by dividing the element with a power of 2 as a break. Since the binomial probability matrix whose logarithmic change gradually increases as the row number or column number increases, it is combined into one element in an exponentially large unit as the row number or column number increases. It is possible to greatly reduce the processing amount while suppressing.

  A basic operation flow of the number counting apparatus according to this embodiment will be described with reference to a flowchart.

  FIG. 2 is a flowchart showing the operation of the number counting device, which is different from the present embodiment. Referring to FIG. 2, the different number counting apparatus 10 first generates a matrix based on the binomial probability as described above by the matrix generation unit 11 (step 101). Subsequently, the different number counting device 10 calculates the number of different populations based on the matrix and the sample.

  Next, a specific calculation process performed by the number counting device according to the present embodiment will be described.

  A frequency vector U of a population of size N is expressed by equation (1), and a frequency vector V of a population composed of M samples is expressed by equation (2).

When a source is extracted from a population at a predetermined extraction rate r, an Nth-order matrix P whose element is the probability that k elements having the same n in the population are extracted as samples is a binomial probability matrix It is. This binomial probability matrix is a matrix that represents the occurrence probability for each frequency of the sample for each frequency of the population. The binomial probability matrix can be expressed by equation (3). If i <j, then P _ij = 0.

  These U, V, and P are in the relationship of Expression (4). When V is given, U can be estimated by Expression (5).

If the population size N is enormous as it is, the binomial probability matrix P becomes enormous and it is difficult to obtain the inverse matrix P- ¹ . Therefore, a logarithmically compressed binomial probability matrix LP obtained by compressing the binomial probability matrix P by logarithmization is introduced. The logarithm-compressed binomial probability matrix LP adds the probability values to the binomial probability matrix P by dividing the columns in the column direction (power of 2) and delimiting the rows in the row direction (power of 2 ÷ extraction rate). It is a matrix that averages the probability values. The operation for compressing the binomial probability matrix into a logarithmic compression binomial probability matrix may be an operation for obtaining a weighted average in the row direction and obtaining a total value in the column direction. What is necessary is just to set the weight of each element suitably. Or you may give a uniform weight to all the elements.

  The elements that are grouped together in the binomial probability matrix are grouped into groups that have a break up to the power of 2 in the row direction and the column direction, and the number of elements increases in the row direction and the column direction. Increase.

  The logarithmic compression binomial probability matrix LP can be expressed by Equation (6).

  Further, the number vector V different in frequency is also compressed by logarithmization in the same manner as the logarithmic compression binary probability matrix LP. The compressed frequency difference number vector (logarithm compression frequency difference number vector) LV can be expressed by equation (7). A sample obtained by dividing the number vector of the sample with different frequency vectors (power of 2) and summing the values is called a number vector with different logarithmic compression frequency of the sample. A frequency vector of a population having a sample with an extraction rate r is referred to as a power vector of the logarithm compression frequency difference of the population, and the sum of values obtained by dividing the number vector (power of 2 ÷ extraction rate).

Then, the estimated value of U can be obtained by Expression (8) using the inverse matrix LP ⁻¹ of LP.

Through the arithmetic processing as described above, the matrix generation unit 11 generates the inverse matrix LP ⁻¹ of the logarithmic compression binomial probability matrix LP shown in the equation (6), and the different number calculation unit 12 uses the equation (8) to calculate the mother matrix. The estimated value of the number vector U with different frequency of the group is calculated.

  Here, in order to facilitate understanding of the calculation and its concept, a description is given of obtaining the binomial probability matrix P and compressing it to generate the logarithmically compressed binomial probability matrix LP. However, actually, the matrix generation unit 11 does not obtain the binomial probability matrix P once, but directly generates the logarithm compression binomial probability matrix LP.

  FIG. 3 is a block diagram illustrating an example of the data analysis apparatus 20 in which the number counting apparatus 10 according to the present embodiment is incorporated. Referring to FIG. 3, the data analysis apparatus 20 includes a storage unit 21, a size calculation unit 22, a sample extraction unit 23, and a data analysis unit 24 in addition to the same matrix generation unit 11 and different number calculation unit 12 as those in FIG. 1. Have.

  The storage unit 21 stores a population data group. As an example, a data group of access logs in which access data for a certain server is accumulated is defined as a population. FIG. 4 is a diagram schematically illustrating an example of an access log. This access log stores access data to a certain server in time series. In the access log, various items including an access number (No.), an originating IP address, and a user ID are stored for each access. The access number is a serial number assigned to each access in the order in which access to the server occurs. The source IP address is the IP address of the access source, and is used for identifying the user of the access source. The usage time is the time during which the use of browsing the site by the access has been continued.

  The size calculation unit 22 calculates the size of the population data group accumulated in the storage unit 21, that is, the original number, and notifies the matrix generation unit 11 and the sample extraction unit 23. For example, the size calculation unit 22 has a log counter, and counts the size of the data group by counting up the log counter each time information on one access is stored in the storage unit 21. Also good. Alternatively, the size calculation unit 22 may calculate the size of the access log from the size of the effective area of the access log in the storage unit 21 when information on the size of the population data group is needed. .

  The sample extraction unit 23 calculates the number of samples to be extracted from the population based on the size of the population notified from the size calculation unit 22 and the given extraction rate, and stores the number of samples. Extracted from the data group of the population of the section 21. For example, if the size of the population is multiplied by the extraction rate, the number of samples to be extracted from the population can be obtained. The extracted sample is notified to the number calculation unit 12 differently.

The matrix generation unit 11 generates an inverse matrix LP ⁻¹ of the logarithmic compression binomial probability matrix LP based on the given extraction rate and the size notified from the size calculation unit 22, and the difference calculation unit 12 Notice. Specifically, the number of rows and the number of columns of the binomial probability matrix P are determined by the notified size. For example, the minimum number of rows and the number of columns that are not less than the notified size and that complete the compression unit in logarithmic compression may be used. By Equation (3), each element of the binomial probability matrix P is determined from the extraction rate. Furthermore, the logarithm compression binomial probability matrix LP is determined from each element of the binomial probability matrix P by Expression (6).

The difference number calculation unit 12 calculates an estimated value of the difference number U of the population based on the inverse matrix LP ^-1 notified from the matrix generation unit 11 and the sample group extracted by the sample extraction unit 23. By analyzing the sample group, the number vector V can be calculated differently. According to Expression (7), the number vector V different in frequency can be logarithmically compressed to obtain a number vector LV different in logarithmic compression frequency. Then, the estimated value of the number U of different populations can be calculated from the logarithm-compressed binomial probability matrix LP and the inverse matrix LP ^{−1 of} the logarithmic-compressed binomial probability matrix LV by Expression (8). The calculated estimated value of the number U of different populations is notified to the data analysis unit 24. For example, the different number calculation unit 12 records the estimated value of the calculated different number U of the population in the storage unit 21, and the data analysis unit 24 reads the estimated value of the different number U from the storage unit 21.

  The data analysis unit 24 analyzes the data group accumulated in the storage unit 21 using the estimated value of the different number calculated by the different number calculation unit 12. For example, the sorting process can be speeded up by using different numbers in sorting of data groups. For example, the data analysis unit 24 may generate access analysis result data from, for example, the access log illustrated in FIG. FIG. 5 is a diagram schematically illustrating an example of access analysis result data. The access analysis result data includes data of items such as the user ID, the number of accesses, and the average usage time for each user. The number of different users indicates the number of users, and it is necessary to secure storage areas for the number of users. For example, the data analysis unit 24 may secure storage areas for the number of users in advance and calculate and record data for each item for each user.

  Next, more specific examples in the above-described embodiment will be described.

  Unlike the present embodiment, the basic configuration of the number counting apparatus is the same as that shown in FIG. 1, and the basic operation is the same as that shown in FIG. Hereinafter, each detailed process in a present Example is demonstrated.

  FIG. 6 is a diagram illustrating how a sample is extracted from a population. Here, the sample extraction rate is 50%. Elements written in large letters in the figure are extracted as samples.

  If the frequency is large, the number of differences in the population and the number of differences in the sample are almost the same. In FIG. 6A, the number of different populations and the number of different samples are both 4, which are not changed. On the other hand, when the frequency is small, the number of differences in the sample decreases from the number of differences in the population. In FIG. 6B, the number of different populations is 4, and the number of different samples is 2.

  The probability that k elements are extracted from the element of frequency n can be represented by the binomial probability of Expression (9).

Here, the different number in which the frequency of the element is _f will be referred to as f−frequency different number U _f .

FIG. 7 is a diagram for explaining the number U _f different from f−frequency. Referring to FIG. 7, since only the frequency of A is 10, the number U ₁₀ = 1 differs from 10−frequency. Further, since the frequency of B and C is 3, the number U ₃ = 2 is different from the 3-frequency. Since the frequencies of D, E, and F are 2, the number U ₂ = 3, which is different from the 2-frequency. Since the frequencies of G, H, I, J, and K are 1, 5 is different and the number U ₁ = 5. The total number of differences is 11 because it is the sum of all f-frequency differences.

  The frequency difference number vector described above in Expression (1), Expression (2), and the like is a vector having f-frequency difference numbers as elements for each frequency.

The expected value V _k ^ of the k-frequency difference number of the sample can be expressed as in Expression (10) using the n-frequency difference number U _n of the population and the binomial probability of Expression (9). it can.

The frequency difference number vector U = (U ₁ ,..., U _N ) of the population of size N and the sample frequency difference number vector V = (V ₁ ,..., V _N ) are as described above. Are shown in equations (1) and (2), respectively.

  Here, when the binomial probability matrix P, which is an Nth-order matrix having binomial probabilities as elements, is used as the expression (3), the expression (4) is established. Therefore, the estimated value U ^ of the number U of different samples is obtained by the equation (5).

  FIG. 8 is a diagram showing the principle of approximation by a binomial probability matrix using a toy model.

  As described above, according to the embodiments and examples of the present invention, the frequency difference vector of the population is approximately calculated from the frequency vector of the sample using the binomial probability matrix. This binomial probability matrix is not compressed.

  Referring to FIG. 8, a sample is extracted from a population of size 222 at an extraction rate of 50%.

  Looking at the frequency vector of the population, 1-frequency difference number = 121, 2-frequency difference number = 27, 3-frequency difference number = 10, 4-frequency difference number = 3. , 5-difference number = 1. The total number of differences in the entire population is 162. In practice, however, the number of different populations is unknown.

  Looking at the frequency difference number vector of the actually extracted sample, 1-frequency difference number = 87, 2-frequency difference number = 10, 3-frequency difference number = 3, 4-frequency difference number = 0, and the number is different from 5−number = 0. The total number of differences in the whole population is 100. Each different number of this specimen is known.

  The estimated value of the frequency difference vector of the population calculated as the product of the frequency vector of this sample and the inverse matrix of the binomial probability matrix (uncompressed) is 1−frequency difference number = 152 And 2-frequency difference number = 4, 3-frequency difference number = 24, 4-frequency difference number = 0, and 5-frequency difference number = 0. The estimated value of the number of differences in the entire population is 180, and it can be seen that the number of differences can be estimated within a certain amount of error.

  FIG. 9 is a diagram illustrating how the binomial probability matrix is compressed.

  The right side of FIG. 9 shows a part of the uncompressed binomial probability matrix, and the left side shows a logarithmically compressed binary probability matrix obtained by compressing the binomial probability matrix.

  The binomial probability matrix on the right is a matrix in which binomial probabilities are arranged as elements so that the product of the population frequency vector and the number vector differs from the sample frequency vector. A logarithm-compressed binary probability matrix on the left side is obtained by compressing the binomial probability matrix in the row direction and the column direction.

  A plurality of elements of the binomial probability matrix are combined into one element in the logarithm-compressed binomial probability example. A state in which a plurality of elements of the binomial probability matrix are combined into one element of the logarithmic compression binomial probability matrix is indicated by a broken line arrow in FIG.

  The number of rows in the binomial probability matrix combined into one element of the logarithmic compression binomial probability matrix increases exponentially as the row number increases, and is combined into one element of the logarithmic compression binomial probability matrix. The number of columns in is compressed so as to increase exponentially as the column number increases. As the number of elements to be collected increases, the effect of reducing the amount of processing increases.

  FIG. 10 is a diagram illustrating the distribution of elements in the binomial probability matrix. In FIG. 10, the change in the value of the element indicated by the binomial probability is indicated by a mountain-shaped curve. From this, it can be seen that, in the binomial probability matrix, the logarithmic change gradually decreases as the row number or column number increases. As described above, a binomial probability matrix whose logarithmically changes gradually as the row number or the column number increases, in this embodiment, one element in an exponentially large unit as the row number or the column number increases. Therefore, it is possible to greatly reduce the processing amount while suppressing deterioration in accuracy.

  FIG. 11 is a diagram showing a state in which a number vector is compressed differently.

  Referring to FIG. 11, two elements of frequencies 3 to 4 in the population frequency difference number vector are compressed into one element in the population logarithm compression frequency difference number vector. The compression here is an operation of adding the number different from 3-frequency and the number different from 4-frequency into one element.

  Similarly, four elements of frequencies 5 to 8 in the number vector different in frequency of the sample are compressed into one element in the number vector in the logarithm compression frequency difference of the population. Compression is an operation in which the number of 53-frequency differences and the number of 8-frequency differences are summed into one element.

  FIG. 12 is a diagram showing the principle of approximation using a logarithmic compression binomial probability matrix.

  In the present embodiment, the frequency vector of the population frequency difference is approximately calculated from the frequency vector of the sample frequency using the compressed binomial probability matrix. In this example, the extraction rate is 1/256. Further, unlike FIG. 8, in this example, the compression of the number vector differs from the matrix and the frequency.

  Referring to FIG. 12, a sample is extracted from a population of size = 7524682 at an extraction rate of 1/256.

  Looking at the logarithm compression frequency difference number vector of the population, 256-frequency difference number = 1708, 512-frequency difference number = 454, 1024-frequency difference number = 64, 2048-frequency difference number = 21. It is. The total number of differences in the entire population is 2248. In practice, however, the number of different populations is unknown.

  Looking at the logarithm compression frequency difference number vector of the actually extracted sample, 1-frequency difference number = 480, 2-frequency difference number = 213, 4-frequency difference number = 134, 8-frequency The difference number = 34. The total number of differences in the whole population is 100. Each different number of this specimen is known.

  The estimated value of the frequency difference vector of the population calculated as the product of the frequency vector of this sample and the inverse matrix of the binomial probability matrix (uncompressed) is 1−frequency difference number = 152 The 2-frequency difference number = 4, the 3-frequency difference number = 24, and the 4-frequency difference number = 0. The estimated value of the number of differences in the entire population is 1790, and it can be seen that the number of differences can be estimated within a certain amount of error.

Hereinafter, it will be shown by experimental results how the estimation accuracy of different numbers is different in the number counting using data.
(Estimation experiment 1)

  An experimental result of the estimation accuracy of the number of differences in the purchase transaction data recorded in the product purchase transaction will be described.

  As a population, 10 million purchase history data having 10 kinds of data items were used. That is, the purchase transaction data includes various variables such as a transaction date, a transaction store, and a transaction amount. Here, ten variables (data items) are selected. Moreover, calculation was performed about 1/4, 1/16, 1/64, 1/256, and 1/1024 as extraction rates.

  FIG. 13 is a diagram illustrating a result of comparing the estimation accuracy by changing the extraction rate in the estimation experiment 1. As a variable, one variable v5 is fixedly used. The true value indicates an estimated value for the true value.

  As can be seen from FIG. 13, when the extraction rate decreases, the estimation accuracy decreases. However, if the extraction rate is about 1/64, an estimation accuracy of 65% is obtained.

FIG. 14 is a diagram illustrating a result of comparing estimation accuracy by changing a variable and an extraction rate in the estimation experiment 1. FIG. 15 is a graph showing the true value in the result of FIG. In FIG. 15, a plot in which the estimation error is within 35% is surrounded by a broken line. As can be seen from this, it can be seen that the estimation error is within 35% at the extraction rate of 1/65 or more in 8 of the 10 variables used in the experiment.
(Estimation experiment 2)

  The experimental results of estimating the number of surnames of households nationwide from the surname data of households with a sampling rate of 1/16 will be described.

  According to the national census, the number of households nationwide is about 50 million. Last name data of 3123979 households was used as a sample with an extraction rate of 1/16. However, random sampling is not performed.

  FIG. 16 is a diagram illustrating an estimation experiment result in the estimation experiment 2. The number of different populations calculated from samples with 70506 different numbers is 131265.

  There are various theories of surnames from dozens of thousands to 300,000. The breadth of this width is greatly affected by the difference between the new and old characters. Therefore, when the last name using the old letter and the last letter using the new letter are added together, it may be the smaller value in the width.

  In this estimation experiment, the surnames using the old letters are added to the surnames using the new letters. Therefore, it can be said that the estimated value of 131265 matches well with the number of surnames of several ten thousand.

  The embodiments and examples of the present invention described above are examples for explaining the present invention, and are not intended to limit the scope of the present invention to only those embodiments or examples. Those skilled in the art can implement the present invention in various other modes without departing from the gist of the present invention.

  In addition, the devices of the above-described embodiments and examples can be realized by causing a computer to execute a software program that defines the processing procedure of each unit constituting the device. In that case, notification of data between the respective units may be performed via a data storage area such as a memory or a register. The data to be notified may be stored in the data storage area in the sending process, and the data may be read from the data storage area in the receiving process.

DESCRIPTION OF SYMBOLS 10 ... Different number counting apparatus, 11 ... Matrix generation part, 12 ... Different number calculation part, 20 ... Data analysis apparatus, 21 ... Memory | storage part, 22 ... Size calculation part, 23 ... Sample extraction part, 24 ... Data analysis part

Claims (8)

A different number counting method for counting the number of different populations of data groups,
Matrix generating means, based on the size of the population and the extraction rate for extracting a sample from the population, a frequency difference number vector indicating the number of differences for each frequency of the population, and the extraction from the population Generating a matrix that defines a relationship with a frequency vector that indicates the number of different samples for each frequency sampled at a rate;
A different number calculating means, based on the matrix generated by the matrix generating means and a sample extracted from the population at the extraction rate, calculating an estimated value of the different number of the population; Having different number counting method. The matrix is the inverse of a compressed matrix obtained by compressing the binomial probability matrix in which probabilities are arranged as elements so that the product of the population frequency vector and the number vector differs from the sample frequency vector. Matrix
The different number calculation means includes a compression frequency difference number vector obtained by compressing the frequency difference number vector of the sample extracted from the population and a product of the estimated compression frequency difference number vector of the population by a product of the inverse matrix and the inverse matrix. And the sum of the elements of the number vector of the number of different compression degrees estimated for the population is an estimated value of the number of differences of the population,
The different number counting method according to claim 1. In the compression matrix, which is a matrix in which elements are grouped by a group in which up to a predetermined line in the row direction in the binomial probability matrix is cut and a predetermined line is cut in the column direction, and the elements are grouped for each group, The number of rows in the group increases as the row number increases, and the number of columns in the group increases as the column number increases.
The different number counting method according to claim 2. In the binary matrix, the compression matrix groups elements by a group in which up to a power-of-two element in the row direction and a power of up to a power-of-two element in the column direction. Is a matrix that summarizes
The method for counting different numbers according to claim 3.   The calculation for obtaining the compression matrix is an operation for obtaining a weighted average of elements in a row direction of the binomial probability matrix and obtaining a total value of elements in a column direction. Different number counting device. The matrix generation means uses the elements of binomial probability matrix P obtained from the extraction ratio r according to equation (1), determine the I Ri logarithmic compression binomial probability matrix LP in equation (2), said compression matrix The different number counting method according to any one of claims 2 to 5, which is used as :

A different number counting device for counting different numbers of populations of data groups,
Based on the size of the population and the extraction rate for extracting a sample from the population, the frequency difference vector indicating the number of differences for each frequency of the population and the extraction rate extracted from the population Matrix generating means for generating a matrix that defines a relationship with a frequency difference number vector indicating a difference number for each frequency of a sample;
A different number calculating means for calculating an estimated value of a different number of the population based on the matrix generated by the matrix generating means and a sample extracted from the population at the extraction rate. Number counting device. A different number counting program for counting the number of different population consisting of data groups,
Computer
Based on the size of the population and the extraction rate for extracting a sample from the population, the frequency difference vector indicating the number of differences for each frequency of the population, and the extraction rate extracted from the population Matrix generating means for generating a matrix that defines a relationship with a frequency difference number vector indicating a difference number for each frequency of a sample;
In order to function as a different number calculation unit that calculates an estimated value of a different number of the population based on the matrix generated by the matrix generation unit and a sample extracted from the population at the extraction rate. Different number counting program.

Images