JP5655764B2 - Sampling apparatus, sampling program, and method thereof - Google Patents

Description

  The present invention relates to a sampling device, a sampling program, and a method thereof.

  A technique is known in which data to be processed is acquired from a large-scale data source such as a database or a distributed file, and distributed processing is performed by a plurality of processing nodes. In addition, research has been conducted on how to efficiently obtain data for distributed processing from a data source and assign it to processing nodes.

  In the distributed processing system, when the size of a task to be generated is biased, the load is concentrated on a node to which a large task is assigned, and the overall throughput (processing performance) is lowered. Therefore, it is necessary to assign tasks per unit performance of processing nodes equally, and an algorithm called Skewed Join is used as a method for that purpose.

In Non-Patent Document 1, in order to realize Skewed Join, a value is sampled from a data source, a histogram showing the appearance frequency of the value of the data in the data source is created, and distributed processing is assigned according to the histogram. Describes how to decide. Skewed Join has the advantage that the size of the acquired data can be made uniform if the created histogram represents the trend of the entire data source.

  When sampling data from a data source, creating a histogram, and determining the task of distributed processing, the sampling result must be as fair as possible, that is, represent the trend of the entire data source. This is because when the distribution of records appearing in the histogram deviates from the actual number of records, the number of records exceeding the processing capacity is allocated to the processing nodes, which causes a reduction in the overall throughput.

  As shown in FIG. 1, the sampling method includes a method of accessing the entire data source and a method of reading only a part. FIG. 1A is a diagram showing a state in which all files in a storage device as a data source are accessed and all records are read. Non-Patent Document 2 discloses a method for sampling by reading the entire file in this way.

  FIG. 1B is a diagram showing a state in which sampling is performed with reference to reference data recorded in some files in the storage device. The reference data generally refers to data prepared for searching such as a database index. The reference data includes, for example, types such as a hash and a tree. By referring to these, it is possible to know at which position in the data source a record having a specific key exists. That is, the sampling means can grasp the data distribution of the entire data source by referring to only a part of the data source.

  Patent Document 1 describes a technique for providing an approximate answer to a database by preparing an approximate inquiry engine as reference data. The invention described in Patent Document 1 prepares an approximate query engine that is reference data corresponding to a database, and updates a plurality of data samples in the approximate query engine with a predetermined probability corresponding to the update of the database. It is characterized by. Since the data in the approximate query engine is equivalent to the data sampled from the database, the database engine can obtain the distribution of data without accessing the entire database by accessing the approximate query engine.

Japanese Patent Laid-Open No. 11-353331

"Skewed Join", [online], Apache Pig Wiki, [searched September 16, 2011], Internet <URL: http://wiki.apache.org/pig/PigSkewedJoinSpec> Jefferey Scott Vitter, “Random Sampling with a Reservoir”, ACM Transactions on Mathematical Software, Vol. 11, pp. 37-57, Mar. 1985

  According to the technique described in Non-Patent Document 2, which is a method for reading the entire file when sampling data from the data source, the data distribution in the data source can be obtained reliably, while the larger the data source, the more the data is read. There is a drawback that it takes time. For example, if there is a data source in which 10 storage devices each having a reading performance of 100 MB per second and a storage capacity of 100 GB are arranged in parallel, it takes about 17 minutes to read all data. This method makes it difficult to obtain a practical speed when the data source becomes large.

  On the other hand, when using the method of creating a histogram by reading only a part of the data source, it is necessary to prepare a data structure with a low search cost such as a hash or tree in advance in the data source in order to perform processing efficiently. There is.

  However, there may be cases where the data source has not been structured for searching since the beginning of writing. For example, it is difficult to assume that data that requires writing throughput, such as a vehicle running log or a web access log, is structured and stored. In order to structure these data, it is necessary to read the entire data source and store it again in a structured state, so that eventually the entire data source is read.

  Thus, in the prior art, there has been a performance problem when sampling from a data source that is not optimized for searching.

  In order to solve this problem, some sampling methods can be considered in addition to the prior art. For example, a method of sampling a specific number of records from the top of the data source. However, according to this method, when recorded data has locality in its distribution like time series data, there is another problem that the histogram is different from the tendency of the entire data source. For example, if data with a certain tendency among recorded data is concentrated behind the data source, sampling the top data will result in a sample result that is different from the actual data. Sampling cannot be realized.

  As another method, there is a method called random sampling in which a position in a data source is randomly specified and a record existing at that position is taken out as a sampling target. However, when this method is used, since a record having a large size is easily selected, an unfair selection is made when there is a variation in the record size. If an unfair choice is made, the acquired histogram will still differ from the overall data source trend.

Thus, if the data stored in the data source is not optimized for searching and has a local tendency, it is fair to get a histogram showing the distribution of the data in the data source There is a problem that reliable sampling cannot be performed, that is, the reliability of the sampling result is lowered.

  The present invention has been made in consideration of the above-mentioned problems, and provides a sampling device that can appropriately acquire an overall trend for a data source that is not optimized for searching. Objective.

  To achieve the above object, the sampling apparatus according to the present invention performs sampling on a data source by the following means.

  A sampling device according to the present invention is a sampling device that randomly samples a record from a plurality of records recorded in a storage device, a random number generation unit that generates a random number corresponding to a recording position, and the plurality of the plurality of records Based on the record selection means for selecting a record having data at a recording position corresponding to the generated random number from the record, record length acquisition means for acquiring a record length of the selected record, and the acquired record length And sample determination means for adopting the selected record as a sample with the probability calculated in the above.

  That is, the length of the record existing at the position specified by the random number is acquired from the data source, and the selected record is adopted as a sample with a probability corresponding to the length. In random sampling in which a recording position in the data source is randomly specified and read, the probability of selection varies depending on the length of the record. On the other hand, the probability that each record is selected can be adjusted by multiplying the adoption probability after selection. This method enables highly reliable sampling.

  Further, it is preferable that the sample determination means calculates the probability so as to be inversely proportional to a record length of the acquired record.

  That is, the longer the record length of the selected record, the lower the probability of adopting it as a sample. As a result, all records are sampled with the same probability.

The sampling device according to the present invention further includes a minimum record length acquisition unit that acquires a record length S _min of a record having a minimum size among the plurality of records, and the sample determination unit is the selected Preferably, the record length of each record is S _k, and the probability is calculated by the formula of S _min / S _k .

That is, the record length of the base and S _min, the S _min, divided by the length of the selected record, to determine the probability of employing the record. With this configuration, the efficiency of random sampling can be maximized.

  In addition, the sampling device according to the present invention may further include a histogram generating means for generating a histogram indicating a frequency distribution of data recorded in the storage device from the record adopted as the sample.

  By generating the histogram, the frequency distribution of the recorded data can be totaled, so that the present invention can be easily applied to the distributed processing system.

The storage device may be a distributed file system, and a process to be executed may be assigned to an external distributed processing node using the histogram.

  By configuring in this way, in the distributed processing system, it becomes possible to average the size of the task to be assigned, so that the processing throughput of the entire system can be improved.

  ADVANTAGE OF THE INVENTION According to this invention, the sampling apparatus which can acquire the whole tendency appropriately with respect to the data source which is not optimized for a search can be provided.

It is a figure explaining the existing data sampling method. It is a figure showing the composition of the system concerning a first embodiment. It is a figure showing the example of the histogram which concerns on 1st embodiment. It is a figure showing the example of the content of the data source which concerns on 1st embodiment. It is a process flowchart of the system which concerns on 1st embodiment. It is a block diagram of the system which concerns on 2nd embodiment. It is a schematic diagram of the system concerning a second embodiment.

(First embodiment)
The sampling device according to the first embodiment will be described in detail with reference to FIG. FIG. 2 is a diagram illustrating a system configuration of the sampling apparatus according to the first embodiment.

  The sampling apparatus 100 according to the present embodiment accesses a data source, a sampling unit 111 that randomly selects a predetermined number of records, and a histogram generation unit that generates a histogram that is a data distribution in the data source based on the sampling result 112 and a minimum record length storage unit 121 that records the size of the minimum record among the records recorded in the data source.

  The above configuration is executed by a central processing unit (CPU), a main storage device (RAM), and an auxiliary storage device (storage medium) (not shown). The program code for executing the processing described in the present embodiment is stored in the auxiliary storage device, and the CPU reads and executes the program code, thereby realizing the functions of the above-described embodiments.

  The data source 120 is means for storing data to be sampled, and corresponds to a relational database, a data file, a disk device, or the like. In the present embodiment, description will be made assuming a single data file, but any means may be used as long as it can record a plurality of records. Further, these data storage means are not limited to one device. For example, virtual storage means such as a disk array or a distributed file system may be used as one data source.

  The data source stores a plurality of records for one record separated by a separator (delimiter) such as a line feed code or a comma. These records have different lengths, and the recording position of the record can be specified by a memory address. The memory address is information for specifying the physical or logical location where the record is recorded. For example, if the data source is a file, the number of bytes from the beginning, and if it is a distributed file system, the record of the file is recorded. This corresponds to a virtual address indicating a position.

The minimum record length storage unit 121 is a means for recording the minimum record length among the records that the data source 120 has. For example, when the minimum value of a record is 512 bytes, the minimum record length storage unit 121 holds a numerical value of 512 bytes. The minimum record length storage unit 121 does not necessarily have to be independent as a device, and may be recorded in the data source 120 together with sampling target data, for example. When a record is added, if the length of the added record is less than the minimum record length, the minimum record length is updated with the latest value.

  The sampling unit 111 is means for performing an operation of acquiring a predetermined number of records at random from the data source 120. In the description of the embodiment, an operation of randomly acquiring a predetermined number of records from a data source is referred to as sampling. Detailed operation will be described later.

  The histogram generation unit 112 is a unit that generates a histogram representing the frequency distribution of records in the data source based on the data sampled by the sampling unit 111. The histogram creates a frequency distribution based on the key of the sampled data. FIG. 3 is an example of the created histogram. In this example, the vehicle travel log is classified by event ID, and represents the number of records corresponding to each ID.

Next, a method in which the sampling unit 111 performs sampling will be described. FIG. 4 is a diagram showing the entire record stored in the data source. Here, for convenience, description will be made assuming that the minimum value of the record length is S _min and the total length of the data source is S _min × 10. The sampling unit 111 generates a uniform random number corresponding to the memory address included in the data source, and determines a memory address to be sampled. If processing is performed on the data source shown in FIG. 4, a random number N that takes a range of 0 ≦ N ≦ 10 × S _min is determined, and a record existing at the determined memory address is acquired. This process identifies a single record in the target data source. For example, when S _min = 1 and N = 3, the record 4 is selected in the case of FIG. 4A, and the record 3 is selected in the case of FIG. 4B.

  FIG. 4A shows a case where records having the same length are recorded uniformly. In this case, if sampling is performed at random, 10 records A to J are acquired with equal probability. That is, since the same number is theoretically sampled for each record, there is no bias in the histogram, and the above-described problem does not occur.

  FIG. 4B illustrates a case where the record lengths are different. For example, record 3 and record 6 are twice as long as record 1, and record 5 is three times as long. In such a case, when a memory address that is a recording position on the data source is randomly determined and an attempt is made to acquire a record at a corresponding location, record 3 is recorded with respect to records 1, 2, and 4 having the smallest size And 6 are easily selected twice, and record 5 is easily selected three times.

  That is, since the records are not selected equally, the distribution appearing in the created histogram is biased even though the actual number of records is equal.

Therefore, in the present embodiment, when the sampling unit selects a record, whether or not the record is adopted as a sample is determined by the following equation (1). S _min is the minimum record length of the records recorded in the data source, and S _k represents the length of the selected record.
Employment probability p = S _min / S _k Formula (1)
The sampling unit 111 determines whether to adopt the selected record as a sample according to the adoption probability p calculated by the equation (1). For example, when the adoption probability p is 0.1, the selected record is adopted with a probability of 10%, and the selection is discarded with a probability of 90%. The calculation of the probability can be performed by a method of newly generating a random number taking a real number in the range of 0 to 1, for example, and adopting the selected record only when the result is equal to or less than the adoption probability.

  According to this method, the length of a record and the probability of adopting a randomly selected record can be made inversely proportional. That is, by multiplying the expression (1) by the adoption probability p that is inversely proportional to the length of the record, the probability of sampling can be made the same for all the records.

If the length of the entire data source is L, the probability that a record having a record length of S _k is selected is S _k / L. Multiply this by equation (1)
S _k / L × S _min / S _k = S _min / L (2)
Thus, it can be confirmed that the probability that all records are sampled is the same.

  Next, the sampling operation will be described with reference to FIG. 5 which is an operation flowchart of the sampling unit 111. When the sampling apparatus according to the present embodiment starts operation, the sampling unit 111 acquires the minimum record length from the minimum record length storage unit 121 (S10). As described above, the minimum record is the minimum value among the variable record lengths.

  Next, a random number for determining a specific position in the data source is generated (S11). The random number to be generated is a uniform random number, and any random number can be generated as long as the recording position of the record recorded in the data source can be specified. For example, in the case of a disk device, the range of random numbers to be generated may be from the minimum value to the maximum value of the sector number in which data is stored.

  Next, a record corresponding to the determined random number is selected (S12). That is, a record belonging to the recording position specified by the random number is selected from the data source. In this embodiment, since a data source that only performs additional writing is assumed, processing is not performed when there is no record at the specified position, but it may be considered that data does not exist due to fragments or the like. Good. In that case, the process may return to step S11 and the process may be performed again, or the records existing immediately before and after may be selected.

  Next, the length of the selected record is acquired (S13), the calculated record length and the minimum record length are used to calculate according to the formula (1), and the record is adopted according to the calculated adoption probability. It is determined whether or not to perform (S14). As described above, the determination method may be determined by newly generating a random number, or another method may be used. If it is determined that the selected record is not adopted, the process returns to step S11 to generate a random number again.

  Next, it is determined whether the number of samplings has reached a specified number (S15). The number of samplings may be unique depending on the device, or may be automatically determined according to the scale of the data source. When the specified number of times of sampling has been completed, the process ends. When the specified number of times has not been reached, the record is selected again.

  Through the above processing, the sampling unit 111 can acquire records for a predetermined number of samples. The acquired record is selected at random regardless of the record length.

When the sampling unit 111 completes sample collection, the histogram generation unit generation unit 112 generates a histogram using the collected samples. The histogram is generated by adding the number of records for each key included in the acquired sample data.

  According to the present embodiment, in the sampling device that samples from different data sources, the record can be equally sampled regardless of the record length by setting the adoption probability so as to be inversely proportional to the record length. it can. That is, since the generated histogram represents the tendency of the entire data source, the tendency of data distribution can be obtained in a short time even for a huge data source.

  In the first embodiment, the histogram generation unit is used to generate the histogram. However, if the histogram can be generated by other means, or if data aggregation means other than the histogram can be used, the histogram generation unit Is not a required configuration.

(Second embodiment)
In the second embodiment, the sampling apparatus 100 in the first embodiment is incorporated into a distributed processing system using Hadoop, which is a distributed processing framework. FIG. 6 is a system configuration diagram according to the second embodiment, and FIG. 7 is a conceptual diagram of a system according to the second embodiment. The configuration of the data source and the record collected by the sampling device according to the second embodiment is the same as that described in the first embodiment.

  In the second embodiment, a distributed file held by a distributed processing node is used as a data source. The distributed files 202a, b, and c are arranged on the computers 201a, b, and c, which are distributed processing nodes, respectively, and can be accessed seamlessly from the sampling unit 111. That is, since it seems that there is one data source from the sampling unit 111, data sampling can be performed by the same method as in the first embodiment.

  Further, the histogram generated by the histogram generation unit 112 is transmitted to the task assignment unit 130. The task allocation unit 130 acquires tasks to be processed from the distributed files 202a, b, and c based on the generated histogram, and allocates processes to the distributed processing nodes 201a, b, and c.

  In the present embodiment, the present invention is applied to a High that operates on Hadoop, which is a distributed processing framework. As shown in FIG. 7, Hive has three phases: Compiler, Optimizer, and Executor. Among these, sampling is performed in the Optimizer phase, and task allocation, that is, scheduling is performed in the Executor phase so that the task size per unit performance of the distributed processing node is equalized from the obtained histogram.

  With this configuration, it is possible to reduce the load unevenness among nodes in distributed processing, and to improve the throughput of the entire system. In this example, an example of application to High is given, but other middleware that operates on Hadoop, such as Pig, may be applied.

(Modification)
The above embodiment is merely an example, and the present invention can be implemented with appropriate modifications within a range not departing from the gist thereof. For example, in the description of the embodiment, an example in which a program for performing processing is recorded in the auxiliary storage device and the processing is performed by the CPU has been described. However, the processing may be performed by an FPGA, or may be designed and executed as hardware. May be.

In the description of the embodiment, the probability p of adopting the selected record as a sample is calculated by the formula of S _min / S _k , but the adoption probability p is a × S _min / S _k (where 0 <a < Even in 1), the probability that each record is sampled can be made the same. However, when the coefficient a is multiplied, the probability that a record is not sampled in one execution increases, so that the adoption probability p is most efficient when S _min / S _k is used.

  Further, in order to obtain the effect of the invention, it is necessary to adjust the probability of adopting the record according to the size of the selected record, that is, the longer the selected record, the lower the adoption probability. May not be inversely proportional to the record length.

For example, the employment probability p may be n / S _k (n is an arbitrary constant), and if p exceeds 1, it may be treated as 1. In this case, the probability that each record is selected is not the same, but can be close to a uniform value. Thus, in order to bring the probability that each record is sampled close to a uniform value, it is only necessary that the adoption probability can be lowered as the selected record becomes longer.

In addition, when the record sizes vary extremely, there may be a case where a necessary number of records cannot be sampled unless selection is repeated many times. For example, if the average size of the record and S _avg, because the probability that a record is selected in one cycle of execution is S _min / S _avg, number of loops required for n times of sampling, (n × S _avg) / S _min . If this value exceeds the total number of records, it is more efficient to read the entire data source.

For example, in the example of FIG. 4B, since S _avg = 1.66... × S _min , when n ≧ 4, the number of loops is equal to or greater than the total number of records. For this reason, if the record size varies to the extent that the above condition is met, the stochastic factor may be reduced from the algorithm.

  For example, by selecting mth, m × 2nd,... Records from randomly selected records, one or more records can be reliably sampled in a single loop. This causes a side effect that the locality of the tendency of the record is reflected in the histogram, but the influence may be reduced by selecting a record away from the randomly selected record.

DESCRIPTION OF SYMBOLS 100 Sampling apparatus 111 Sampling part 112 Histogram generation part 120 Data source 121 Minimum record length memory | storage part 130 Task allocation part 201a, b, c Distributed processing node 202a, b, c Distributed file node

Claims (11)

A sampling device that randomly samples a record from a plurality of records recorded in a storage device,
Random number generating means for generating a random number corresponding to the recording position;
Record selecting means for selecting a record having data at a recording position corresponding to the generated random number from the plurality of records;
Record length acquisition means for acquiring a record length of the selected record;
Sample determination means that employs the selected record as a sample with a probability calculated based on the acquired record length;
A sampling device characterized by comprising: The sampling apparatus according to claim 1, wherein the sample determination unit calculates the probability so as to be inversely proportional to a record length of the acquired record. A minimum record length obtaining means for obtaining a record length S _min of a record having a minimum size among the plurality of records;
The sampling apparatus according to claim 2, wherein the sample determination unit calculates the probability using a formula of S _min / S _k , where S _k is a record length of the selected record. The sampling apparatus according to claim 3, further comprising a histogram generation unit that generates a histogram indicating a frequency distribution of data recorded in the storage device from the record adopted as the sample. The sampling device according to claim 4, wherein the storage device is a distributed file system, and processing to be executed is assigned to an external distributed processing node using the histogram. A sampling method performed by a sampling device that randomly samples a record from a plurality of records recorded in a storage device,
Generating a random number corresponding to the recording position;
Selecting a record having data at a recording position corresponding to the generated random number from the plurality of records;
Obtaining a record length of the selected record;
Adopting the selected record as a sample with a probability calculated based on the acquired record length;
A sampling method characterized by comprising: The sampling method according to claim 6, wherein the probability is calculated so as to be inversely proportional to a record length of the acquired record. A step of obtaining a record length S _min of a record having a minimum size among the plurality of records;
The sampling method according to claim 7, wherein a record length of the selected record is S _k, and the probability is calculated by a formula of S _min / S _k . On the computer,
A program for randomly sampling a record from a plurality of records recorded in a storage device,
Processing to generate a random number corresponding to the recording position;
A process of selecting a record recorded at a recording position corresponding to the generated random number from the plurality of records;
Processing for obtaining a record length of the selected record;
A process of adopting the selected record as a sample with a probability calculated based on the acquired record length;
A program characterized by having executed. In the computer,
The program according to claim 9, wherein the probability is calculated so as to be inversely proportional to a record length of the acquired record. In the computer,
A process of acquiring a record length S _min of a record having a minimum size among the plurality of records,
11. The program according to claim 10, wherein a record length of the selected record is S _k and the probability is calculated by a formula of S _min / S _k .

Images