JP4408078B2 - Sort processing apparatus, sort processing method and program - Google Patents

Description

  The present invention relates to a sort processing technique for sorting large-capacity sort target data on a main storage device.

In the conventional sort processing method, two areas are provided on the main memory, the data to be sorted (records) is divided into a sort key part and other parts, and stored separately in the two areas, and only the sort key part is sorted. Then, by combining and outputting the sorted sort key and the part other than the corresponding sort key, the cache hit rate is improved so that memory access does not enter in a wide range (Patent Document 1).
Japanese Patent Laid-Open No. 2000-10760

In the conventional sort processing method described above, when the size of the sort key is large, memory access is made in a wide range, so that cache misses frequently occur. For example, when the size of the sort key is large and the value is close to the size of the record, memory access can be performed over a wide range even if the record is divided.
An object of the present invention is to solve the above-described problems, and it is an object of the present invention to increase the speed of the sorting process and to improve the cache hit rate.

The sort processing device according to the present embodiment is
A sort processing device that sorts data to be sorted using a cache memory,
A block division unit that calculates a block size based on the cache size of the cache memory, and divides the data to be sorted into a plurality of blocks based on the calculated block size;
Using the cache memory, an intra-block sort processing unit that performs intra-block sorting for each block divided by the block dividing unit;
It has an inter-block merge processing unit that uses the cache memory to merge blocks after sorting by the intra-block sort processing unit and sort the data to be sorted.

  According to the present invention, by dividing into block sizes adapted to the cache size, high-speed sort processing can be realized by effectively using the cache memory, and the cache hit rate can be improved. .

Embodiment 1 FIG.
FIG. 1 is a diagram showing the configuration of an embodiment of the present invention. In FIG. 1, the sort processing device 100 executes the sort processing of the sort target data 2 on the main storage device 1, and outputs the sorted result data 3 after sorting to the main storage device 1. Although not shown in FIG. 1, the sort processing device 100 executes sort processing using a CPU (Central Processing Unit) having a cache memory (primary cache, secondary cache). The sort processing apparatus 100 includes an intra-block sort processing unit 4, an inter-block merge processing unit 5, a block information holding unit 6, a block dividing unit 10, a CPU ID detection unit 101, and a CPU information holding unit 102. It is.

  The sort processing apparatus 100 may be configured by hardware, or all or part of the elements of the sort processing apparatus 100 may be configured by a program operable on the CPU. Alternatively, it may be realized by firmware stored in the ROM. Alternatively, it may be realized by a combination of software and hardware, a combination of software, hardware and firmware, or the like.

The CPU ID detection unit detects the ID (identification information) of the CPU (mounted on the computer) used for the sort process. The CPU ID detection unit 101 is an example of an identification information detection unit.
The CPU information holding unit 102 holds primary cache size and secondary cache size information (cache size information) of the CPU. FIG. 2 shows an example of cache size information held by the CPU information holding unit. For example, if the CPU ID detected by the CPU ID detection unit is “A02”, information that the primary cache size is 16 KB and the secondary cache size is 1 MB is extracted. The CPU information holding unit 102 is an example of a cache size information holding unit.

  The block division unit 10 determines the block size when the sort target data 2 is divided into blocks based on the CPU ID detected by the CPU ID detection unit 101 and the cache size information obtained from the CPU information holding unit 102, and the sort target data 2 Is divided into a plurality of blocks. Details of the block size calculation procedure will be described later. The block dividing unit 10 stores the address information of the divided blocks in the block information holding unit 6.

The intra-block sort processing unit 4 performs intra-block sorting of the data to be sorted in all blocks stored in the block information holding unit.
The inter-block merge processing unit 5 executes merging of the sorted blocks, and outputs the result as sort result data 3 on the main storage device.

Next, the operation of the sort processing apparatus 100 will be described with reference to FIGS.
First, in step S301, the CPU ID detection unit 101 detects the CPU ID by a micro instruction. For example, in the Xeon (registered trademark) processor of the microprocessor manufactured by Intel (registered trademark), the CPU ID can be directly read from the CPU by the instruction “cpuid”.

  Next, in step S302, the block dividing unit 10 acquires the cache memory information of the corresponding CPU from the CPU information holding unit 102 based on the CPU ID obtained by the CPU ID detecting unit 101, and the acquired cache memory information is included in the acquired cache memory information. Based on this, the cache size of the cache memory is determined.

Next, in step S303, the block dividing unit 10 calculates a block size for dividing the sort target data 2 (block dividing step).
The standard for calculating the block size is such that the size of the partial data string formed by the minimum value of each block hits the CPU primary cache and the size of one block hits the CPU secondary cache. As shown in FIG. 6, after the sorting target data 2 is divided into blocks, the minimum value of each block is extracted, and a partial data string composed of the minimum value of each block is generated. The block size is calculated so that the partial data string composed of the minimum value of each block is within the range of the primary cache size and the size of one block is within the secondary cache size.

CPU primary cache size: C1, CPU secondary cache size: C2, sort target data per record length: RL, data size of all sort target data (record length per record x number of records) : AR, number of blocks (when sorting target data is divided): BN, block size (when sorting target data is divided): BS, BN = C1 / RL, BS = AR / BN.
Here, by satisfying BS <C2, sorting within the block in the secondary cache of the CPU can be performed.

Next, in step S304, the sort target data 2 is divided into a plurality of blocks based on the block size calculated in S303 (block division step).
For example, if the CPU ID detected by the CPU ID detection unit 101 is “A02”, it can be seen from the CPU information holding unit shown in FIG. 2 that the primary cache size is 16 KB and the secondary cache size is 1 MB. Next, in FIG. 4, if there is a record with a record length of 100B and 1,000,000 records, the sort target data is 100,000,000B. This is divided into blocks of 1,000,000B-100B (one record) (401). The reason for calculating by reducing 100B (one record) is to add a record for data exchange at the time of sorting. When the sort target data is divided by 1,000,000B-100B, 100 blocks of 9,999 records and one block of 100 records are divided into one. When one record for data exchange is added to each block, a block of size 1,000,000B (9,999 records (9,999 records × 100B = 999,900B) + one record for data exchange (100B) ) 100 and a size of 10,100B (100 records (100 records × 100B = 10,000B) + one record for data exchange (100B)) are obtained. In this case, the minimum value data string is generated by 101 records, but the size of the minimum value data string is 101, and if the primary cache size is 16 KB when the primary cache size is 16 KB, 1 for data exchange at the time of sort execution Even if records are included, the size fits in the primary cache memory.

  Further, when the block dividing unit 10 performs block division in step S304, the block information holding unit 6 is notified of the address information of each block, and the block information holding unit 6 holds the address information of each block. For this reason, the intra-block sort processing unit 4 can obtain the address information of each block by the block information holding unit 6. In step S305, the intra-block sort processing unit 4 performs intra-block sort using the secondary cache for all blocks divided by the block dividing unit 10 (intra-block sort processing step). For example, if quick sort is used, both data reference and data exchange can be executed at high speed on the cache memory.

  When the intra-block sorting is completed for all the blocks, the inter-block merge processing unit 5 then merges each block in step S306 (inter-block merge processing step). The merge processing between blocks will be described with reference to the flowchart of FIG. First, a data string based on the minimum value of N blocks sorted in a block is generated (S3061). Next, the generated data string is sorted (S3062). Each block sorted at this time can be identified by the block information holding unit 6 as to which block data. The minimum value of the sorted data string is output on the main memory as the minimum value of the sort result data (S3063). It is determined whether there is the next minimum value data in the block to which the output data belonged (S3064). If there is the next minimum value data, it is inserted into the minimum value data string ( S3065), the process is repeated from the process of sorting the minimum value data string (S3062). In the case of FIG. 6, since “8” belonging to block 2 is output as the minimum value in the minimum value data string, “22” that is the next minimum value of block 2 is inserted into the minimum value data string. .

In this way, the next minimum value is inserted into the minimum value data string from the block of data that has output the minimum value, thereby sorting the entire data to be sorted.
When a block becomes empty, the block is removed from the merge target, and the merge process is performed only with the remaining blocks. When the output minimum value block is empty, the next smallest data is output after the output minimum value. Then, the next smallest data is inserted from the block to which the output data belongs to the minimum value data string.

When the block to which the output data belongs is inserted into the next smallest minimum value data string and the minimum value of the data string is output to the main memory, when all the blocks are empty Sorting of the data to be sorted is completed (S3066).
Through the above processing, sort result data 3 is obtained in the main storage device 1.

  In the above processing, the minimum value in each block is extracted to generate a minimum value data string, and the minimum value in the minimum value data string is sequentially output to the main storage device. The maximum value may be extracted to generate a maximum value data string, and the maximum value in the maximum value data string may be output to the main storage device in order.

  As described above, the sort processing apparatus according to the present embodiment sorts the large-capacity sort target data on the main storage device, the CPU ID detection unit for detecting the CPU ID, the CPU ID and the CPU ID CPU information holding unit for holding cache memory information, block dividing unit for dividing sort target data into a plurality of blocks based on CPU cache memory information obtained by the CPU information holding unit, and block information for holding divided block information The block division includes a holding unit, an intra-block sort unit that sorts the sort target data for each divided block, and an inter-block merge processing unit that performs a merge on each sorted block and obtains a sort result of the entire sort target data. The CPU 10 uses the CPU's primary cache and secondary cache for the sort process. The size information is acquired from the CPU information holding unit 102, the block size for dividing the sort target data 2 is calculated based on the cache sizes of the primary cache and the secondary cache, and the sort target is calculated based on the calculated block size. The data is divided into a plurality of blocks, and the intra-block sort processing unit 4 sorts the blocks for each block divided by the block division unit 10 using the CPU secondary cache, and the inter-block merge processing unit 5 However, using the primary cache, merging each block after sorting by the intra-block sort processing unit 4 to sort the data to be sorted, and outputting the sorted result data 3 to the main storage device 1 It is characterized by.

  The sort processing device according to the present embodiment can execute the intra-block sort at high speed on the secondary cache memory by setting the block size for dividing the data to be sorted within the secondary cache size. it can. Further, by setting the block size for dividing the data to be sorted to a size in which the data size of the minimum value data string is within the primary cache size, the sorting of the minimum value data string after the intra-block sort is performed in the primary cache memory. It can be executed at high speed. Thereby, large-scale data on the main memory can be sorted at a high speed, and the cache hit rate can be improved.

Embodiment 2. FIG.
In the first embodiment described above, the CPU ID is directly read out. Next, an embodiment in which this is input from the outside will be described.

  FIG. 7 is a diagram showing a configuration of the present embodiment, in which a CPU information reading unit 103 is inserted instead of the CPU ID detection unit 101 and the CPU information holding unit 102 of the first embodiment. Other components of the sort processing apparatus 100, the intra-block sort processing unit 4, the inter-block merge processing unit 5, the block information holding unit 6, and the block dividing unit 10 are the same as those in the first embodiment.

The CPU information reading unit 103 inputs CPU cache size information from the user. The CPU information reading unit 103 is an example of a cache size information input unit.
Also in this embodiment, the sort processing apparatus 100 may be configured by hardware, or all or part of the elements of the sort processing apparatus 100 may be configured by a program operable on the CPU. . Alternatively, it may be realized by firmware stored in the ROM. Alternatively, it may be realized by a combination of software and hardware, a combination of software, hardware and firmware, or the like.

Next, the operation of the sort processing apparatus 100 according to the present embodiment will be described with reference to FIG.
In step S801, the CPU information reading unit 103 inputs CPU cache size information from the user. The input cache size information indicates the size of the primary cache memory and the size of the secondary cache memory of the CPU. Next, in step S <b> 802, the block dividing unit 10 calculates a block size when dividing the sort target data 2 based on the cache size information read by the CPU information reading unit 103. The block size calculation method is the same as that shown in the first embodiment. In the same manner as in the first embodiment, the data to be sorted is divided (S803), the divided blocks are sorted (S804), and the blocks after the intra-block sorting are merged (S805). ).

  As described above, the sort processing device according to the present embodiment has a CPU information reading unit and a CPU information reading unit that read information in the cache memory of the CPU in order to sort a large amount of data to be sorted on the main storage device. A block division unit that divides the data to be sorted into a plurality of blocks based on the CPU cache memory information obtained by the above, a block information holding unit that holds information on the divided blocks, and a block that sorts the data to be sorted for each divided block The sorting unit includes an inter-block merge processing unit that performs merging on each sorted block and obtains a sorting result of the entire sort target data.

  In the first embodiment, when the CPU information holding unit does not have the read CPU ID cache memory information, the information for dividing the block cannot be obtained. However, in this embodiment, the user does not have the CPU cache memory information. , And the CPU information reading unit reads the block division size.

Embodiment 3 FIG.
In the first and second embodiments described above, the block size is calculated based on the assumption that the memory capacity of the cache memory can be used 100% regardless of the use state of the cache memory of the CPU. In the embodiment, a case where the block size is calculated in consideration of the usage state of the cache memory will be described.

FIG. 9 is a diagram showing a configuration of the present embodiment, which is a configuration in which a cache usage rate information holding unit 104 is added to the sort processing apparatus shown in the first embodiment.
The cache usage rate information holding unit 104 holds statistical information on the usage rates of the primary cache and the secondary cache for each CPU. For example, the cache usage rate information holding unit 104 holds information indicating the average usage rate of the primary cache and the secondary cache for each time period.

  In addition, the block dividing unit 10 according to the present embodiment uses the CPU cache size information of the CPU information holding unit 102 and the cache usage rate information of the cache usage rate information holding unit 104 to perform sorting target data division. Calculate the block size. For example, it is assumed that the CPU ID “A02” in FIG. 2 is detected by the CPU ID detection unit 101. At this time, if the cache usage rate information of the cache usage rate information holding unit 104 indicates the usage rate of the CPU “AO2”, for example, primary cache: 50%, secondary cache: 50%, the block dividing unit 10 is 50% of each cache size, primary cache: 8 KB, secondary cache: 0.5 MB is used as a reference value for block size calculation, and the size of the minimum value data string is primary cache: 8 KB. The block size is calculated so that it is within the range and one block is within the secondary cache: 0.5 MB.

The block size calculation is the same as that of the first embodiment except that the cache usage rate information in the cache usage rate information holding unit 104 is referred to.
In FIG. 9, the cache usage rate information holding unit 104 is added to the configuration of FIG. 1, but the cache usage rate information holding unit 104 may be added to the configuration of FIG. 7. In this case, the block dividing unit 10 uses the cache size information of the CPU read by the CPU information reading unit 103 and the cache usage rate information of the cache usage rate information holding unit 104 to block the size of the data to be sorted. Is calculated.

  Also in this embodiment, the sort processing apparatus 100 may be configured by hardware, or all or part of the elements of the sort processing apparatus 100 may be configured by a program operable on the CPU. . Alternatively, it may be realized by firmware stored in the ROM. Alternatively, it may be realized by a combination of software and hardware, a combination of software, hardware and firmware, or the like.

  According to the present embodiment, the block size can be calculated in consideration of the usage state of the cache memory.

1 is a block diagram showing a configuration of a first embodiment. It is a figure which shows the example of the cache size information which a CPU information holding part hold | maintains. FIG. 6 is a flowchart illustrating an operation example of the sort processing apparatus according to the first embodiment. It is the schematic diagram which showed the process which divides sorting object data, sorts in a block, and performs a merge. It is the flowchart figure which showed the example of merge processing after the sorting in a block was completed. It is the schematic diagram which showed the method of the merge after dividing | segmenting sorting object data and sorting in a block. 5 is a block diagram showing a configuration of a second embodiment. FIG. FIG. 10 is a flowchart illustrating an operation example of the sort processing apparatus according to the second embodiment. FIG. 6 is a block diagram showing a configuration of a third embodiment.

Explanation of symbols

  1 main storage device, 2 sort target data, 3 sort result data, 4 intra-block sort processing unit, 5 inter-block merge processing unit, 6 block information holding unit, 10 block dividing unit, 100 sort processing unit, 101 CPU ID detection unit , 102 CPU information holding unit, 103 CPU information reading unit, 104 Cache usage rate information holding unit.

Claims (6)

A sort processing device that sorts data to be sorted using a cache memory having a primary cache and a secondary cache ,
A block division unit that calculates a block size based on the cache size of the cache memory, and divides the data to be sorted into a plurality of blocks based on the calculated block size;
Using the cache memory, an intra-block sort processing unit that performs intra-block sorting for each block divided by the block dividing unit;
Using the cache memory, the maximum value data or the minimum value data in each block is extracted from each block after being sorted by the intra-block sort processing unit, and the extracted maximum value data of each block Alternatively, a data string based on the minimum value data is generated, each data is extracted in order from the maximum value data or the minimum value data in the generated data string, and each block after sorting by the intra-block sort processing unit is extracted. possess the interblock merge processing unit for merging,
The block dividing unit
The block size for dividing the data to be sorted is a size within the cache size of the secondary cache, and the data size of the data string generated by the inter-block merge processing unit is within the cache size of the primary cache. Sort processing apparatus characterized by having a size of The intra-block sort processing unit
Using the secondary cache, sorting within the block for each block divided by the block dividing unit,
The inter-block sort processing unit
The sort processing apparatus according to claim 1, wherein each block after sorting by the intra-block sort processing unit is merged using the primary cache. The sort processing device further includes:
An identification information detection unit for detecting identification information of a CPU (Central Processing Unit) of the cache memory used for sorting the data to be sorted;
A cache size information holding unit for holding cache size information indicating the cache size of each cache memory for a plurality of types of CPUs;
The block dividing unit
Based on the CPU identification information detected by the identification information detection unit and the cache size information of the cache size information holding unit, the cache size of the cache memory used for sorting the sort target data is determined, and based on the determined cache size 2. The sort processing apparatus according to claim 1, wherein a block size is calculated. The sort processing device further includes:
A cache size information input unit for inputting cache size information indicating the cache size of the cache memory used for sorting the data to be sorted;
The block dividing unit
The cache size of the cache memory used for sorting the sort target data is determined based on the cache size information input by the cache size information input unit, and the block size is calculated based on the determined cache size. The sort processing apparatus according to 1. A sort processing method for sorting data to be sorted using a cache memory having a primary cache and a secondary cache ,
A block dividing step of calculating a block size based on the cache size of the cache memory and dividing the sort target data into a plurality of blocks based on the calculated block size;
An intra-block sorting process step that sorts the blocks for each block divided by the block dividing step using the cache memory;
Using the cache memory, the maximum value data or the minimum value data in each block is extracted from each block after the sorting by the intra-block sorting process step, and the extracted maximum value data of each block Alternatively, a data string based on the minimum value data is generated, each data is extracted in order from the maximum value data or the minimum value data in the generated data string, and each block after sorting by the intra-block sort processing step is extracted. have a merge process steps between blocks to be merged,
In the block dividing step ,
The block size for dividing the data to be sorted is within the cache size of the secondary cache, and the data size of the data string generated by the inter-block merge processing step is within the cache size of the primary cache A sort processing method characterized in that a size is calculated . A program for sorting data to be sorted using a cache memory having a primary cache and a secondary cache ,
A block division process for calculating a block size based on the cache size of the cache memory and dividing the sort target data into a plurality of blocks based on the calculated block size;
Using the cache memory, an intra-block sort process that performs an intra-block sort for each block divided by the block division process;
Using the cache memory, the maximum value data or the minimum value data in each block is extracted from each block after being sorted by the intra-block sort process, and the maximum value data of each extracted block or A data string based on the minimum value data is generated, each data is extracted in order from the maximum value data or the minimum value data in the generated data string, and each block after sorting by the intra-block sort processing is merged. Let the computer execute the merge process between blocks ,
In the block division process,
In the computer,
The block size for dividing the data to be sorted is a size within the cache size of the secondary cache, and the data size of the data string generated by the inter-block merge process is within the cache size of the primary cache. The program characterized by calculating the size which becomes .

Images