JPWO2008155852A1 - Method and apparatus for aggregating tabular data in a shared memory parallel processing system - Google Patents

Abstract

In a shared memory multiprocessor system, in order to extract a unique set of item values related to a set of predetermined items, the number of occurrences of a set of item value numbers corresponding to the set of item values related to the set of predetermined items is calculated in parallel. Generate a cumulative frequency distribution array that expresses the cumulative frequency distribution of the number of occurrences in the shared memory, read the item value number pairs in parallel using the element value of the record number array as a subscript, Reads the elements of the cumulative frequency distribution array in parallel using the tuple as a subscript, and stores the values of the elements of the record number array in parallel in the sorted record number array using the elements of the cumulative frequency distribution array as the index pointer And the record number array sorted according to the value of the element of the cumulative frequency distribution array used as the index pointer has a unique set of field values Divided into groups of record number code, and extracts one record number from the group of segmented record number.

Description

  The present invention relates to a method and apparatus for extracting a unique set of item values from tabular data in a memory-sharing parallel processing system, particularly, a symmetric multiprocessor system. The present invention also relates to a program for causing a computer including a plurality of processors accessing a shared memory to execute a code for extracting a unique set of item values from tabular data. The present invention also relates to a computer program product for causing a computer including a plurality of processors that access a shared memory to execute the above method, and a recording medium on which the computer program is recorded.

  The present invention further relates to a method and apparatus for tabulating tabular data in a multi-dimensional manner in a memory-sharing parallel processing system, particularly a symmetric multiprocessor system. The present invention also relates to a program for causing a computer including a plurality of processors that access a shared memory to execute a code for tabulating tabular data in a multidimensional manner. The present invention also relates to a computer program product for causing a computer including a plurality of processors that access a shared memory to execute the above method, and a recording medium on which the computer program is recorded.

  Conventionally, it is required to process large-scale data at high speed in various industrial fields. For large-scale data processing, development of hardware technologies such as high-speed memory access by cache and prefetch, high-speed memory, and high-speed arithmetic processing such as parallel processing of processors, and data processing The development of algorithms continues to speed up. For example, the present inventor has proposed an “on-memory data processing algorithm” as described in Patent Document 1 in order to process large-scale data at high speed.

  In addition, efficient algorithms have been developed for processing large-scale data. Sorting is a process that occurs frequently when processing large-scale data, particularly large-scale tabular data. As an efficient sorting algorithm, a radix (RADIX) sort and a counting (COUNTING) sort (also called a count sort or a distribution counting sort) are known. The counting sort is sometimes used for sorting each digit of the radix sort and is an efficient algorithm.

  In recent years, a multiprocessor in which a plurality of processors operate in parallel, a multicore processor in which a plurality of CPU cores are mounted on one chip, in particular, a symmetric multiprocessor in which a plurality of CPUs share processing equally. A shared memory parallel processing system has been designed. Since processing such as retrieval, aggregation, and sorting of large-scale data often applies the same operation to large-scale data, it is considered that a symmetric multiprocessor is particularly suitable for high-speed processing of large-scale data. In this regard, the inventor has proposed a memory-shared multiprocessor system combined with an efficient sorting algorithm to process large-scale tabular data as described in Patent Document 2. Yes.

  In the memory shared multiprocessor system described in Patent Document 2, first, a record to be processed is divided and assigned to a plurality of processors. Next, each processor counts the number of local occurrences of the item value number associated with the record to be processed. Next, the local appearance count of the item value number counted by each processor is converted into a global cumulative number of item value numbers, that is, a cumulative number commonly used among a plurality of processors. Finally, each processor changes the order of the allocated records by using the global cumulative number as a pointer. Allocation of records to be processed to a plurality of processors, counting the number of local appearances, and changing the order of allocated records can be processed in parallel by a plurality of processors. In addition, the global cumulative number may be calculated by using parallel processing of multiple processors, but because the cache hit rate is high because the memory can be accessed sequentially, only one or a part of the processors can be calculated. Responsible for maintaining high speed.

  Thus, Patent Document 2 discloses a memory shared parallel sort technique, but does not describe a memory shared parallel tabulation technique. The inventor has already proposed a memory distributed parallel tabulation technique as described in Patent Document 3. Regardless of the memory sharing type or the memory distributed type, the aggregation is, for example, an item value (item) having a certain item (dimension) from tabular data represented as an array of records including item values corresponding to information items. For each dimension value), a quantity (measure) based on the item value of another item is calculated. Calculation of a measure means counting the number of measures, calculating the sum of measures, or calculating an average value of measures. The number of dimensions may be two or more. For example, in the case of tabular data including an item of age, an item of height, and an item of weight, the process of calculating the average value of weight by age and height is based on age and height as the dimensions, and the average value of weight This is a tabulation process as a measure.

In such a tabulation process, it is necessary to first set a dimension and classify records for each dimension value. In order to classify records for each dimension value, it is possible to use a process for sorting records with respect to dimension values. In particular, in multidimensional tabulation, tabulation of entire tabular data is achieved by grouping records into sets of dimension values and calculating measures based on item values included in records belonging to the same group. Therefore, in a memory shared multiprocessor system, in order to tabulate large-scale tabular data at high speed, a technique for creating a dimension for tabulation, particularly for multi-dimensional tabulation, at high speed is required.
International Publication WO00 / 10103 International Publication WO2006 / 126467 International Publication WO2005 / 041067

  In view of the above-described problems of the prior art, an object of the present invention is to efficiently generate a set of unique item values representing a dimension for multidimensional aggregation from tabular data in a memory shared parallel processing system. It is to provide a method of extraction.

  In view of the above-described problems of the conventional technology, a further object of the present invention is to efficiently generate a set of unique item values representing a dimension for multidimensional aggregation from tabular data in a memory shared parallel processing system. It is to provide a device for extraction.

  It is another object of the present invention to provide a method and apparatus for efficiently executing multidimensional aggregation of tabular data in a memory shared parallel processing system.

In order to achieve the object of the present invention, according to one aspect of the present invention, a record number array in which record numbers of records of tabular data are stored in a predetermined record order, a predetermined number of records of the tabular data An item value number array in which item value numbers corresponding to item values of items are stored according to record numbers, and an item value array in which the item values of the above-mentioned tabular data are stored in the order of item value numbers corresponding to the item values A shared memory for storing
A plurality of processors capable of accessing the shared memory;
In a shared memory multiprocessor system with
A method for extracting a unique set of item values for a given set of items, comprising:
(A) Using the plurality of processors, the number of occurrences of item value number pairs corresponding to the item value pairs relating to the predetermined item set is counted in parallel to express the cumulative frequency distribution of the appearance times. Generating a cumulative frequency distribution array in the shared memory;
(B) Using the plurality of processors, using the values of the elements of the record number array as subscripts, the item value number pairs are read in parallel, and the values calculated from the item value number pairs are subscripted. Are used to read the elements of the cumulative frequency distribution array in parallel, and use the elements of the cumulative frequency distribution array as index pointers to convert the values of the elements of the record number array into the record number arrays of sorted versions (ie, sorted Stored in parallel in the record number array),
(C) partitioning the sorted record number array according to the value of the element of the cumulative frequency distribution array used as the index pointer into a group of record numbers of records having the set of unique item values;
(D) identifying one set of unique item values by extracting one record number from each group of the segmented record numbers;
A method comprising:

  Further, in the method of the present invention, the step (a) of generating the cumulative frequency distribution array in parallel in the shared memory is performed for each record number stored in the record number array using the plurality of processors. In addition, by setting the radix according to the range of the item value numbers included in the set of item value numbers corresponding to the set of item values related to the set of predetermined items, the set of item value numbers is A step of converting each item value number included in the set of value numbers into a virtual item value number in a radix format expressing each digit is provided.

Further, the method of the present invention includes the step (i) of generating the cumulative frequency distribution array in parallel in the shared memory for each item included in the predetermined item set, and the predetermined item set. (Ii) storing the values of the elements of the record number array in parallel in the sorted record number array for each item, and the sorted record number array for each item included in the predetermined set of items And (iii) dividing the record into groups of record numbers of records having the unique field value pairs,
The step (i) of generating the cumulative frequency distribution array in the shared memory in parallel uses the plurality of processors to count in parallel the number of occurrences of the item value number corresponding to the item value related to the item, Generating a cumulative frequency distribution array representing the cumulative frequency distribution of the occurrence counts in parallel in the shared memory,
The step (ii) of storing the elements of the record number array in parallel in the sorted record number array is performed using the plurality of processors and the values of the element of the item value number array as subscripts. Read the item value number in parallel, read the element of the cumulative frequency distribution array in parallel using the item value number as a subscript, and use the element of the cumulative frequency distribution array as an index pointer in the record number array A step of storing the element values in parallel in the sorted record number array is provided.

  Furthermore, the method of the present invention further includes the step of (e) totalizing item values related to different items for each set of the predetermined item values in parallel using the plurality of processors. .

Further, according to the method of the present invention, (f) using the plurality of processors, the record number in the sorted record number array and the group number of the group corresponding to the record number are used as the subscript. And storing in parallel in an auxiliary record number array and an auxiliary group number array;
(G) Using the plurality of processors, elements in the item value number array are read out in parallel from the item value number array relating to the other item using the elements in the auxiliary record number array as subscripts. Is used as a subscript to read the item value in parallel from the item value array for the other item, and the auxiliary record number array element is used as a subscript to read the auxiliary group number array element. Storing the aggregation result of the read item value in the aggregation array using an element of a typical group number array as a subscript of the aggregation array;
Is further provided.

  Furthermore, the method of the present invention includes (h) using the plurality of processors to read elements in parallel from the item value number array relating to the other item using the elements of the sorted record number array as subscripts. , Reading the item value in parallel from the item value array relating to the other item using the element of the item value number array as a subscript, and the group number of the group corresponding to the element of the sorted record number array is The method further comprises the step of storing the totaled result of the read item values in the totaling array using the subscript as a subscript.

In order to achieve the further object of the present invention, according to one aspect of the present invention, a record number array in which record numbers of records in tabular data are stored in a predetermined record order, a predetermined number of records in the tabular data, An item value number array in which item value numbers corresponding to item values of items in the table are stored according to the record number, and an item value in which the item values of the tabular data are stored in the order of the item value numbers corresponding to the item values A shared memory for storing the array;
A plurality of processors capable of accessing the shared memory;
In a shared memory multiprocessor system with
A device that extracts a unique set of item values for a given set of items,
(A) Using the plurality of processors, the number of occurrences of a set of item value numbers corresponding to the set of item values relating to the set of predetermined items is counted in parallel, and the cumulative frequency distribution of the number of appearances is expressed. Means for generating a cumulative frequency distribution array in the shared memory;
(B) Using the plurality of processors, using the values of the elements of the record number array as subscripts, the item value number pairs are read in parallel, and the values calculated from the item value number pairs are subscripted. Are used to read the elements of the cumulative frequency distribution array in parallel, and use the elements of the cumulative frequency distribution array as index pointers to change the values of the elements of the record number array to the sorted version of the record number array (i.e., sorted A record number array) in parallel,
(C) means for dividing the sorted record number array into record number groups of records having the unique item value pairs according to the values of the elements of the cumulative frequency distribution array used as the index pointer;
(D) means for identifying the set of unique item values by extracting one record number from each group of the divided record numbers;
An apparatus comprising:

Furthermore, according to one aspect of the present invention, a record number array in which the record numbers of the records in the tabular data are stored according to a predetermined record order, and the item values corresponding to the item values of the predetermined items in the records of the tabular data An item value number array in which numbers are stored according to record numbers, and a shared memory for storing item value arrays in which the item values of the tabular data are stored in accordance with the order of the item value numbers corresponding to the item values
A plurality of processors capable of accessing the shared memory;
A computer-readable program that causes a computer to execute code for extracting a unique set of item values for a predetermined set of items, loaded on a computer comprising:
(A) Using the plurality of processors, the number of occurrences of item value number pairs corresponding to the item value pairs relating to the predetermined item set is counted in parallel to express the cumulative frequency distribution of the appearance times. Code for generating a cumulative frequency distribution array in the shared memory;
(B) Using the plurality of processors, using the values of the elements of the record number array as subscripts, the item value number pairs are read in parallel, and the values calculated from the item value number pairs are subscripted. Are used to read the elements of the cumulative frequency distribution array in parallel, and use the elements of the cumulative frequency distribution array as index pointers to convert the values of the elements of the record number array into the record number arrays of sorted versions (ie, sorted Code stored in parallel in the record number array)
(C) a code that divides the sorted record number array according to the value of the element of the cumulative frequency distribution array used as the index pointer into a group of record numbers of records having the unique item value pairs;
(D) a code for identifying a set of unique item values by extracting one record number from each group of the record numbers that are classified;
Is provided.

  Furthermore, according to one aspect of the present invention, a record number array in which the record numbers of the records in the tabular data are stored according to a predetermined record order, and the item values corresponding to the item values of the predetermined items in the records of the tabular data A shared memory for storing an item value number array in which numbers are stored according to record numbers, and an item value array in which the item values of the tabular data are stored in accordance with the order of the item value numbers corresponding to the item values; A computer program product for causing a computer to execute the method of the present invention for extracting a set of unique item values relating to a predetermined set of items, loaded on a computer comprising a plurality of processors capable of accessing a memory Is provided.

  Furthermore, according to one aspect of the present invention, a record number array in which the record numbers of the records in the tabular data are stored according to a predetermined record order, and the item values corresponding to the item values of the predetermined items in the records of the tabular data A shared memory for storing an item value number array in which numbers are stored according to record numbers, and an item value array in which the item values of the tabular data are stored in accordance with the order of the item value numbers corresponding to the item values; A computer program for causing a computer to execute the method of the present invention, which is loaded into a computer having a plurality of processors capable of accessing a memory and extracts a set of unique item values relating to a predetermined set of items. A recorded storage medium is provided.

  According to the present invention, it is possible to realize multidimensional tabulation processing of large-scale tabular data at high speed.

FIG. 1 is a schematic diagram of a computer system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of tabular data for explaining the data management mechanism. FIG. 3 is an explanatory diagram of a data management mechanism according to an embodiment of the present invention. FIG. 4 is a flowchart of multidimensional tabulation processing according to an embodiment of the present invention. FIG. 5 is an explanatory diagram of multi-item batch sort processing according to an embodiment of the present invention. FIG. 6 is an explanatory diagram of processing for calculating a virtual item value number according to an embodiment of the present invention. FIG. 7 is an explanatory diagram of processing for counting up virtual item value numbers according to an embodiment of the present invention. FIG. 8 is an explanatory diagram of processing for generating a cumulative frequency distribution array according to an embodiment of the present invention. FIG. 9 is an explanatory diagram of processing for rearranging the elements of the record number array according to one embodiment of the present invention. FIG. 10 is an explanatory diagram of the result of the process of rearranging the elements of the record number array according to one embodiment of the present invention. FIG. 11 is an explanatory diagram of unique row extraction processing according to an embodiment of the present invention. FIG. 12 is an explanatory diagram of a process of rearranging a set of unique item values extracted by the unique row extraction process according to an embodiment of the present invention in the order of appearance in the original record number array. FIG. 13 is an explanatory diagram of processing for rearranging a set of unique item values extracted by the unique row extraction processing according to an embodiment of the present invention in the order in which they appear in the original record number array. FIG. 14 is a schematic explanatory diagram of multi-item sequential sorting processing according to an embodiment of the present invention. FIG. 15 is an explanatory diagram of the parallel count-up process in the low-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. FIG. 16 is an explanatory diagram of the cumulative frequency distribution array generation process in the low-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. FIG. 17 is an explanatory diagram of the process of rearranging the elements of the record number array in the lower-order sort process of the multi-item sequential sort process according to one embodiment of the present invention. FIG. 18 is an explanatory diagram of the parallel count-up process in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. FIG. 19 is an explanatory diagram of the cumulative frequency distribution array generation process in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. FIG. 20 is an explanatory diagram of the process of rearranging the elements of the record number array in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. FIG. 21 is an explanatory diagram of the rearrangement process in the measure number ascending order type measure parallel calculation process according to an embodiment of the present invention. FIG. 22 is an explanatory diagram of the aggregation process in the measure parallel calculation process of the record number ascending order type according to one embodiment of the present invention. FIG. 23 is an explanatory diagram of a counting process in the group number ascending type measure parallel calculation process according to an embodiment of the present invention. 24A and 24B are explanatory diagrams of the data structure to be sorted in the multi-stage parallel sort according to the first embodiment. FIG. 25 is a flowchart of the multistage parallel sorting method according to the first embodiment. 26A and 26B are explanatory diagrams (part 1) of the count-up step in the first stage of the multi-stage parallel sorting method according to the first embodiment. 27A and 27B are explanatory diagrams (part 2) of the count-up step in the first stage of the multi-stage parallel sorting method according to the first embodiment. 28A and 28B are explanatory diagrams (part 3) of the count-up step in the first stage of the multi-stage parallel sorting method according to the first embodiment. FIGS. 29A and 29B are explanatory diagrams of the first-stage cumulative numbering step of the ascending order multi-stage parallel sorting method according to the first embodiment. 30A and 30B are explanatory diagrams (part 1) of the transfer step in the first stage of the ascending multi-stage parallel sort method according to the first embodiment. FIGS. 31A and 31B are explanatory diagrams (part 2) of the transfer step in the first stage of the ascending order multi-stage parallel sort method according to the first embodiment. FIGS. 32A and 32B are explanatory diagrams (part 3) of the transfer step of the first stage of the ascending order multi-stage parallel sort method according to the first embodiment. FIG. 33 is an explanatory diagram of the initialization step of the second stage of the multi-stage parallel sorting method according to the first embodiment. FIGS. 34A and 34B are explanatory diagrams (part 1) of the second-stage count-up step of the multi-stage parallel sorting method according to the first embodiment. FIGS. 35A and 35B are explanatory diagrams (part 2) of the second count-up step of the multi-stage parallel sorting method according to the first embodiment. 36A and 36B are explanatory diagrams (part 3) of the count-up step in the second stage of the multi-stage parallel sorting method according to the first embodiment. FIG. 37 is an explanatory diagram of the second-stage accumulation step in the ascending multi-stage parallel sorting method according to the first embodiment. 38A and 38B are explanatory diagrams (part 1) of the second-stage transfer step of the ascending multi-stage parallel sort method according to the first embodiment. FIGS. 39A and 39B are explanatory diagrams (part 2) of the second-stage transfer step of the ascending multi-stage parallel sort method according to the first embodiment. 40A and 40B are explanatory diagrams (part 3) of the second-stage transfer step of the ascending multi-stage parallel sort method according to the first embodiment. FIGS. 41A to 41C are diagrams (part 1) illustrating a result of applying the ascending multi-stage parallel sorting method according to the first embodiment. 42A and 42B are diagrams (part 2) illustrating a result of applying the ascending multi-stage parallel sorting method according to the first embodiment. FIG. 43 is a data structure diagram for explaining multi-stage sorting according to the second embodiment. FIGS. 44A to 44D are explanatory diagrams of count-up and cumulative numbering in the first stage of multi-stage sorting according to the second embodiment. 45A and 45B are explanatory diagrams of record number transfer in the first stage of multistage sorting according to the second embodiment. FIG. 46 is an explanatory diagram of the result of record number transfer in the first stage of multi-stage sorting according to the second embodiment. FIG. 47 is a diagram illustrating an initial state of the second stage of the multistage sorting according to the second embodiment. 48A to 48C are explanatory diagrams of the second stage count-up and cumulative numbering in the multi-stage sort according to the second embodiment. 49A and 49B are explanatory diagrams of record number transfer in the second stage of multistage sorting according to the second embodiment. 50A to 50C are diagrams for explaining the contents of the tabulation processing in the third embodiment. FIG. 50A shows an output tabulation result table, FIG. 50B shows a tabulation source table composed of 1 million rows, and FIG. This represents a total result table composed of 632334 records. 51A-C are diagrams showing the measurement results of Example 3, FIG. 51A shows the required time (milliseconds) of processing by the five methods of Example 3, and FIG. 51B shows the method according to the present invention in the old method. FIG. 51C is a graph showing the same high speed magnification as FIG. 51B. 52A-C are diagrams for explaining the contents of the tabulation process in the fourth embodiment. FIG. 52A shows an output tabulation result table, FIG. 52B shows a tabulation source table consisting of 1 million rows, and FIG. This represents a total result table composed of 632334 records. 53A-C are diagrams showing the measurement results of Example 4, FIG. 53A shows the required time (milliseconds) of processing by the five methods of Example 4, and FIG. 53B shows that the method according to the present invention is changed to the old method. FIG. 53C is a graph showing the same high speed magnification as FIG. 53B.

Explanation of symbols

10 Computer system 12-1, 12-2, ..., 12-p CPU
14 Shared memory 16 ROM
18 Fixed storage device 20 CD-ROM driver 22 I / F
24 input device 26 display device

  Hereinafter, various embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[Computer system configuration]
FIG. 1 is a schematic diagram of a computer system 10 that performs tabular data tabulation processing according to an embodiment of the present invention. As shown in FIG. 1, the computer system 10 includes p processors (CPU) 12-1, 12-2,... That control the entire system and individual components by executing a program. . . 12-p, for storing work data and the like, for example, a shared memory 14 such as a RAM (Random Access Memory), a ROM (Read Only Memory) 16 for storing a program and the like, a fixed storage medium 18 such as a hard disk, and the like A CD-ROM driver 20 for accessing the CD-ROM 19; an interface (I / F) 22 connected to an external terminal connected to the CD-ROM driver 20 and an external network (not shown); An input device 24 such as a keyboard and a mouse, and a display device 26 such as a computer monitor are provided. The CPU 12, RAM 14, ROM 16, external storage medium 18, I / F 22, input device 24, and display device 26 are connected to each other via a bus 28. Although not shown, each CPU has a unique local memory, and a cache memory is mounted on the local memory.

  A program for causing the computer system 10 to execute the method according to the present embodiment may be stored in the CD-ROM 19 and read by the CD-ROM driver 20 or may be stored in the ROM 16 in advance. Further, what is once read from the CD-ROM 19 may be stored in a predetermined area of the external storage medium 18. Alternatively, the program may be supplied from the outside via a network (not shown), an external terminal, and the I / F 22.

  In addition, the memory sharing multiprocessor system according to the embodiment of the present invention is realized by causing the computer system 10 to execute a program that tabulates tabular data.

[Data management mechanism based on information blocks]
FIG. 2 is a diagram illustrating an example of tabular data for explaining the data management mechanism. This tabular data is stored in the computer as a data structure as shown in FIG. 3 by using the data management mechanism proposed in the above-mentioned International Publication No. WO00 / 10103. This data structure has been proposed to realize retrieval, sorting, aggregation, etc. of large-scale tabular data using hardware resources of commercially available computers, for example, personal computers, in particular, processors and memories. It should be noted that the data structure of tabular data placed on the computer memory.

  As shown in FIG. 3, a record number array 301 (hereinafter, this array is abbreviated as “OrdSet”) that associates the order number of the records in the tabular data with the order number of the internal data. In the table format, the order number of the internal data is arranged as a value for each record in the table format. In this example, since all tabular data is represented as internal data, the record number of the tabular data matches the arrangement order number of the internal data.

  Here, the notation of the arrangement | sequence in this specification is demonstrated. In general, an array element A can be expressed as A [i], where i is a subscript. However, in the drawing, an array element A [i] is within a region surrounded by a solid line. The boundary between the element A [i] and the element A [i + 1] is indicated by a dotted line. The subscript i of element A [i] is shown on the left side of element A [i]. Further, the subscript i of the array is represented by an integer starting from 0.

  Returning to FIG. 3 again, regarding gender, it can be seen that the arrangement order number of the internal data corresponding to the record 0 of the tabular data is “0” from the array OrdSet 301. The actual gender value relating to the record whose arrangement order number is “0”, that is, “male” or “female” is an item value array 303 (hereinafter referred to as a value list) in which the actual values are sorted in a predetermined order. The item value array 302, which is a pointer array to the item value array, that is, the value list is abbreviated as “VL” (hereinafter, the item value number array, ie, the pointer array is abbreviated as “VNo”). Can be obtained by referring to The pointer array 302 stores pointers that point to elements in the actual value list 303 in the order of the arrangement order numbers stored in the array OrdSet 301. As a result, the gender item value corresponding to the record “0” of the tabular data is (1) the arrangement order number “0” corresponding to the record “0” is extracted from the array OrdSet 301 and (2) a pointer to the value list The element “1” corresponding to the sequence number “0” is extracted from the array 302, and (3) the element “woman” indicated by the element “1” extracted from the value array 303 from the pointer array 302 to the value list. Can be obtained by taking out

  The item values can be acquired in the same manner for other records and also for age and height.

  In this way, the tabular data is expressed by a combination of the value list VL and the pointer array VNo to the value list, and this combination is particularly referred to as an “information block”. In FIG. 3, information blocks regarding gender, age, and height are shown as information blocks 308, 309, and 310, respectively.

  If a single computer is a single memory (which can be physically multiple, but a single memory in the sense that it is located and accessed in a single address space) An ordered set array OrdSet, a value list VL constituting each information block, and a pointer array VNo may be stored. However, in order to hold a large number of records, the memory capacity increases with the size, so it is desirable that these large numbers of records can be processed in parallel.

  Therefore, in this embodiment, a plurality of processors access the record data stored in the shared memory, and high-speed aggregation is realized by parallel processing of the plurality of processors.

[Multidimensional aggregation processing]
Next, multidimensional tabulation processing in the memory shared multiprocessor system according to the present embodiment will be described. The aggregation process according to the present embodiment is achieved essentially by a plurality of processors executing the same process. In other words, this aggregation process is configured so that a plurality of processors can operate in parallel and execute the aggregation process by giving a single aggregation processing instruction to a plurality of processors. Since a plurality of processors execute the same operation, parallel processing can be realized only by creating one program.

  FIG. 4 is a flowchart of multi-dimensional tabulation processing of tabular data according to an embodiment of the present invention. This multidimensional tabulation process is executed by a computer system 10 as shown in FIG. The shared memory 14 of the computer system 10 includes a record number array in which the record numbers of the tabular data records are stored in a predetermined record order, and an item value number corresponding to the item value of the predetermined item of the record of the tabular data. Is stored according to the record number, and the item value array in which the item values of the tabular data are stored according to the order of the item value numbers corresponding to the item values is stored. The plurality of processors 12-1,..., 12-p can access the shared memory 14.

  Each processor enumerates in parallel the number of occurrences of the set of item value numbers corresponding to the set of item values relating to the set of predetermined items (step 401), and a cumulative frequency distribution array expressing the cumulative frequency distribution of the number of appearances Is generated in the shared memory (step 402). The predetermined item set corresponds to a dimension set used in multi-dimensional aggregation, and the item value set corresponds to a dimension value set.

  Next, each processor reads the item value number set in parallel using the element value of the record number array as a subscript, and uses the value calculated by the item value number set as a subscript to perform the accumulation. Reads the elements of the frequency distribution array in parallel and uses the elements of the cumulative frequency distribution array as an index pointer to convert the values of the elements of the record number array into a sorted version of the record number array (ie, the sorted record number array) (Step 403). For example, a set of item value numbers can be converted to a value expressed in a radix so that each item value number corresponds to each digit. The conversion to the radix representation is accelerated by using a bit shift operation instead of multiplication. However, the conversion from a pair of item value numbers to a value is not limited to the radix representation, and any conversion can be used as long as it can be converted back from the converted value to the original set of item value numbers. I do not care. As a result, the elements of the original record number array are sorted with respect to the set of item value numbers corresponding to the set of item values, and a sorted version of the record number array in which the sorted elements are stored is generated.

  Further, at least one processor converts the record number array of the sorted version into the record number group of the record having the unique item value pair according to the value of the element of the cumulative frequency distribution array used as the index pointer. Partition (step 404). As a result, the records are classified into groups of records having the same set of field values, that is, the same set of dimension values.

  At least one processor identifies one set of the unique item values by extracting one record number from each group of the divided record numbers (step 405). Thereby, a unique set of dimension values used in multidimensional aggregation is specified.

  Finally, each processor totals item values related to different items for each set of the predetermined item values in parallel (step 406).

  In the following, multidimensional parallel sort processing (multi-item batch sort processing or multi-item sequential sort processing), unique row extraction processing, and measure parallel calculation processing (record number ascending order summarization processing or group number) used in multidimensional parallel tabulation processing (Ascending process) will be described in detail.

  Furthermore, in various embodiments of the present invention, in order to fully demonstrate the features of the symmetric multiprocessor system, the processing applied to the various arrays is performed by assigning each array to a plurality of processors substantially evenly. Parallel processing is performed by a plurality of processors. In the following, in order to help the understanding of the invention, it is assumed that the number of processors n is four, but it will be apparent to those skilled in the art that the number of processors is not limited to four.

[Multi-item batch sort processing]
The multi-item batch sort process is a process that batch-sorts a plurality of items of tabular data, and (i) the sort process is accelerated, (ii) the program description is simplified, (iii) ) There is an advantage that the extraction of records having the same set of field values is simplified.

  FIG. 5 is an explanatory diagram of multi-item batch sort processing according to an embodiment of the present invention. The tabular data 501 shown in the figure includes 16 records whose record numbers are represented by 0 to 15, and each record includes an item Age “age” and an item class “class”. ing. The item “age” corresponds to the first dimension Dim-1 of the multidimensional aggregation, and the item “class” corresponds to the second dimension Dim-2 of the multidimensional aggregation. In this embodiment, the tabular data is referred to as a record number array 503, an item value number array 511 and an item value array 512 for the first dimension, and an item value number array 521 and an item value array 522 for the second dimension. It is constructed on the shared memory 14 in the form of a data structure 502.

  Such a tabular data is a table such as that shown on the right side of the arrow in the figure when “Class” is the lower dimension, “Age” is the upper dimension, and the “Class” and “Age” dimensions are sorted. Converted to format data. The sort order matches the order of the item values stored in the item value array of each item. Therefore, in the example of FIG. 3, “Age” is sorted in ascending order, and “Class” is sorted in alphabetical order. However, since the actual sorting is performed not on the item value itself but on the item value number corresponding to the item value, this embodiment has an advantage that it can be applied regardless of the data format of the actual data.

  The tabular data 551 is obtained by changing the order of the rows of tabular data according to the result of the multi-item batch sort. In this embodiment, this sort result is expressed by changing the order of elements from the record number array 503 to the record number array 553. In the description of the present embodiment, the number of dimensions of the multi-item batch sort process is described as 2, but it will be apparent to those skilled in the art that the order is not limited to 2.

  Next, the multi-item batch sort process will be described in detail using an example of the tabular data shown in FIG.

  The multi-item batch sort process generates a new virtual item value number (hereinafter referred to as a virtual item value number) by combining each item value related to each of a plurality of items, and based on this virtual item value number This process sorts record numbers at a time. The combination of field value numbers is a virtual field value number by expressing the field value number pair in radix format so that the field value number of the higher dimension is the upper digit and the field value number of the lower dimension is the lower digit. Is converted to The radix is determined according to the range of values that each item value number can take. For example, if the item value number of the first dimension takes a value from 0 to 2 and the item value number of the second dimension takes a value from 0 to 3, as in the example of FIG. By setting the above integers, the combination of item value numbers can be converted into virtual item value numbers expressed in a radix format with a base of 4.

  FIG. 6 is a diagram for explaining calculation of a virtual item value number according to an embodiment of the present invention. In the present embodiment, each of the four processors CPU-0, CPU-1, CPU-2, and CPU-3 is responsible for each part of the item value number array having a size that matches the number of record numbers. A virtual item value number is generated by combining the item value number of the first dimension and the item value number of the second dimension.

Specifically, i represents a record number (in this example, i = 0, 1, 2,..., 15), and an item value number array of the first dimension is designated as VNo_1 (511 in FIGS. 5 and 6). The second dimension item value number array VNo_2 (indicated by 521 in FIGS. 5 and 6), and the virtual item value number array V_VNo (indicated by 601 in FIG. 6). ,
CPU-0 is in charge range 602, i.e., i = 0, 1, 2, 3,
V_VNo [i] = VNo_1 [i] * 4 + VNo_2 [i]
Similarly, CPU-1 is in charge range 603, i.e., for i = 4, 5, 6, 7
V_VNo [i] = VNo_1 [i] * 4 + VNo_2 [i]
CPU-2 calculates the range of responsibility 604, i.e., i = 8, 9, 10, 11
V_VNo [i] = VNo_1 [i] * 4 + VNo_2 [i]
CPU-3 calculates the range of responsibility 605, i.e., i = 12, 13, 14, 15,
V_VNo [i] = VNo_1 [i] * 4 + VNo_2 [i]
Calculate

  In the above description, the calculation of the virtual item value number array is realized by multiplication and addition. However, in order to speed up the calculation, a bit shift operation is preferably employed instead of multiplication.

  Next, each processor counts up the number of occurrences of the virtual item value number with respect to the elements within the range of the record number array that it handles. FIG. 7 is an explanatory diagram of processing for counting up virtual item value numbers according to an embodiment of the present invention. This counting process is also shared by the processor. In this example, the record number array 503 is divided into the respective handling ranges 701, 702, 703, and 704 of the four processors.

Count that stores the number of virtual item value numbers counted by processor j, with record number i, record number array OrdSet, virtual item value number array V_VNo, processor number j (j = 0, 1, 2, 3) When the array is Count [j], the counting process according to the present embodiment is, for example, in C language style.
for (i = 0; i <4; i ++) {
for (j = 0; j <4; j ++) {
Count [j] [V_VNo [OrdSet [j * 4 + i]]] ++;
}
}
Is described. In the example of FIG. 7, the initialized count arrays 710, 711, 712, and 713 become count arrays 720, 721, 722, and 723 by the virtual item value number counting process. In this example, when the first item value array is VL_1 and the second item value array is VL_2, the size of the count array Count is equal to the size of (VL_1 size) × (VL_2).

  As a result of the counting process, for example, one virtual rank number = 0, 2, 5, and 9 exists in the CPU-0 range, and in the CPU-1 range, the virtual rank number = There is one each of 6, 7, 10 and 11, and within the CPU-2 range, there is one virtual rank number 0, 3, 4 and 5, and within the CPU-3 range. , The virtual rank numbers = 2, 5, 10 and 11 each exist.

Subsequently, in one embodiment of the present invention, the computer system 10 accumulates each element of the virtual item value number array to generate a cumulative frequency distribution array of the number of appearances of the virtual item value number. FIG. 8 is an explanatory diagram of processing for generating a cumulative frequency distribution array according to an embodiment of the present invention. When the count array within the range of the processor j is Count [j] and the cumulative frequency distribution array is Aggr [j], the cumulative frequency distribution is
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Aggr [3] [0] + Count [3] [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Aggr [3] [1] + Count [3] [1]
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・
Omitted along the way ...
Aggr [3] [10] = Aggr [2] [10] + Count [2] [10]
Aggr [0] [11] = Aggr [3] [10] + Count [3] [10]
Aggr [1] [11] = Aggr [0] [11] + Count [0] [11]
Aggr [2] [11] = Aggr [1] [11] + Count [1] [11]
Aggr [3] [11] = Aggr [2] [11] + Count [2] [11]
Is calculated as This calculation may be performed by any one of the four processors alone, but the range of virtual item value numbers is assigned to the four processors and performed in parallel by the four processors. Also good.

When the cumulative frequency distribution is calculated in parallel by four processors, first, the total value Sum [i] of the number of occurrences of each virtual item value number i is, for example,
for (i = 0; i <12; i ++) {
for (j = 0; j <4; j ++) {
Sum [i] = Count [j] [i];
}
}
To calculate in parallel. In this embodiment, for example, CPU-0 is in charge of a loop of i = 0, 1, 2, CPU-1 is in charge of a loop of i = 3, 4, 5, and CPU-2 is i = 6, 7 , 8 and CPU-3 is responsible for i = 9, 10, 11 loops.

Subsequently, for example, four processors are connected in parallel,
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Sum [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Sum [1]
・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・ ・
Omitted along the way ...
Aggr [3] [10] = Aggr [2] [10] + Count [2] [10]
Aggr [0] [11] = Sum [10]
Aggr [1] [11] = Aggr [0] [11] + Count [0] [11]
Aggr [2] [11] = Aggr [1] [11] + Count [1] [11]
Aggr [3] [11] = Aggr [2] [11] + Count [2] [11]
To obtain a cumulative frequency distribution array.

  FIG. 8 shows final cumulative frequency distribution arrays 801, 802, 803, and 804.

In the multi-item batch sort process according to the embodiment of the present invention, finally, the elements of the original record number array are transferred to the sorted record number array. FIG. 9 is an explanatory diagram of the process of rearranging the elements of the record number array according to one embodiment of the present invention. In this process, the record number array 503 is divided into four processor ranges 701, 702, 703, and 704 as in FIG. 7, and the four processors execute parallel processing. If the original record number array is OrdSet, the sorted record number array is OrdSet ', the virtual item value number array is V_VNo, and the cumulative frequency distribution array is Aggr [j] (j is a processor number), processor 0, The reordering process by CPU-0 is
for (i = 0; i <4; i ++) {
OrdSet ′ [Aggr [0] [V_VNo [OrdSet [i]]]] = OrdSet [i];
Aggr [0] [V_VNo [OrdSet [i]]] ++;
}
Can be described. Here, Aggr [0] [V_VNo [OrdSet [i]]] ++ indicates that the elements of the cumulative frequency distribution array are used as index pointers. For example, in FIG. 8, for record number 0,
OrdSet [0] = 0
V_VNo [OrdSet [0]] = V_VNo [0] = 5
Aggr [0] [V_VNo [OrdSet [0]]] = Aggr [0] [5] = 6
OrdSet ′ [Aggr [0] [V_VNo [OrdSet [0]]]] = OrdSet ′ [6]
And therefore
OrdSet ′ [6] = 0
Aggr [0] [5] = 7
It has been shown that

Furthermore, assuming that the processor number j is a loop variable, the process of rearranging the elements of the record number array according to this embodiment is as follows:
for (j = 0; j <4; j ++) {
for (i = 0; i <4; i ++) {
OrdSet ′ [Aggr [j] [V_VNo [OrdSet [j * 4 + i]]]] = OrdSet [j * 4 + i];
Aggr [j] [V_VNo [OrdSet [j * 4 + i]]] ++;
}
}
Can be described.

  FIG. 10 is a diagram for explaining the result of the process of rearranging the elements of the record number array according to an embodiment of the present invention. Cumulative frequency distribution arrays 1001, 1002, 1003, and 1004 represent sorted cumulative frequency distribution arrays, and a sorted version record number array 1005 represents a sorted record number array. As described above, the element of the original record number array 503 is collectively sorted with respect to the two dimensions of the first dimension and the second dimension by the process of rearranging the elements of the record number array according to the embodiment of the present invention. And stored in the sorted version record number array 1005.

[Unique row extraction processing]
By the multi-item batch sort process, the records to be subjected to multi-dimensional aggregation are sorted in a predetermined order with respect to a plurality of dimensions. In order to perform multidimensional aggregation, it is necessary to specify a dimension value. Therefore, in one embodiment of the present invention, a representative record is extracted from records having the same dimension value. In this specification, the process of extracting such a record is also referred to as a unique row extraction process.

  In the unique row extraction process, a record in which any one of a plurality of processors has a set version of a record number array of a sorted version according to the value of an element of a cumulative frequency distribution array used as an index pointer. Are grouped into groups of record numbers, and one record number is extracted from each group of the segmented record numbers, thereby identifying a unique set of item values.

  When an element of the cumulative frequency distribution array is used as an index pointer when storing the record number in the record number array, the element of the cumulative frequency distribution array is incremented each time the record number is stored. As a result, the element of the cumulative frequency distribution array represents the position where the record number is stored in the record number array. Thus, after completion of the sorted version record number array, the value of the element of the cumulative frequency distribution array points to the boundary of the group of record numbers in the sorted version record number array.

FIG. 11 is an explanatory diagram of unique row extraction processing according to an embodiment of the present invention. Elements of the cumulative frequency distribution array after sorting, in this example, the elements of the cumulative frequency distribution array 1004 based on the count array generated by the CPU-4 in charge of the last range of the record number array when counting the number of appearances Can be used to divide record numbers into groups. More specifically, from the fact that the cumulative frequency distribution array is used as an index pointer, the elements of the sorted version of the record number array 1005 pointed to by each element of the final cumulative frequency distribution array 1004, that is,
OrdSet ′ [Aggr [3] [0]] = OrdSet ′ [2]
OrdSet ′ [Aggr [3] [1]] = OrdSet ′ [2]
OrdSet ′ [Aggr [3] [2]] = OrdSet ′ [4]
OrdSet ′ [Aggr [3] [3]] = OrdSet ′ [5]
OrdSet ′ [Aggr [3] [4]] = OrdSet ′ [6]
OrdSet ′ [Aggr [3] [5]] = OrdSet ′ [9]
OrdSet ′ [Aggr [3] [6]] = OrdSet ′ [10]
OrdSet ′ [Aggr [3] [7]] = OrdSet ′ [11]
OrdSet ′ [Aggr [3] [8]] = OrdSet ′ [11]
OrdSet ′ [Aggr [3] [9]] = OrdSet ′ [12]
OrdSet ′ [Aggr [3] [10]] = OrdSet ′ [14]
OrdSet ′ [Aggr [3] [11]] = OrdSet ′ [16]
Represents the boundary of a group of records having a unique set of field values.

Thus, in the example of FIG. 11, the record number is
Group 0 = {1, 9} (refer to 1101 in the figure)
Group 1 = {3, 15} (refer to 1102 in the figure)
Group 2 = {11} (refer to 1103 in the figure)
Group 3 = {8} (Refer to 1104 in the figure)
Group 4 = {0, 10, 14} (refer to 1105 in the figure)
Group 5 = {5} (See 1106 in the figure)
Group 6 = {4} (Refer to 1107 in the figure)
Group 7 = {2} (refer to 1108 in the figure)
Group 8 = {6,12} (refer to 1109 in the figure)
Group 9 = {7, 13} (refer to 1110 in the figure)
It can be seen that it is classified into 10 groups.

  Since there are only 10 groups in this way, a unique set of item values is extracted by extracting one record number per group, that is, a total of 10 record numbers as representative record numbers. Can do.

  In FIG. 11, a representative record number array OrdSet ″ including the extracted representative record number is shown as 1120.

Next, i is a loop variable, VL_1 and VNo_1 are the first dimension item value array and item value number array, VL_2 and VNo_2 are the second dimension item value array and item value number array, If the dimension value array of the dimension is Dim-1, and the dimension value array of the second dimension is Dim-2, for example,
for (i = 0; i <10; i ++) {
Dim-1 [i] = VL_1 [VNo_1 [OrdSet ''[i]]];
Dim-2 [i] = VL_2 [VNo_2 [OrdSet ''[i]]];
}
Is executed in parallel by a plurality of processors to create a total dimension 1130.

  In the above unique row extraction process, unique item value pairs are extracted in the order in which the item values are sorted. For example, in order to extract unique item value pairs in the order of appearance in the original record number array OrdSet, the elements of the representative record number array may be rearranged in the order of appearance in the original record number array. FIGS. 12 and 13 are explanatory diagrams of processing for rearranging the sets of unique item values extracted by the unique row extraction processing according to the embodiment of the present invention in the order of appearance in the original record number array.

  In this process, a bitmap flag corresponding to the record number stored in the representative record number array is created, and the bitmap flag is checked in the order of the original record number, and the record number in which the flag is set is changed to the original number. Extracting in the order of the record numbers stored in the record number array. These steps can be processed in parallel by assigning the representative record number array and the original record number array to a plurality of (four in this example) CPUs.

  Referring to FIG. 12, the representative record number array OrdSet ″ 1120 includes a CPU-0 coverage range 1200, a CPU-1 coverage range 1201, a CPU-2 coverage range 1202, and a CPU-3 coverage range 1203. It is divided into and. Each processor reads an element within the range of the processor itself, that is, a record number, from the representative record number array 1120, and the record number read in each bitmap flag array 1210, 1211, 1212, and 1213. Set the bit corresponding to. For example, the CPU-0 reads out three record numbers of OrdSet ″ [0] = 1, OrdSet ″ [1] = 3 and OrdSet ″ [2] = 11, and outputs three in the bitmap flag array. Set the flag. That is, Flag_0 [1] = 1, Flag_0 [3] = 1, and Flag_0 [11] = 1. Other elements of the bitmap flag array Flag_0 remain 0. Similarly, CPU-1, CPU-2, and CPU-3 set bitmap flag arrays Flag_1, Flag_2, and Flag_3, respectively. These processes are executed in parallel by a plurality of processors.

Next, each of the bitmap flag arrays 1210, 1211, 1212 and 1213 is assigned to four processors, CPU-0, CPU-1, CPU-2 and CPU-3, in order from the top, A bitmap flag array Flag is created. Specifically, each processor performs an OR operation in parallel on each of the responsible ranges 1220, 1221, 1222, and 1223, so that as a whole,
for (i = 0; i <16; i ++) {
Flag [i] = Flag_1 [i] OR Flag_2 [i] OR Flag_3 [i] OR Flag_4 [i];
}
Is executed. As a result, the overall bitmap flag array 1230 of FIG. 12 is obtained.

  Referring to FIG. 13, each processor sets the flags of the elements of the overall bitmap flag array 1230 in the order of the elements in the ranges 701, 702, 703, and 704 of the overall original record number array 503. The record number for which the flag is set is stored in the work array 1300 in the order of check, and finally the work array is stored in the further representative record number array 1310. The further representative record number array 1310 is an array in which the representative record numbers are stored in the order of the record numbers stored in the original record number array.

Next, i is a loop variable, VL_1 and VNo_1 are the first dimension item value array and item value number array, VL_2 and VNo_2 are the second dimension item value array and item value number array, If the dimension value array of the dimension is Dim-1, and the dimension value array of the second dimension is Dim-2, for example,
for (i = 0; i <10; i ++) {
Dim-1 [i] = VL_1 [VNo_1 [OrdSet '''[i]]];
Dim-2 [i] = VL_2 [VNo_2 [OrdSet '''[i]]];
}
Is executed in parallel by a plurality of processors, thereby creating a total dimension 1320.

[Multi-item sequential sorting]
The multi-item parallel sort process includes a multi-item sequential sort process described below in addition to the above-described multi-item batch sort process. For example, when the number of bits of the virtual item value number exceeds the range of integers that can be processed on the computer, the multi-item batch sort process cannot be applied, so this multi-item sequential sort process is applied. Hereinafter, the multi-item sequential sorting process according to the embodiment of the present invention will be described using the tabular data in FIG. In the following example, the number of dimensions is described as 2. However, it is understood by those skilled in the art that multi-item sequential sort processing of any order can be realized by multi-stage multi-item sequential sort processing of dimension 2 It will be obvious.

  In the multi-item sequential sort process, the sort related to the items in the lower dimension is executed first than the sort related to the items in the higher dimension. The concept of the upper dimension and the lower dimension is as described for the multi-item batch sort process. Then, by introducing the concept of the group number described for the unique row extraction process in each of the lower dimension sort and the upper dimension sort, the grouping is performed without performing the comparison operation. By sorting the group numbers obtained by sorting the lower dimensions together with the record numbers when sorting the upper dimensions, the grouping of the upper dimensions is simplified. If it is guaranteed that the item value number is used continuously from 0, it is possible to use the item value number as a group number. However, in general, it is actually used by adding / deleting data. Since the item value number being used may become a skipped value, the group number is preferably used separately from the item value number.

  FIG. 14 is a schematic explanatory diagram of multi-item sequential sorting processing according to an embodiment of the present invention. The tabular data 501 before the sorting process is the same as the data shown in FIG. When the tabular data is sorted in the ascending order (for example, alphabetical order) of the item values of “class” with respect to the item “class” which is the lower dimension, tabular data 1400 sorted in the lower dimension is obtained. At this time, a first group number array 1401 based on the item value of “class” is assigned to the tabular data 1400.

  Next, when the tabular data 1400 and the first group number array 1401 are sorted in ascending order of the item value of the item “age” which is the higher dimension, the sorted tabular data 1410 and the first saw-shaped data are sorted. A group number array 1411 is obtained. At this time, a new second group number array 1412 is assigned to the tabular data 1410 based on the sorting result.

  Finally, referring to the second group number array 1412 and extracting a unique row from the tabular data 1410, the same total dimension 1130 as the total dimension created by the multi-item batch sort process is obtained.

  Hereinafter, the multi-item sequential sorting process according to an embodiment of the present invention will be described in more detail with reference to the drawings. FIG. 15 is an explanatory diagram of the parallel count-up process in the low-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. The lower dimension sorting process is a process of rearranging the record numbers stored in the record number array in ascending order of the item value numbers related to the lower dimension items. This process is similar to the process of rearranging record numbers in ascending order of virtual item value numbers in the multi-item batch sort process. In the example of FIG. 15, the number of occurrences of the item value number corresponding to the item value of the item of the lower dimension is counted. This counting process is also shared by the processor. In this example, the record number array 503 is divided into the respective handling ranges 1501, 1502, 1503, and 1504 of the four processors.

A count array for storing the number of item value numbers counted by the processor j, where the record number is i, the record number array is OrdSet, the item value number array is VNo_2, and the processor number is j (j = 0, 1, 2, 3). If Count [j], the counting process according to the present embodiment is, for example, in C language style.
for (i = 0; i <4; i ++) {
for (j = 0; j <4; j ++) {
Count [j] [V_VNo [OrdSet [j * 4 + i]]] ++;
}
}
Is described. In the example of FIG. 15, the initialized count arrays 1510, 1511, 1512, and 1513 become count arrays 1520, 1521, 1522, and 1523 by the item value number counting process. In this example, the size of the count array Count is equal to (VL size) × N, where the item value number array of the target item is VL and the number of processors is N.

Subsequently, in one embodiment of the present invention, the computer system 10 generates a cumulative frequency distribution array of the number of occurrences of item value numbers by accumulating each element of the item value number array. FIG. 16 is an explanatory diagram of a process for generating a cumulative frequency distribution array in the low-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. When the count array within the range of the processor j is Count [j] and the cumulative frequency distribution array is Aggr [j], the cumulative frequency distribution is
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Aggr [3] [0] + Count [3] [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Aggr [3] [1] + Count [3] [1]
Aggr [1] [2] = Aggr [0] [2] + Count [0] [2]
Aggr [2] [2] = Aggr [1] [2] + Count [1] [2]
Aggr [3] [2] = Aggr [2] [2] + Count [2] [2]
Aggr [0] [3] = Aggr [3] [2] + Count [3] [2]
Aggr [1] [3] = Aggr [0] [3] + Count [0] [3]
Aggr [2] [3] = Aggr [1] [3] + Count [1] [3]
Aggr [3] [3] = Aggr [2] [3] + Count [2] [3]
Is calculated as This calculation may be performed by any one of the four processors alone, but the range of virtual item value numbers is assigned to the four processors and performed in parallel by the four processors. Also good.

When the cumulative frequency distribution is calculated in parallel by four processors, first, the total value Sum [i] of the number of occurrences of each virtual item value number i is, for example,
for (i = 0; i <4; i ++) {
for (j = 0; j <4; j ++) {
Sum [i] = Count [j] [i];
}
}
To calculate in parallel. In this embodiment, for example, CPU-0 is in charge of the i = 0 loop, CPU-1 is in charge of the i = 1 loop, CPU-2 is in charge of the i = 2 loop, and CPU-3 is in charge of the loop. Take charge of the loop of i = 3.

Subsequently, for example, four processors are connected in parallel,
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Sum [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Sum [1]
Aggr [1] [2] = Aggr [0] [2] + Count [0] [2]
Aggr [2] [2] = Aggr [1] [2] + Count [1] [2]
Aggr [3] [2] = Aggr [2] [2] + Count [2] [2]
Aggr [0] [3] = Sum [2]
Aggr [1] [3] = Aggr [0] [3] + Count [0] [3]
Aggr [2] [3] = Aggr [1] [3] + Count [1] [3]
Aggr [3] [3] = Aggr [2] [3] + Count [2] [3]
To obtain a cumulative frequency distribution array.

  FIG. 16 shows final cumulative frequency distribution arrays 1600, 1601, 1602, and 1603.

Next, in the low-order sort of the multi-item sequential sort process according to the embodiment of the present invention, the elements of the original record number array are transferred to the sorted record number array. FIG. 17 is an explanatory diagram of processing for rearranging elements of a record number array according to an embodiment of the present invention. In this processing, the record number array 503 is divided into four processor ranges 1501, 1502, 1503, and 1504 as in FIG. 15, and the four processors execute parallel processing. If the original record number array is OrdSet, the sorted record number array is OrdSet ', the item value number array is VNo_2, and the cumulative frequency distribution array is Aggr [j] (j is a processor number), processor 0, that is, CPU Sorting by -0 is
for (i = 0; i <4; i ++) {
OrdSet ′ [Aggr [0] [VNo — 2 [OrdSet [i]]]] = OrdSet [i];
Aggr [0] [VNo_2 [OrdSet [i]]] ++;
}
Can be described. Here, Aggr [0] [VNo_2 [OrdSet [i]]] ++ indicates that the elements of the cumulative frequency distribution array are used as index pointers.

Furthermore, assuming that the processor number j is a loop variable, the process of rearranging the elements of the record number array according to this embodiment is as follows:
for (j = 0; j <4; j ++) {
for (i = 0; i <4; i ++) {
OrdSet ′ [Aggr [j] [VNo — 2 [OrdSet [j * 4 + i]]]] = OrdSet [j * 4 + i];
Aggr [j] [VNo_2 [OrdSet [j * 4 + i]]] ++;
}
}
Can be described.

  When the process of rearranging the elements of the record number array is completed, the cumulative frequency distribution arrays 1700, 1701, 1702, and 1703 represent the sorted cumulative frequency distribution arrays, and the sorted version record number array 1710 is the sorted record number array. Represents. Thus, by the process of rearranging the elements of the record number array according to one embodiment of the present invention, the elements of the original record number array 503 are sorted with respect to the lower dimension and stored in the sorted version of the record number array 1710. . Tabular data 1400 indicates tabular data sorted with respect to the lower dimensions.

  Furthermore, the record numbers can be grouped as shown in the group number array 1720 by using the value of the cumulative frequency distribution array 1703 at the time of completion of sorting. This group number is sorted along with the record number when the record number is later sorted with respect to the upper dimension.

  In the multi-dimensional sequential sorting process according to the embodiment of the present invention, the sorting for the upper dimension is performed following the sorting for the lower dimension. Since the sort related to the upper dimension is similar to the sort related to the lower dimension, it will be briefly described. FIG. 18 is an explanatory diagram of the parallel count-up process in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. Again, the number of appearances of elements in the higher-order item value number array VNo_1 is counted up in parallel by four processors. The count up results are shown in count arrays 1820, 1821, 1822, 1823.

  FIG. 19 is an explanatory diagram of the cumulative frequency distribution array generation process in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. Also in the case of the higher-order sort processing, cumulative frequency distribution arrays 1900, 1901, 1902, and 1903 are generated in parallel using a plurality of processors, similarly to the lower-order sort processing.

  FIG. 20 is an explanatory diagram of the process of rearranging the elements of the record number array in the higher-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. Also in the case of the higher-order sort process, the record number array 1710 is rearranged in parallel to the record number array 2010 using a plurality of processors as in the lower-order sort process. At this time, the group number array 1720 is also rearranged in parallel to the group number array 2020. Tabular data 1410 indicates tabular data sorted with respect to the upper dimension following the lower dimension.

  The record number array 2010 is the same as the record number array 1005 obtained by the multi-item batch sort process described with reference to FIG. 10, and the same result is obtained in both the multi-item batch sort process and the multi-item sequential sort process. It was confirmed that

  As shown by the elements of the final cumulative frequency distribution array 2003, the group number array 2020 includes a first section from Group2 [0] to Group2 [4], and from Group2 [5] to Group2 [10]. The second section and the third section from Group 2 [11] to Group 2 [15] are grouped into three sections. Further, the first section is divided into three parts 0, 2, 3 according to the element value, the second section is divided into four parts 0, 1, 2, 3, and the third section is 1, 2, 3 is divided into three. Therefore, the complete group number array 2030 can be obtained by sequentially comparing the group number array 2020 from the top, for example, by counting up using a for loop. It should be noted that the cache prefetch is effective for such a sequential access.

  As described above, by applying the above-described unique row extraction process to the record number array 2010 obtained by the multi-item sequential sort process, a unique eigenvalue pair, that is, a total dimension can be created.

[Measure parallel calculation processing]
Finally, a multi-dimensional tabulation process is realized by calculating a measure based on the tabulation dimension obtained by the unique row extraction process. Hereinafter, calculation of this measure will be described. The result of the tabulation process is expressed by a tabulation result table with group numbers as rows and dimensions and measures as columns. Here, two types of measure calculation methods are proposed depending on the size of the tabulation result table, that is, the relationship between the number of rows and the size of the cache memory. Considering the final record number array and group number array obtained by the unique row extraction process, elements are arranged in ascending order in the group number array. On the other hand, elements are randomly arranged in the record number array. Therefore, when the calculation of the measure is performed, if the number of rows in the aggregation result table is small, the elements in the record number array are rearranged sequentially (record number ascending order type) in order to increase the cache hit rate as much as possible. As a result, the access to the record number array becomes sequential access, and the cache prefetch mechanism increases the cache hit rate. On the other hand, the group number array has a high cache hit rate because fluctuations in the values of elements in the array are relatively small. Conversely, when the number of rows in the tabulation result table is relatively large, the record number array is not rearranged (group number ascending order type) to simplify the processing procedure and reduce the amount of work memory used. .

[Measure number parallel calculation process of record number ascending order type]
Hereinafter, the measure parallel calculation processing of the record number ascending order type will be described. This process has the advantage of a high cache hit rate. The elements of the record number array are unique, but the cache hit rate is increased by the prefetch mechanism because sequential access is performed by converting the elements into the ascending order of the record numbers. In addition, since the group number array has a small fluctuation, a high cache hit rate can be maintained. In this process, since each processor may access the entire group number array, a totaling work area is prepared for each processor, and the totaling result of each processor is collected to create a final counting result. Note that when there are a large number of measures, the cost of rearranging the record number array per measure is reduced, and access from the item value number array to the item value array is random access. Should.

  Ascending record number order type measure parallel calculation processing uses a plurality of processors to rearrange the elements of the record number array and the elements of the group number array so that the elements of the record number array are in ascending order, Using the above-mentioned processors, the measures are aggregated in parallel, and the measures calculated by the processors are synthesized.

  FIG. 21 is an explanatory diagram of the rearrangement process in the measure number ascending order type measure parallel calculation process according to an embodiment of the present invention.

If the record number array before sorting is OrdSet ″, the group number array is Group3, the sorted record number array is OrdSet ″, the group number array is Group4, and the loop variable is i, this sorting process is ,
for (i = 0; i <16; i ++) {
OrdSet ′ ″ [OrdSet ″ [i]] = OrdSet ″ [i];
Group4 [OrdSet '' [i]] = Group4 [i];
}
Can be described. This processing is executed in parallel by a plurality of processors by dividing the record number array 2010 into the handling ranges 2100, 2101, 1022, 2103 for each processor as shown in FIG.

  FIG. 22 is an explanatory diagram of the aggregation process in the measure parallel calculation process of the record number ascending order type according to one embodiment of the present invention. By dividing the record number array 2110 into four, such as the handling ranges 2200, 2201, 2202, 2203 for each processor, the measure parallel calculation processing is executed in parallel by the four processors. FIG. 22 shows an item value number array VNo_3 and an item value array VL_3 of the item “point” for which the measure is to be calculated. In this example, the measure is the total value of “points”. Therefore, the total value of “points” is calculated with respect to the already calculated dimension 1130.

Since four processors are used, four intermediate result storage arrays Sum_0, Sum_1, Sum_2, Sum_3 and a final result storage array Sum are secured on the shared memory, and the element value is initialized to 0. The For example, the process of CPU-0 uses i as a loop variable.
for (i = 0; i <4; i ++) {
Sum_0 [Group4 [OrdSet ′ ″ [i]] = + VL_3 [VNo — 3 [OrdSet ′ ″ [i]]];
}
Is described as The same applies to CPU-1, CPU-2, and CPU-3.

Next, the final result storage array Sum is divided into four, and each processor has a responsibility range.
Sum [i] = Sum_0 [i] + Sum_1 [i] + Sum_2 [i] + Sum_3 [i]
As a result, the final result storage array 2240 as shown in the figure is obtained. The elements of this final result storage array are the calculated measures, and the corresponding dimension is shown in FIG. 22 as dimension 1130.

[Group number ascending type measure parallel calculation processing]
Next, group number ascending type measure parallel calculation processing will be described. This process does not require an intermediate result storage array such as Sum_0, Sum_1, Sum_2, and Sum_3, so that the amount of memory used does not increase even if the number of rows in the aggregation result table is large. There are advantages. Further, since this process does not require rearrangement of the record number array, there is an advantage that the processing time is shortened. On the other hand, it should be noted that access to the item value number array using the elements of the record number array as subscripts is random access. Access from the item value number array to the item value array is also random access.

  FIG. 23 is an explanatory diagram of a counting process in the group number ascending type measure parallel calculation process according to an embodiment of the present invention. The record number array 2010 is divided into the handling ranges 2300, 2301, 2302, 2303 for each of the four processors. Here, it should be noted that the handling ranges 2300, 2301, and 2302, so that the record numbers belonging to the same group are not assigned to different processors and the handling ranges of the respective processors are made as uniform as possible. 2303 must be determined. For example, the coverage range is, for example, a processor in which any one processor sets the limit of the coverage range per unit, scans the group number array 2030 in order from the top, and has the same record number belonging to the same group. Determined by selecting a boundary within the bounds of the coverage. When the responsible range of each processor is determined, the measure parallel calculation processing is executed in parallel by the four processors. FIG. 23 shows an item value number array VNo_3 and an item value array VL_3 of the item “point” for which the measure is to be calculated. In this example, the measure is the total value of “points”. Therefore, the total value of “points” is calculated with respect to the already calculated dimension 1130.

If the record number array is OrdSet ″, the group number array is Group3, the result storage array is Sum (initial value is 0), and the loop variable is i, the entire measure calculation process is as follows:
for (i = 0; i <16; i ++) {
Sum [Group3 [OrdSet ″ [i]]] = + VL — 3 [VNo — 3 [OrdSet ″ [i]]];
}
Is described as In practice, the instructions in this loop are expanded into the processing of each processor and executed in parallel.

  The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention described in the claims, and these are also included in the scope of the present invention. Needless to say.

[Multi-stage parallel sort]
Here, an embodiment of multi-stage parallel sorting realized by combining parallel sorting based on counting sort and the concept of radix sort as described in Patent Document 3 will be described. This multi-stage parallel sort is a description that is the basis of the above-described multi-item batch sort processing and multi-item sequential sort processing.

  When the size of the item value array VL is large, that is, when there are a large number of item value numbers, the item value numbers are expressed in radixes, and parallel sorting is performed for each digit to achieve efficient sorting. It is possible. In the multi-stage parallel sort of this example, the sorting process for the current digit is sequentially performed starting from the least significant digit, and the final sorting is completed by finally performing the sorting process for the most significant digit.

  In an example of the multistage parallel sorting method, the data structure of FIG. 24 is used. In the present embodiment, the number of CPUs is four, and an example in which all CPUs operate in parallel is considered. It should be noted that the total number of CPUs in the system and the number of CPUs operating in parallel are not limited to this example. In the following, for convenience of explanation, consider a case where the items of age are sorted in ascending order of age. The elements of the item value array for age are arranged in ascending order of age. In the data structure of FIG. 4, the item value number VNo relating to age can take a value from 0 to 4. Therefore, when the item value number is decomposed with the radix = 4, the item value number has two digits, the lower digit and the upper digit. Is broken down into Specifically, the modulo (4) value of the item value number is the lower digit value, and the quotient obtained by dividing the item value number by 4 is the upper digit value.

  FIG. 25 is a flowchart of the multi-stage parallel sorting method according to the present embodiment. The multi-stage parallel sorting method includes five steps from step 2501 to step 2505.

  Step 2501: Select the radix of the item value number (in this example, radix = 4) according to the range of the item value number, set the initial record number array OrdSet to the current record number array, and set the lowest item value number (In this example, the modulo (4) value of the item value number) is set to the current digit.

  Step 2502: The current record number array is divided and assigned to four processors.

  Step 2503: In each of the four processors, the number of occurrences of the current digit value of the item value number corresponding to the record included in the allocated record number array portion is counted.

  Step 2504: The range of the current digit value of the item value number is divided and assigned to four processors.

  Step 2505: In each of the four processors, the order of the current digit value of the item value number is in accordance with the order of the part of the record number array within the range where the current digit value of the item value number matches. The number of occurrences of the current digit value of the assigned item value number is converted into a cumulative number.

  Step 2506: In each of the four processors, the cumulative number of occurrences of the current digit value of the item value number corresponding to the record included in the assigned record number array portion is used as a pointer. The record number included in the allocated record number array portion is stored in a further record number array.

  Step 2507: It is determined whether or not the sorting process has been performed up to the most significant digit of the item value number expressed in the radix. If the sorting process has been performed up to the most significant digit, the multistage parallel sorting process is terminated.

  Step 2508: If an unprocessed digit remains, the digit is set to the current digit, the further record number array is set as the current record number array, and the process returns to Step 2502.

  In the multi-stage parallel sorting method according to the above-described embodiment, the sorting processing from step 2502 to step 2506 is the same processing as the parallel sorting method of the present invention described above, and the current item value number is used instead of the item value number. The only difference is that the value of the digit is used.

  Next, the multi-stage parallel sorting method according to the present embodiment will be specifically described. In this example, the data shown in FIG. 24 is sorted in ascending order of age using four CPUs. The initialization step 2501 sets a sort process related to the value of the modulo 4 (MOD 4) of the age item value number (the value of the lower digit) as the first stage sort process, and as the second stage sort process, A sorting process is set for the value of the quotient (DIV 4) divided by 4 of the item value number of age.

  In an initialization step 2501, an array for counting the number of occurrences of the value of the current digit of the item value number is prepared.

  FIGS. 26A to 28B are explanatory diagrams of the counting step 2503 of the first stage of the multi-stage parallel sorting method. In sub-step 1 of FIG. 26A, for example, CPU-0 reads the value 0 of OrdSet [0], reads the value 1 of VNo [0] using the read value 0 as a subscript, and modulo this value 1 4 (MOD4) value 1 is subscripted and Count-0 [1] value 0 is incremented to 1. Similarly, the CPU-1 reads the value 5 of OrdSet [5], reads the value 2 of VNo [5] using this value 5 as a subscript, and counts the value 2 of MOD4 of this value 2 as a subscript. The value 0 of [2] is incremented to 1. Subsequently, by executing sub-step 2 in FIG. 26, sub-step 3 in FIG. 27A, sub-step 4 in FIG. 27B and sub-step 5 in FIG. 28A, a count-up result as shown in FIG. 28B is obtained. The element Count-0 [i] of the array Count-0 in FIG. 28B is an item value number of the age corresponding to each record in the range of OrdSet [0] to OrdSet [4] of the array OrdSet that the CPU-0 was in charge of. It represents the number of appearances of the value i of the lower digit. For example, Count-0 [0] indicates that the number of occurrences of the value 0 of the lower digit of the item value number within the CPU-0's assigned range is 2, and Count-3 [1] is CPU-3. This indicates that the number of occurrences of the value 1 of the lower digit of the item value number in the assigned range is 2 times.

  FIGS. 29A and 29B are explanatory diagrams of the cumulative number step in the first stage of the multi-stage parallel sorting method. In this example, in accordance with the ascending order sort, the cumulative number is performed in ascending order of the value of the lower digit of the item value number. CPU-0 is in charge of accumulating the first row of the array count (that is, the value 0 of the lower digit of the item value number), and CPU-1 to CPU-3 are respectively 2 to 4 of the array count. Responsible for accumulating the number of rows (that is, values 1 to 3 in the lower digits of the item value number). As shown in FIG. 29A, the cumulative numbering is performed by giving priority to the horizontal direction of the array count (that is, the row with the same subscript), and then the cumulative number of the preceding row is changed to the cumulative number of the subsequent row. By adding, the total number of totals is determined. The cumulative number in the horizontal direction can be executed in parallel by the CPUs as described above, but a single CPU may be in charge.

  30A to 32B are explanatory diagrams of a transfer step of storing record numbers in a further record number array in the first stage of the multistage parallel sort method. In the transfer step, each CPU reads the record number within the range that it is in charge of from the record number array OrdSet, and then reads the value of the lower digit of the item value number from the pointer array VNo using the record number as a subscript. Further, using the value of the lower digit of this item value number as a subscript, the cumulative numerical value is read from the accumulated count array associated with the processor, and the further cumulative record number is pointed to the read cumulative numerical value. The record number is stored in the array OrdSet ′, and the cumulative value of the Count array is incremented by one. FIG. 32B shows the record number array OrdSet 'obtained in the first stage as a result of such a transfer step.

  In the second stage, with the record number array OrdSet 'obtained in the first stage as an initial condition, ascending order sorting is performed on the value of the upper digit (value of DIV 4) of the item value number of age.

  FIG. 33 is a diagram showing a state in which the current record number array OrdSet ′ is assigned to four CPUs and the respective Count arrays are prepared in step 2502 of the second stage of the multi-stage parallel sorting method according to the present embodiment. .

  FIG. 34A thru | or 36B is explanatory drawing of the count step of the 2nd step of a multistep parallel sort method. In substep 1 of FIG. 34A, for example, CPU-0 reads the value 2 of OrdSet '[0], reads the value 4 of VNo [2] using the read value 2 as a subscript, Using the value 1 of the quotient (DIV4) divided by 4 as a subscript, the value 0 of Count-0 [1] is incremented to 1. Similarly, CPU-1 reads the value 12 of OrdSet '[5], reads this value 12 as a subscript, reads the value 4 of VNo [12], uses the value 1 of DIV4 of this value 4 as a subscript, and count- The value 0 of 1 [1] is incremented to 1. Subsequently, by executing sub-step 2 in FIG. 34B, sub-step 3 in FIG. 35A, sub-step 4 in FIG. 35B and sub-step 5 in FIG. 36A, the count-up result of the second stage as shown in FIG. 36B is obtained. can get. The element Count-0 [i] of the array Count-0 in FIG. 36B is the item value of the age corresponding to each record in the range of OrdSet ′ [0] to OrdSet [4] of the array OrdSet ′ that the CPU-0 was responsible for. It represents the number of occurrences of the value i of the upper digit of the number. For example, Count-0 [0] indicates that the number of occurrences of the value 0 of the upper digit of the item value number within the CPU-0's assigned range is four, and Count-3 [1] is CPU-3. This means that the number of occurrences of the value 1 of the upper digit of the item value number in the assigned range is 0 times.

  FIG. 37 is an explanatory diagram of the cumulative number step in the second stage of the multi-stage parallel sorting method. In this example, corresponding to the ascending sort, the cumulative number is performed in ascending order of the value of the upper digit of the item value number. Since the number of the upper digits of the item value number is reduced to two by multi-stepping, in this example, for example, CPU-0 is in charge of accumulating all values. As shown in FIG. 37, the CPU-0 counts Count-0 [0], Count-1 [0], Count-2 [0], Count-3 [0], Count-0 [1], Count- Accumulation is performed in the order of 1 [1], Count-2 [1], and Count-3 [1]. Of course, in the case of this example, the CPU 0 and CPU-1 may be assigned the upper digits 0 and 1 of the item value number to the two CPUs, and the two CPUs may perform the cumulative number calculation. .

  FIGS. 38A to 40B are explanatory diagrams of a transfer step of storing record numbers in a further record number array in the second stage of the multistage parallel sort method. In the transfer step, each CPU reads the record number within the range that it is in charge of from the record number array OrdSet, and then reads the value of the upper digit of the item value number from the pointer array VNo using the record number as a subscript. Further, using the value of the upper digit of this item value number as a subscript, the cumulative numerical value is read from the accumulated count array associated with the processor, and the further cumulative record number is pointed to the read cumulative numerical value. The record number is stored in the array OrdSet ″, and the cumulative value of the Count array is incremented by 1. FIG. 40B shows the record number array OrdSet ″ obtained in the second stage as a result of such a transfer step.

  Since the multi-stage parallel sorting method of the present embodiment is composed of two stages of the lower digit and the upper digit of the item value number, no further sort processing is performed. Therefore, the record number array OrdSet "obtained in the second stage is the result of sorting the first record number array OrdSet in ascending order with respect to age.

  41A and 42B are diagrams showing the results of applying the ascending multi-stage parallel sorting method according to the embodiment of the present invention to the data structure shown in FIG. In this example, since the ascending order regarding age is performed, records having the age item values of 16, 18, 20, 21, and 23 are arranged in order of age in the resulting record number array OrdSet ”. In addition, the order of records having the same age is stored in the original record number array OrdSet.

  Although the above-described multi-stage parallel sorting method is ascending order sorting, the multi-stage parallel sorting according to the present embodiment operates similarly in descending order sorting. Further, the cumulative number calculation in each stage of the multi-stage parallel sort may be processed in parallel by a plurality of processors, or may be processed by at least one, preferably one processor alone.

[Multi-level sorting]
In the above multi-stage parallel sort, the final sort is completed by performing the sort process for the current digit in order starting from the least significant digit and finally the sort process for the most significant digit. On the other hand, it is also possible to complete the final sorting by performing the sorting process for the current digit in order starting from the most significant digit and finally the sorting process for the least significant digit. In the following, a method of performing multi-stage sort processing in order from the highest level to the lowest level will be briefly described. This multi-stage technique is similarly applied to the above-described multi-item sequential sorting process.

  In this example, a data structure as shown in FIG. 43 is used. In this example, the number of CPUs is one. In the following description, it is assumed that the items of age are sorted in ascending order of age. The total number of records is 20 from record number 0 to record number 19, and the item value number is 9 from 0 to 8. That is, there are nine actual age values of 15, 16, 18, 19, 20, 21, 23, 25, and 28. In the data structure of FIG. 43, the item value number VNo regarding age can take a value from 0 to 8, so when the item value number is decomposed with the radix = 4, the quotient obtained by dividing the item value number by 4 is the upper digit. It is a value, and the value of the modulo (4) of the item value number is the value of the lower digit. The upper digit of the item value number can take three values, 0, 1, and 2, and the lower digit can take four values, 0, 1, 2, and 3.

  First, in the first stage, an array Count-1 for counting the number of occurrences of the upper digits 0, 1, and 2 is prepared, and the elements are initialized with 0. For example, Count-1 [0] is an area for counting the number of records in which the value of the upper digit of the item value number is 0.

  Next, in order from the first element (that is, record) of the record number array OrdSet, the item value number corresponding to the element is read from the array VNo, and the quotient value obtained by dividing the item value number by 4 is used as a pointer. Then, the value of the element of the array Count-1 is incremented. 44A to 44D calculate the values of the upper digits of the item value numbers for the three record numbers of OrdSet [0] = 0, OrdSet [7] = 7, and OrdSet [19] = 19. It is explanatory drawing of the example which counts up the counter to perform, and makes it a cumulative number next. As can be seen from FIG. 44C, by this first-stage count-up process, the number of records in which the upper digit value of the item value number is 0 is 12, and the number of records in which the upper digit value is 1 is The number of records having 7 and the value of the upper digit is 2 is 1. Further, as shown in FIG. 44D, this count value is accumulated.

  Next, the record number array OrdSet is converted into a further record number array OrdSet 'using the array Aggr-1 in which the number of appearances of the upper digits of the item value number is accumulated. Specifically, if OrdSet [i] = j, VNo [j] is read, and if the quotient (VNo [j] DIV 4) obtained by dividing VNo [j] by 4 is k, Aggr-1 The value of [k] is read, record number j is set in OrdSet [Aggr-1 [k]], and Aggr-1 [k] is incremented. 45A and 45B are explanatory diagrams of record number transfer processing in such a multi-stage sort. FIG. 45A shows transfer of OrdSet [0], and FIG. 45B shows transfer of OrdSet [19]. FIG. 46 shows a record number array OrdSet 'as a result of the first-stage record number transfer and a range in which the upper digit values are distributed. For example, records whose upper digit value is 0 are distributed in the range (Section 0) from OrdSet ′ [0] to OrdSet ′ [11] of the record number array OrdSet ′, and the value of the upper digit is 1. Are distributed in the range (Section 1) from OrdSet ′ [12] to OrdSet ′ [18] of the record number array OrdSet ′, and the record whose upper digit value is 2 is OrdSet ′ [19] of the record number array OrdSet ′. It exists in (Section 2).

  Next, in the second stage of multi-stage sorting, the record numbers are sorted by the value of the lower digit of the item value number within each section. For example, section 1 of OrdSet ′ is transferred to section 1 corresponding to OrdSet ″. In the second sort, since the section is already determined by the upper digit, the record number is transferred outside the section. There is nothing.

  FIG. 47 is a diagram illustrating an initial state of the second stage of the multistage sorting. In the following description, section 1 of OrdSet ′ will be described. For example, when there are a plurality of processors, it is possible to parallelize the following processes by assigning a processor to each section. Count-2 is an array for counting the number of appearances of the value (0, 1, 2, 3) of the lower digit of the item value number in section 1.

  FIGS. 48A to 48C are explanatory diagrams of the second stage count-up and accumulation in the multi-stage sort. By counting up sequentially starting from FIG. 48A, a count-up sequence as shown in FIG. 48B is obtained. This count-up array is accumulated as shown in FIG. 48C.

  Finally, using the second cumulative number array Aggr-2 as a pointer, the section 1 of the record number array OrdSet 'is transferred to the section 1 of the record number array OrdSet ", thereby completing the multi-stage sorting. 49A and 49B are explanatory diagrams of the transfer of the record number in the second stage of the multi-stage sort, specifically, if OrdSet ′ [i] = j, VNo [j] is read and this VNo [j ] Is divided by 4 (VNo [j] MOD 4), k is read, the value of Aggr-2 [k] is read, record number j is set in OrdSet ”[Aggr-2 [k]], and Aggr -2 [k] is incremented. 49A shows the transfer of OrdSet '[14], and FIG. 49B shows the transfer of OrdSet' [18]. The section 1 of “OrdSet” in FIG. 49B represents the final sorting result of the section 1.

  Similar to the section 1, by applying the second-stage count-up, accumulation, and record number transfer to the other sections 0 and 2, the entire record number array OrdSet is changed to the record number array OrdSet ”. Transfer and complete sorting.

In the third embodiment, the three-dimensional tabular table of 1 million rows (1M), 5 million rows (5M), 10 million rows (10M), 50 million rows (50M) and 100 million rows (100M) is sorted and unique. The following five methods, that is, a totaling process for extracting a set of item values and counting the number of occurrences of each set of item values, are as follows:
(1) Multi-stage parallel sort method (old method)
(2) A method combining multi-item sequential sorting and record number ascending order according to the present invention (OrdSet ascending order method)
(3) A method combining multi-item batch sorting and record number ascending order according to the present invention (OrdSet ascending order method (three-dimensional batch sorting))
(4) Method combining multi-item sequential sorting and group number ascending order according to the present invention (dimensional ascending order method)
(5) Method combining multi-item batch sorting and group number ascending order according to the present invention (dimensional ascending order method (three-dimensional batch sorting))
It carried out using. Then, the processing time of each method was measured, and a speed-up magnification representing how fast the method according to the present invention was compared with the old method was calculated. The measurement environment of this example is as follows.
Computer model: Dell PowerEdge ^™ 6800
CPU: Intel (registered trademark) dual-core processor x 2
Number of clocks: 3.16 GHz
Memory size: 32GB
L2 cache: 1MB x 4
OS: SuSE 2.6.5-7-191-smp
50A to 50C are diagrams for explaining the contents of the aggregation process in the third embodiment. FIG. 50A shows an output totaling result table, FIG. 50B shows a totaling source table consisting of 1 million rows, and FIG. 50C shows a totaling result table consisting of 632334 records.

  51A to 51C are diagrams showing measurement results of Example 3. FIG. FIG. 51A shows the required time (milliseconds) of processing by the above five methods, and FIG. 51B shows that the above methods (2) to (5) according to the present invention are faster than the old method (1). FIG. 51C is a graph showing the same high speed magnification as FIG. 51B.

  (5) Dimension ascending order method (3D batch sort), (3) OrdSet ascending order method (3D batch sort), (4) Dimension ascending order method, (2) OrdSet ascending order method. Depending on the situation, it has reached 10 times the old method.

  In Example 4, in the same measurement environment as Example 3, 3 million lines (1M), 5 million lines (5M), 10 million lines (10M), 50 million lines (50M), and 100 million lines (100M) A tabulation process for sorting the dimension table format table, extracting a set of unique item values, and calculating the total of the measures was performed using the above five methods in the same manner as in the second embodiment. Then, the processing time of each method was measured, and a speed-up magnification representing how fast the method according to the present invention was compared with the old method was calculated.

  52A to 52C are diagrams for explaining the contents of the aggregation process in the fourth embodiment. FIG. 52A shows an output totaling result table, FIG. 52B shows a totaling source table consisting of 1 million rows, and FIG. 52C shows a totaling result table consisting of 632334 records.

  53A to 53C are diagrams showing measurement results of Example 4. FIG. FIG. 53A shows the required time (milliseconds) of processing by the above five methods, and FIG. 53B shows that the above methods (2) to (5) according to the present invention are faster than the old method (1). FIG. 53C is a graph showing the same high speed magnification as FIG. 53B.

  Regardless of the size of the tabulation result table, the three-dimensional batch sort method realizes tabulation processing faster than the sequential sort method. Furthermore, when the size of the aggregation result table is relatively small (less than 5 million rows), the dimension ascending method is faster than the OrdSet ascending method, but when the size of the aggregation result table is increased, the OrdSet ascending method is dimensioned. It was found that it was faster than the ascending method.

  From Example 3 and Example 4, the method of the present invention realizes multi-dimensional tabulation processing of large-scale tabular data at higher speed than the prior art, regardless of which method is (2) to (5). It was proved.

[0020]
For example, in C language style,
for (i = 0; i <4; i ++) {
for (j = 0; j <4; j ++) {
Count [j] [V_VNo [OrdSet [j * 4 + i]]] ++;
}
}
Is described. In the example of FIG. 7, the initialized count arrays 710, 711, 712, and 713 become count arrays 720, 721, 722, and 723 by the virtual item value number counting process. In this example, when the first item value array is VL_1 and the second item value array is VL_2, the size of the count array Count is equal to the size of (VL_1 size) × (VL_2).
[0057]
As a result of the counting process, for example, one virtual item value number = 0, 2, 5, and 9 exists in the CPU-0 range, and a virtual item value exists in the CPU-1 range. Number = 6, 7, 10 and 11 respectively, and within the CPU-2 range, there is one virtual item value number 0, 3, 4 and 5, respectively, and CPU-3 In the range, one virtual item value number = 2, 5, 10 and 11 is present.
[0058]
Subsequently, in one embodiment of the present invention, the computer system 10 accumulates each element of the virtual item value number array to generate a cumulative frequency distribution array of the number of appearances of the virtual item value number. FIG. 8 is an explanatory diagram of processing for generating a cumulative frequency distribution array according to an embodiment of the present invention. When the count array within the range of the processor j is Count [j] and the cumulative frequency distribution array is Aggr [j], the cumulative frequency distribution is
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Aggr [3] [0] + Count [3] [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]

[0030]
1523. In this example, the size of the count array Count is equal to (VL size) × N, where the item value number array of the target item is VL and the number of processors is N.
[0086]
Subsequently, in one embodiment of the present invention, the computer system 10 generates a cumulative frequency distribution array of the number of occurrences of item value numbers by accumulating each element of the item value number array. FIG. 16 is an explanatory diagram of a process for generating a cumulative frequency distribution array in the low-order sort process of the multi-item sequential sort process according to an embodiment of the present invention. When the count array within the range of the processor j is Count [j] and the cumulative frequency distribution array is Aggr [j], the cumulative frequency distribution is
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Aggr [3] [0] + Count [3] [0]
Aggr [1] [1] = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Aggr [3] [1] + Count [3] [1]
Aggr [1] [2] = Aggr [0] [2] + Count [0] [2]
Aggr [2] [2] = Aggr [1] [2] + Count [1] [2]
Aggr [3] [2] = Aggr [2] [2] + Count [2] [2]
Aggr [0] [3] = Aggr [3] [2] + Count [3] [2]
Aggr [1] [3] = Aggr [0] [3] + Count [0] [3]
Aggr [2] [3] = Aggr [1] [3] + Count [1] [3]
Aggr [3] [3] = Aggr [2] [3] + Count [2] [3]
Is calculated as This calculation may be performed by any one of the four processors alone, or may be performed in parallel by four processors by assigning a range of item value numbers to the four processors. Good.

[0031]
[0087]
When the cumulative frequency distribution is calculated in parallel by four processors, first, the total value Sum [i] of the number of occurrences of each item value number i is, for example,
for (i = 0; i <4: i ++) {
for (j = 0; j <4; j ++) {
Sum [i] = Count [j] [i];
}
}
To calculate in parallel. In this embodiment, for example, CPU-0 is in charge of the i = 0 loop, CPU-1 is in charge of the i = 1 loop, CPU-2 is in charge of the i = 2 loop, and CPU-3 is in charge of the loop. Take charge of the loop of i = 3.
[0088]
Subsequently, for example, four processors are connected in parallel,
Aggr [0] [0] = 0
Aggr [1] [0] = Aggr [0] [0] + Count [0] [0]
Aggr [2] [0] = Aggr [1] [0] + Count [1] [0]
Aggr [3] [0] = Aggr [2] [0] + Count [2] [0]
Aggr [0] [1] = Sum [0]
Aggr [1] “1” = Aggr [0] [1] + Count [0] [1]
Aggr [2] [1] = Aggr [1] [1] + Count [1] [1]
Aggr [3] [1] = Aggr [2] [1] + Count [2] [1]
Aggr [0] [2] = Sum [1]
Aggr [1] [2] = Aggr [0] [2] + Count [0] [2]
Aggr [2] [2] = Aggr [1] [2] + Count [1] [2]
Aggr [3] [2] = Aggr [2] [2] + Count [2] [2]
Aggr [0] [3] = Sum [2]
Aggr [1] [3] = Aggr [0] [3] + Count [0] [3]
Aggr [2] [3] = Aggr [1] [3] + Count [1] [3]
Aggr [3] [3] = Aggr [2] [3] + Count [2] [3]
To obtain a cumulative frequency distribution array.

[0038]
Here, an embodiment of multi-stage parallel sorting realized by combining parallel sorting based on counting sort and the concept of radix sort as described in Patent Document 3 will be described. This multi-stage parallel sort is a technology that is the basis of the above-described multi-item batch sort process and multi-item sequential sort process.
[0113]
When the size of the item value array VL is large, that is, when there are a large number of item value numbers, the item value numbers are expressed in radixes, and parallel sorting is performed for each digit to achieve efficient sorting. It is possible. In the multi-stage parallel sort of this example, the sorting process for the current digit is sequentially performed starting from the least significant digit, and the final sorting is completed by finally performing the sorting process for the most significant digit.
[0114]
In an example of the multistage parallel sorting method, the data structure of FIG. 24 is used. In the present embodiment, the number of CPUs is four, and an example in which all CPUs operate in parallel is considered. It should be noted that the total number of CPUs in the system and the number of CPUs operating in parallel are not limited to this example. Further, in the following, for convenience of explanation, consider a case where the items of age are sorted in ascending order of age. The elements of the item value array for age are arranged in ascending order of age. In the data structure of FIG. 4, the item value number VNo relating to age can take a value from 0 to 4. Therefore, when the item value number is decomposed with the radix = 4, the item value number has two digits, the lower digit and the upper digit. Is broken down into Specifically, the modulo (4) value of the item value number is the lower digit value, and the quotient obtained by dividing the item value number by 4 is the upper digit value.
[0115]
FIG. 25 is a flowchart of the multi-stage parallel sorting method according to the present embodiment. The multi-stage parallel sorting method includes five steps from step 2501 to step 2505.
[0116]
Step 2501: Select the radix of the item value number (in this example, radix = 4) according to the range of the item value number, set the initial record number array OrdSet to the current record number array, and set the lowest item value number (In this example, the modulo (4) value of the item value number) is set to the current digit.
[0117]
Step 2502: The current record number array is divided and assigned to four processors.
[0118]
Step 2503: In each of the four processors, the number of occurrences of the current digit value of the item value number corresponding to the record included in the allocated record number array portion is counted.

Claims (20)

Record number array in which record numbers of records in tabular data are stored according to a predetermined record order A shared memory that stores a number array and an item value array in which the item values of the tabular data are stored according to the order of the item value numbers corresponding to the item values;
A plurality of processors capable of accessing the shared memory;
In a shared memory multiprocessor system with
A method for extracting a unique set of item values for a given set of items, comprising:
Cumulative frequency distribution that uses the plurality of processors to count in parallel the number of occurrences of a set of item value numbers corresponding to a set of item values relating to the set of predetermined items, and expresses the cumulative frequency distribution of the number of appearances Generating an array in the shared memory;
Using the plurality of processors, reading the item value number pairs in parallel using the element value of the record number array as a subscript, and using the value calculated from the item value number set as a subscript Reading the elements of the cumulative frequency distribution array in parallel, and storing the values of the elements of the record number array in parallel in the sorted record number array using the elements of the cumulative frequency distribution array as an index pointer; ,
Dividing the sorted record number array according to the value of the element of the cumulative frequency distribution array used as the index pointer into a group of record numbers of records having the unique field value pairs;
Identifying the unique set of field values by extracting one record number from each group of the segmented record numbers;
A method comprising: The step of generating the cumulative frequency distribution array in parallel in the shared memory is performed by using the plurality of processors, and for each record number stored in the record number array, an item value related to the predetermined item set By setting the radix according to the range of the item value numbers included in the set of item value numbers corresponding to the set of item values, the set of item value numbers is changed to each item value number included in the set of item value numbers. Converting to a virtual field value number in a radix format representing each digit,
The method of claim 1. (I) generating the cumulative frequency distribution array in parallel in the shared memory for each item included in the set of predetermined items, and the record number array for each item included in the set of predetermined items. A step (ii) of storing element values in parallel in the sorted record number array, and the sorted record number array for each item included in the predetermined set of items, Performing the step (iii) of classifying records into groups of record numbers of records having
The step (i) of generating the cumulative frequency distribution array in the shared memory in parallel uses the plurality of processors to count in parallel the number of occurrences of the item value number corresponding to the item value related to the item, Generating a cumulative frequency distribution array representing the cumulative frequency distribution of the occurrence counts in parallel in the shared memory,
The step (ii) of storing the elements of the record number array in parallel in the sorted record number array is performed using the plurality of processors and the values of the element of the item value number array as subscripts. Read the item value number in parallel, read the element of the cumulative frequency distribution array in parallel using the item value number as a subscript, and use the element of the cumulative frequency distribution array as an index pointer in the record number array Storing element values in parallel in the sorted record number array;
The method of claim 1.   The method according to claim 1, further comprising the step of aggregating item values related to different items for each set of the predetermined item values in parallel using the plurality of processors. Using the plurality of processors, the record number in the sorted record number array and the group number of the group corresponding to the record number are used as an auxiliary record number array and auxiliary number using the record number as a subscript. Storing in parallel group number arrays;
Using the plurality of processors, the elements of the auxiliary record number array are used as subscripts to read out elements in parallel from the item value number array relating to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array for the other item, to read the element of the auxiliary group number array using the element of the auxiliary record number array as a subscript, and to Storing the aggregation result of the read item values in the aggregation array using the elements of the number array as a subscript of the aggregation array;
The method of claim 1, further comprising:   Using the plurality of processors, the elements of the sorted record number array are used as subscripts to read out elements in parallel from the item value number array relating to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array for the other item, and read the item value using the group number of the group corresponding to the element of the sorted record number array as the subscript of the aggregate array The method according to claim 1, further comprising the step of storing the totaling result in the totaling array. Record number array in which record numbers of records in tabular data are stored according to a predetermined record order A shared memory that stores a number array and an item value array in which the item values of the tabular data are stored according to the order of the item value numbers corresponding to the item values;
A plurality of processors capable of accessing the shared memory;
In a shared memory multiprocessor system with
A device that extracts a unique set of item values for a given set of items,
Cumulative frequency distribution that uses the plurality of processors to count in parallel the number of occurrences of the set of item value numbers corresponding to the set of item values related to the set of predetermined items, and express the cumulative frequency distribution of the number of appearances Means for generating an array in the shared memory;
Using the plurality of processors, the item value number pairs are read in parallel using the element value of the record number array as a subscript, and the value calculated from the item value number set is used as a subscript. Means for reading the elements of the cumulative frequency distribution array in parallel, and storing the values of the elements of the record number array in parallel in the sorted record number array using the elements of the cumulative frequency distribution array as an index pointer; ,
Means for dividing the record number array sorted according to the element value of the cumulative frequency distribution array used as the index pointer into a group of record numbers of records having the unique item value pairs;
Means for identifying the set of unique item values by extracting one record number from each group of the record numbers that are classified;
A device comprising: The means for generating the cumulative frequency distribution array in parallel in the shared memory, using the plurality of processors, for each record number stored in the record number array, an item value relating to the predetermined set of items By setting the radix according to the range of the item value numbers included in the set of item value numbers corresponding to the set of item values, each item value number included in the set of item value numbers Comprises means for converting to a virtual item value number in a radix format representing each digit,
The apparatus of claim 7. The means for generating the cumulative frequency distribution array in parallel in the shared memory uses the plurality of processors for each item included in the predetermined item set, and the item value corresponding to the item value related to the item Means for counting up the number of appearances of the number in parallel, and generating a cumulative frequency distribution array expressing the cumulative frequency distribution of the number of appearances in the shared memory in parallel;
The means for storing the elements of the record number array in parallel in the sorted record number array uses the plurality of processors for each item included in the set of the predetermined items, and the item value number array The item value number is read in parallel using the value of the element in the subscript, the element of the cumulative frequency distribution array is read in parallel using the item value number as the subscript, and the element of the cumulative frequency distribution array is indexed Means for storing the values of the elements of the record number array in parallel in the sorted record number array by using as pointers;
The apparatus of claim 7.   The apparatus according to claim 7, further comprising means for collecting, in parallel, item values related to different items for each set of the predetermined item values using the plurality of processors. Using the plurality of processors, the record number in the sorted record number array and the group number of the group corresponding to the record number are used as an auxiliary record number array and auxiliary number using the record number as a subscript. Means for storing in parallel group number arrays;
Using the plurality of processors, the elements of the auxiliary record number array are used as subscripts to read out elements in parallel from the item value number array relating to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array for the other item, to read the element of the auxiliary group number array using the element of the auxiliary record number array as a subscript, and to Means for storing the aggregation result of the read item value in the aggregation array using an element of the number array as a subscript of the aggregation array;
The apparatus of claim 7, further comprising:   Using the plurality of processors, the elements of the sorted record number array are used as subscripts to read out elements in parallel from the item value number array relating to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array for the other item, and use the group number of the group corresponding to the element of the sorted record number array as the subscript of the aggregate array to read the item value The apparatus according to claim 7, further comprising means for storing a totaling result of the data in the totaling array. Record number array in which record numbers of records in tabular data are stored according to a predetermined record order A shared memory that stores a number array and an item value array in which the item values of the tabular data are stored according to the order of the item value numbers corresponding to the item values;
A plurality of processors capable of accessing the shared memory;
A computer-readable program that causes a computer to execute code for extracting a unique set of item values for a predetermined set of items, loaded on a computer comprising:
Cumulative frequency distribution that uses the plurality of processors to count in parallel the number of occurrences of the set of item value numbers corresponding to the set of item values related to the set of predetermined items, and express the cumulative frequency distribution of the number of appearances Code for generating an array in the shared memory;
Using the plurality of processors, the item value number pairs are read in parallel using the element value of the record number array as a subscript, and the value calculated from the item value number set is used as a subscript. A code for reading the elements of the cumulative frequency distribution array in parallel, and storing the values of the elements of the record number array in parallel in the sorted record number array using the elements of the cumulative frequency distribution array as an index pointer; ,
A code for classifying the sorted record number array according to the value of the element of the cumulative frequency distribution array used as the index pointer into a group of record numbers of records having the unique set of field values;
A code that identifies the unique set of field values by extracting one record number from each group of the record numbers that are segmented;
Including programs. The code for generating the cumulative frequency distribution array in parallel in the shared memory is an item value related to the predetermined set of items for each record number stored in the record number array using the plurality of processors. By setting the radix according to the range of the item value numbers included in the set of item value numbers corresponding to the set of item values, the set of item value numbers is changed to each item value number included in the set of item value numbers. Comprises a code that converts to a virtual field value number in radix format representing each digit,
The program according to claim 13. Code (i) for generating the cumulative frequency distribution array in parallel in the shared memory for each item included in the predetermined item set, and the record number array for each item included in the predetermined item set A code (ii) for storing element values in parallel in the sorted record number array, and the sorted record number array for each item included in the predetermined set of items A code (iii) for executing a code for dividing the record into groups of record numbers of records having
The code (i) for generating the cumulative frequency distribution array in parallel in the shared memory counts in parallel the number of occurrences of the item value number corresponding to the item value related to the item using the plurality of processors, A code for generating a cumulative frequency distribution array expressing the cumulative frequency distribution of the appearance frequency in the shared memory in parallel;
A code (ii) for storing the elements of the record number array in parallel in the sorted record number array, using the plurality of processors, the value of the element of the item value number array as a subscript Read the item value number in parallel, read the element of the cumulative frequency distribution array in parallel using the item value number as a subscript, and use the element of the cumulative frequency distribution array as an index pointer in the record number array Comprising code for storing element values in parallel in the sorted record number array;
The program according to claim 13.   The program according to claim 13, further comprising a code that aggregates item values related to different items in parallel for each set of the predetermined item values using the plurality of processors. Using the plurality of processors, the record number in the sorted record number array and the group number of the group corresponding to the record number are used as an auxiliary record number array and auxiliary number using the record number as a subscript. Code to store in parallel group number array,
Using the plurality of processors, the elements of the auxiliary record number array are used as subscripts to read the elements in parallel from the item value number array related to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array relating to the other item, to read the element of the auxiliary group number array using the element of the auxiliary record number array as a subscript, and to A code for storing the aggregation result of the read item value in the aggregation array using an element of the number array as a subscript of the aggregation array;
The program according to claim 13, further comprising:   Using the plurality of processors, the elements of the sorted record number array are used as subscripts to read out elements in parallel from the item value number array relating to the other item, and the elements of the item value number array are used as subscripts. To read the item value in parallel from the item value array for the other item, and use the group number of the group corresponding to the element of the sorted record number array as the subscript of the aggregate array to read the item value The program according to claim 13, further comprising a code for storing the totaling result in the totaling array.   A record number array in which the record numbers of the records in the tabular data are stored according to a predetermined record order, and an item value in which the item value numbers corresponding to the field values of the predetermined items in the above records of the tabular data are stored according to the record numbers A shared memory for storing an item value array in which an item value of the table data is stored in accordance with the order of the item value number corresponding to the item value, and a plurality of processors accessible to the shared memory And a computer program product for causing the computer to execute the method of claim 1 for extracting a unique set of item values for a predetermined set of items.   A record number array in which the record numbers of the records in the tabular data are stored according to a predetermined record order, and an item value in which the item value numbers corresponding to the field values of the predetermined items in the above records of the tabular data are stored according to the record numbers A shared memory for storing an item value array in which an item value of the table data is stored in accordance with the order of the item value number corresponding to the item value, and a plurality of processors accessible to the shared memory And a computer-readable storage medium storing a computer program for causing the computer to execute the method according to claim 1, which is loaded into a computer comprising:

Images