JPWO2009028050A1 - Multi-core compatible data processing method, multi-core processing apparatus, and program for manipulating tabular data - Google Patents

Abstract

The method for building a data structure for multi-core processing devices from tabular data is to create an array in which the control unit stores the block number corresponding to the record assigned to each arithmetic unit, and the arithmetic unit stores the record sequence number of the assigned record. An array that stores field value access information for accessing field values included in the assigned record is created in parallel for each data item, and an arithmetic unit creates an array for each data item. In addition, the item values expanded so as to be accessed by the item value access information are created in parallel.

Description

  In the present invention, tabular data represented as an array of records including item values corresponding to data items is shared and operated by a plurality of arithmetic units, and in particular, tabular data is constructed and items are generated from tabular data. It relates to the data processing method for obtaining values.

  In the present invention, tabular data represented as an array of records including item values corresponding to data items is shared and operated by a plurality of arithmetic units, and in particular, tabular data is constructed and items are generated from tabular data. It also relates to a multi-core processing device that acquires values.

  Furthermore, the present invention relates to a program for causing a computer including a multi-core processor to execute the data processing method, a computer program product, and a recording medium on which the computer program is recorded.

  Conventionally, it is required to process large-scale data at high speed in various industrial fields. For large-scale data processing, development of hardware technologies such as high-speed memory access by cache and prefetch, high-speed memory, and high-speed arithmetic processing such as parallel processing of processors, and data processing The development of algorithms continues to speed up.

  The inventor has proposed a basic data processing algorithm for processing large-scale data at high speed, for example, an “on-memory data processing algorithm” as described in Patent Document 1. This technique is based on the idea that the tabular data is decomposed into items (ie, columns) instead of records (ie, rows) as in the prior art. More specifically, the tabular data includes (1) an array representing the record order, (2) a value table in which unique item values belonging to the items are arranged in a predetermined order (for example, ascending order), and (3 ) The item value corresponding to each record is represented by a data structure including an array representing position information stored in the value table. By adopting such a data structure, processes such as retrieval, sorting, merging, and joining of tabular data are realized at high speed.

  Furthermore, the present inventor has proposed various on-memory data processing algorithms corresponding to the parallelization of processors such as a memory distributed multiprocessor system and a memory shared multiprocessor system. For example, Patent Literature 2 describes a search / sort algorithm corresponding to a memory distributed multiprocessor system, and Patent Literature 3 describes a tabulation algorithm. Furthermore, Patent Document 4 describes an efficient sorting algorithm corresponding to a memory-shared multiprocessor system.

By the way, in recent years, a processor architecture including a plurality of (or many) cores in one processor has been proposed. Cell Broadcast Engine ^TM is known as an example of a multi-core type processor (see Non-Patent Document 1). This type of processor is intended to be applied to, for example, high-speed processing of multimedia data and distributed computing. In this architecture, each core is not large-capacity, but has a dedicated local memory, and can perform calculations independently of other cores. Actually, since the memory capacity of the local memory is insufficient for processing multimedia data or the like, an external global memory is provided. The multi-core processor architecture has excellent extensibility because it increases parallelism and processing capacity by adding cores, rather than relying on clock speedup.

Therefore, it is desired that such a multi-core processor architecture is applied not only to multimedia data processing but also to various applications that require high speed.
"Cell Broadband EngineArchitecture", Version 1.01, October 3, 2006, [Search on July 26, 2007], Internet (URL: http://cell.scei.co.jp/pdf/CBE_Architecture_v101.pdf) International Publication No. 00/10103 International Publication No. 2005/041066 International Publication No. 2005/041067 International Publication No. 2006/126467

  The present inventor has recognized the importance of the technology using the multi-core processor architecture having excellent extensibility in order to process large-scale tabular data at high speed.

  However, in an application using a large-scale memory such as a database, since all of data to be processed cannot be accommodated in a local memory attached to the core, the complexity of the data processing algorithm increases. For example, if data that is too large to be accommodated in the local memory associated with each core is randomly accessed, access to an external global memory frequently occurs and processing performance is significantly reduced. Therefore, a new data structure that does not cause such a problem is required.

  Accordingly, in a computer including a multi-core processor, tabular data represented as an array of records including item values corresponding to data items is processed with a small working memory without reducing parallel processing performance. Preferably, a data processing method can be provided.

  In addition, it is possible to provide a multi-core information processing apparatus that processes tabular data represented as an array of records including item values corresponding to data items, using a work memory with a small capacity without reducing parallel processing performance. Is preferred.

  Furthermore, in a computer having a multi-core processor, tabular data represented as an array of records including item values corresponding to data items is processed with a small working memory without reducing parallel processing performance. It is preferable that a program, a computer program product, and a recording medium on which the computer program is recorded can be provided.

  According to at least one embodiment of the present invention, tabular data is handled in two types of data formats in order to handle tabular data using a small capacity local memory without reducing the parallel processing performance of the multi-core processor. Describe by. The first type of data is a group of arrays that are divided into small pieces so that they can be accommodated in the local memory and are held in the global memory (or disk). Since the first type of data can be transferred from the global memory to the local memory at once, no delay occurs even if random access is performed. The second type of data is guaranteed to be accessed continuously in a predetermined order (eg, ascending or descending order) when accessing a large amount of data, and is held in global memory (or disk). Sequence group. Since the second type of data cannot be accommodated in the local memory as it is, it is transferred by sequential access from the global memory to the local memory in the size that can be accommodated in the local memory. Of course, the first type of data may be directly accessed by an element partially stored in the global memory without being sequentially accessed.

  In this document, tabular data means data represented as an array of records including item values corresponding to data items.

  In addition, in this document, the multi-core processing device or processor means a plurality of arithmetic units including a dedicated local memory, a global memory connected to the plurality of arithmetic units, and the plurality of arithmetic units. Means at least one control unit connected to the global memory and the plurality of arithmetic units.

  At least one embodiment of the present invention combines the above-described concept of component decomposition and the above-described two types of data format concepts to construct tabular data on a multi-core processing apparatus.

  Thus, according to at least one embodiment of the invention, the tabular data record (s) is divided into blocks identified by block numbers. Initially, this block corresponds to an arithmetic unit in charge of processing the records included in this block. The record that each arithmetic unit is responsible for is referred to as the charge record in this document. Then, a block number array in which the block numbers are stored in the order of the original record position numbers is created on the global memory. The block number array has a second type of data. The original record position number corresponds to a position where each record is accommodated in the original tabular data, for example, a line number.

  Each arithmetic unit must access the record sequence number array in which the record sequence number of the record in charge (initially matches the original record position number) is stored in order of the record sequence number in order to recognize the record in charge. Can do. This record sequence number array has the first type of data, and is transferred from the global memory to the local memory in each arithmetic unit as necessary.

  Furthermore, since each arithmetic unit accesses the item value included in the assigned record, it can access the item value access information array in which the item value access information is stored in the order of the record sequence numbers. This item value access information array has the first type of data, and is transferred from the global memory to the local memory in each arithmetic unit as necessary.

  The item values included in the record in charge of each arithmetic unit are held in the global memory so that each arithmetic unit can access each data item using the item value access information array. It is transferred to the local memory in each arithmetic unit.

  The item values of the tabular data are constructed on the global memory as a global item value array in which unique item values are stored in a predetermined order (ascending order or descending order) for each data item. This global item value array has a second type of data. Since each arithmetic unit uses the field value access information array to access the field value included in the assigned record, the local field value number that identifies the field value included in the assigned record is the source record position number for each data item. The local item value number array stored in this order and the item value specification pointer for specifying the position where the item value represented by the local item value number is stored in the global item value array are in a predetermined order (ascending or The item value designation pointer array stored in the descending order) is constructed on the global memory, and transferred from the global memory to the local memory in each arithmetic unit as necessary. The local item value number array and the item value designation pointer array have the first type of data. As described above, the item values of the tabular data are expanded in the form of a global item value array, a local item value number array, and an item value designation pointer array for each data item.

In accordance with the above concept, one embodiment of the present invention is
Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
In a multi-core processing apparatus comprising:
It is represented as an array of records including item values corresponding to data items, and is a method of constructing tabular data that is shared and operated by the plurality of arithmetic units in the global memory,
A block number array in which the control unit divides the record into blocks including records in charge of each arithmetic unit and stores block numbers corresponding to the records in the order of the original record position numbers in the tabular data. Creating and storing in the global memory;
The plurality of arithmetic units operate in parallel to create a record sequence number array that stores the original record position numbers of the assigned records in the order of record sequence numbers on the local memory in each arithmetic unit, Transferring to the global memory;
Item value access information arrays for storing item value access information for accessing the item values included in the record in charge in the order of the record sequence numbers when the plurality of operation units operate in parallel. Creating on the local memory and transferring to the global memory;
The item values are stored in each operation unit for each data item so that the plurality of operation units operate in parallel and the item values included in the assigned record are accessed using the item value access information. Expanding on the local memory and transferring the expanded item value to the global memory;
A method comprising:

The above method
The control unit refers to the block number array on the global memory and determines a block number of a block including a predetermined record and an arithmetic unit responsible for the predetermined record;
The control unit notifying the determined arithmetic unit of a record sequence number of the predetermined record;
The arithmetic unit notified of the record sequence number transfers the record sequence number array and the item value access information array related to the record in charge of the arithmetic unit from the global memory to the local memory of the arithmetic unit; ,
The arithmetic unit notified of the record sequence number specifies the position where the notified record sequence number is stored in the transferred record sequence number array; and
The arithmetic unit notified of the record sequence number specifies the item value access information specified by the specified position in the transferred item value access information array;
The arithmetic unit notified of the record sequence number acquires the item value specified by the specified item value access information from the global memory for each data item, and uses the acquired item value as the global value. Transferring to memory;
Is further provided.

Also, the above method
The step of operating the plurality of arithmetic units in parallel, expanding the item value for each data item in the local memory in each arithmetic unit, and storing the expanded item value in the global memory,
The plurality of arithmetic units operate in parallel to transfer, for each data item, the item values included in a single block from the global memory to the local memory, and items included in the single block. Items included in the assigned record in the order of the local field value work array for storing unique values among the values in a predetermined order, and the original record position number of the assigned record included in the single block Creating a local item value number array in the local memory for storing a local item value number specifying a position where a value is stored in the local item value work array, and transferring the local item value number array to the global memory;
The plurality of arithmetic units operate in parallel and store, for each data item, the block number corresponding to a unique value among the item values included in the block, related to a pair of blocks. Block number work array, local item value work array, and local item value designation pointer work array for storing a pointer for designating a position where the item value included in the block is stored in the local item value work array A pair consisting of a further block number work array, a further local item value work array, and a further local item value specification pointer work array associated with the block into which the pair of blocks is merged. Performing a merge process to create
The plurality of arithmetic units operate in a parallel and hierarchical manner, and the above merge processing is repeated until each data item is merged into one final block. And transferring the final local field value work array and the final local field value specification pointer work array to the global memory,
The plurality of arithmetic units operate in parallel, and for each data item, an element in the final local item value designation pointer work array is designated by a corresponding element in the final block number work array. By distributing each block number and arranging in a predetermined order, the item value represented by the local item value number matches the global item value array storing the item values in a predetermined order. Creating an item value specification pointer array for storing a pointer for specifying a position stored in a local item value work array and transferring the pointer to the global memory;
including.

According to another embodiment of the present invention, a multi-core processing apparatus for performing the above method is provided. According to this embodiment, the multi-core processing apparatus is
Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
With
Tabular data that is expressed as an array of records including item values corresponding to data items and is shared and operated by the plurality of arithmetic units is constructed in the global memory. In this multi-core type information processing apparatus,
A block number array in which the control unit divides the record into blocks including records in charge of each arithmetic unit and stores block numbers corresponding to the records in the order of the original record position numbers in the tabular data. Including means for creating and storing in the global memory;
Each arithmetic unit is
Operate in parallel with other arithmetic units, create a record sequence number array in the local memory in each arithmetic unit to store the original record position number of the record in charge in the order of the record sequence number, and Means for transferring to memory;
An item value access information array that operates in parallel with other arithmetic units and stores the item value access information for accessing the item values included in the record in charge in the order of the record sequence numbers. Means for creating on local memory and transferring to the global memory;
Operate in parallel with other arithmetic units, so that the item values included in the record in charge are accessed using the item value access information, the item values for each data item are Means for expanding on a local memory and transferring the expanded item value to the global memory;
including.

In the multi-core processing apparatus,
The control unit is
Means for determining a block number of a block including a predetermined record and the arithmetic unit in charge of the predetermined record with reference to the block number array on the global memory;
Means for notifying the determined arithmetic unit of the record sequence number of the predetermined record;
Further including
The arithmetic unit notified of the record sequence number is
Means for transferring the record sequence number array and the item value access information array relating to the record in charge of the arithmetic unit from the global memory to the local memory of the arithmetic unit;
Means for identifying the position where the notified record sequence number is stored in the transferred record sequence number array;
Means for specifying item value access information specified by the specified position in the transferred item value access information array;
Means for acquiring the item value specified by the specified item value access information from the global memory for each data item, and transferring the acquired item value to the global memory;
Further included.

In the multi-core processing apparatus,
Each arithmetic unit is
Means for operating in parallel with other arithmetic units, expanding the item values for each data item in the local memory in each arithmetic unit, and storing the expanded item values in the global memory;
By operating in parallel with other arithmetic units, the item values included in a single block are transferred from the global memory to the local memory for each data item, and the item values included in the single block are transferred. The field values included in the assigned record are in the order of the local field value work array for storing the unique values of them in a predetermined order and the original record position number of the assigned record included in the single block. Means for creating a local item value number array for storing a local item value number for designating a position stored in the local item value work array on the local memory and transferring the local item value number array to the global memory;
A block number that operates in parallel with another arithmetic unit and stores the block number corresponding to a unique value among the item values included in the block, related to a pair of blocks, for each data item A work array, the local item value work array, and a local item value designation pointer work array for storing a pointer for designating a position where the item value included in the block is stored in the local item value work array. From this pair, a pair consisting of a further block number work array, a further local item value work array, and a further local item value specification pointer work array related to the block in which the pair of blocks is merged is created. Means for executing the merge process;
It operates in parallel and hierarchically with other arithmetic units, repeats the merging process until each data item is merged into one final block, and the final block number work array obtained, Means for transferring the final local field value work array and the final local field value specification pointer work array to the global memory;
A block that operates in parallel with another arithmetic unit and for each data item, the element in the final local item value designation pointer work array is designated by the corresponding element in the final block number work array. The final local, wherein the item values represented by the local item value numbers match the global item value array storing the item values in a predetermined order by distributing by number and arranging in a predetermined order Means for creating an item value designation pointer array for storing a pointer for designating a position stored in the item value work array and transferring it to the global memory;
Further included.

Furthermore, according to another embodiment of the present invention,
Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
A method for constructing tabular data in the global memory, which is loaded into a computer comprising: A computer program product for causing a computer to execute is provided.

Furthermore, according to another embodiment of the present invention,
Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
A method for constructing tabular data in the global memory, which is loaded into a computer including the data and is represented as an array of records including item values corresponding to data items, and which is shared and operated by the plurality of arithmetic units. A storage medium in which a computer program to be executed by a computer is recorded is provided.

  According to at least one embodiment of the present invention, tabular data can be continuously accessed in a predetermined order and a group of data divided so as to be accommodated in a local memory in each arithmetic unit of a multi-core processor. It is expressed by a certain data group. Thereby, an efficient access from each arithmetic unit to the global memory is realized. Therefore, according to at least one embodiment of the present invention, large-scale tabular data can be processed at high speed in a computer having a multi-core processor.

It is the schematic of the multi-core type processing apparatus by one Embodiment of this invention. 1 is a schematic diagram of a computer system according to an embodiment of the present invention. It is a figure showing an example of the tabular data for demonstrating the data management mechanism used as the foundation of one Embodiment of this invention. It is explanatory drawing of the basic data management mechanism used as the foundation of one Embodiment of this invention. It is explanatory drawing of the data structure for multi-core type information processing apparatuses by one Embodiment of this invention. It is explanatory drawing of the data structure for multi-core type information processing apparatuses by one Embodiment of this invention. It is explanatory drawing of the data structure for multi-core type information processing apparatuses by one Embodiment of this invention. It is explanatory drawing of the data structure for multi-core type information processing apparatuses by one Embodiment of this invention. 3 is a flowchart of a method for constructing a data structure for a multi-core processing device on a global memory according to an embodiment of the present invention. It is a flowchart of the item value acquisition method in the data structure for multi-core type processing apparatuses by one Embodiment of this invention. It is a schematic flowchart of the compilation process by one Embodiment of this invention. It is explanatory drawing of the order information creation process by one Embodiment of this invention. It is explanatory drawing of the order information creation process by one Embodiment of this invention. It is a schematic diagram of the compile processing in a block by one Embodiment of this invention. It is a schematic diagram of the compile processing in a block by one Embodiment of this invention. It is a schematic diagram of the compile processing in a block by one Embodiment of this invention. It is a schematic diagram of one Example of the compile processing in a block. It is explanatory drawing of the initialization process of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the merge process of the 1st step of the compilation process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 2nd merge process of the compile process in a block. It is explanatory drawing of the 3rd merge process of the compile processing in a block. It is explanatory drawing of the 3rd merge process of the compile processing in a block. It is explanatory drawing of the 3rd merge process of the compile processing in a block. It is explanatory drawing of the 3rd merge process of the compile processing in a block. It is explanatory drawing of the value list creation process of the compile process in a block. It is a schematic diagram of the merge process in the inter-block compilation process by one Embodiment of this invention. It is a schematic diagram of the merge process in the inter-block compilation process by one Embodiment of this invention. It is a schematic diagram of the merge process in the inter-block compilation process by one Embodiment of this invention. It is explanatory drawing of the result of the compile processing in a block to which the compile processing between blocks by one Embodiment of this invention is applied. It is explanatory drawing of the result of the compile processing in a block to which the compile processing between blocks by one Embodiment of this invention is applied. It is explanatory drawing of the 1st merge process in the compiling process between blocks by one Embodiment of this invention. It is explanatory drawing of the 1st merge process in the compiling process between blocks by one Embodiment of this invention. It is explanatory drawing of the 1st merge process in the compiling process between blocks by one Embodiment of this invention. It is explanatory drawing of the 2nd merge process in the compile processing between blocks by one Embodiment of this invention. It is explanatory drawing of the 2nd merge process in the compile processing between blocks by one Embodiment of this invention. It is explanatory drawing of the 2nd merge process in the compile processing between blocks by one Embodiment of this invention. It is explanatory drawing of the 2nd merge process in the compile processing between blocks by one Embodiment of this invention. It is a figure explaining the result of the merge process of the 3rd step in the inter-block compilation process by one Embodiment of this invention. It is a figure explaining the result of the merge process of the 3rd step in the inter-block compilation process by one Embodiment of this invention. It is explanatory drawing of the distribution process in the compile processing between blocks by one Embodiment of this invention. It is a figure explaining the result of the distribution process in the compile processing between blocks by one Embodiment of this invention. It is explanatory drawing of the block grouping process in the block compiling process by alternative embodiment of this invention. It is a figure explaining the result of the block grouping process in the compile processing between blocks by alternative embodiment of this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 Multicore type processor 101 Multicore type processor chip 110,120,130,140 Arithmetic unit 111,121,131,141 Core 112,122,132,142 Local memory 150 In-chip bus 160,161,162,163 Bus 170, 171, 172, 173 Global memory 200 Computer system 202 Multi-core processor 210 CPU
212 RAM
214 ROM
216 Fixed storage device 218 CD-ROM
220 CD-ROM driver 222 I / F
224 Input device 226 Display device 228 Bus 500 Tabular data 501 Data item “School”
502 Data item “Age”
510 records 0
511 Record 14
520, 521, ..., 527 Block 530 Order information 531 Item information "School"
532 Item information "Age"
540 Block number array 551-0, 551-1, ..., 551-7 Record sequence number array 552-0, 552-1, ..., 552-7 Item value access information array 560-0, 560-1 , ..., 560-7 Block information "School"
561-0, 561-1, ..., 561-7 Local item value number array "School"
562-0, 562-1, ..., 562-7 Item value designation pointer array "School"
570 Global Item Value Array “School”
580-0, 580-1,..., 580-7 Block information “Age”
581-0, 581-1,..., 581-7 Local item value number array “Age”
582-0, 582-1, ..., 582-7 Item value designation pointer array "Age"
590 Global Item Value Array “Age”

  Hereinafter, various embodiments for carrying out the present invention will be described in detail with reference to the drawings.

[Multi-core processing equipment]
First, a multi-core type processing apparatus that implements data processing according to an embodiment of the present invention will be described. FIG. 1 is a schematic view of an embodiment of a multi-core processing apparatus. The multi-core processing apparatus 100 is provided with a plurality of (for example, two, four, eight, etc., four in this example) arithmetic units 110, 120, 130, 140 on a multi-core processor chip 101. . Each arithmetic unit 110, 120, 130, 140 includes a core 111, 121, 131, 141 for data processing and a local memory 112, 122, 132, 142 dedicated to the core. Each arithmetic unit 110, 120, 130, 140 is connected by a bus 150 in the chip. This bus 150 is preferably a ring bus. Since the arithmetic units are connected by a bus 150 in the chip, data communication can be performed at high speed. Furthermore, each arithmetic unit 110, 120, 130, 140 is connected to global memory 170, 171, 172, 173 externally attached to the chip 101 via buses 160, 161, 162, 163 that support DMA transfer. ing.

  The storage capacity of the local memories 112, 122, 132, 142 in the chip is about 256 KB (kilobytes), for example, while the global memories 170, 171, 172, 173 are large-capacity memories of several tens GB (gigabytes). is there. In the figure, the global memories 170, 171, 172, and 173 are distinguished from each other. This is because when each core accesses the global memory with a single bus, the communication performance of the bus becomes a bottleneck. Therefore, a dedicated memory interface (not shown) is provided for each core. It shows access through this memory interface. Of course, even with such a configuration, it is possible to manage the global memory so that it appears as one logically continuous memory as a whole, as in the NUMA (non-uniform memory access) system. In an alternative embodiment, each computing unit is connected to a physically integrated external global memory via one bus.

Furthermore, in a processor such as the above-mentioned Cell Broadband Engine ^™ , a general-purpose processor core and an arithmetic processor core are mounted in one chip. The general-purpose processor core can control operations of a plurality of arithmetic processor cores. Therefore, the multi-core type processor preferably includes a control unit such as a general-purpose processor core, but the control unit does not need to be mounted in the chip and may be provided outside the chip.

  The control unit and the arithmetic unit, or the arithmetic units can communicate with each other using, for example, a mailbox or a signal mechanism.

[Computer system configuration]
FIG. 2 is a schematic diagram of a computer system 200 for manipulating tabular data according to one embodiment of the present invention. The computer system 200 performs multi-core processing as shown in FIG. 1 in which tabular data represented as an array of records including item values corresponding to data items is shared and operated by a plurality of arithmetic units. A device 202 is provided. As shown in FIG. 2, the computer system 200 further stores a CPU 210 that controls the entire system and individual components by executing a program, work data, and the like, for example, a RAM (Random Access Memory). ), A ROM (Read Only Memory) 214 that stores programs, a fixed storage medium 216 such as a hard disk, a CD-ROM driver 220 for accessing the CD-ROM 218, and a CD-ROM driver 220 and an interface (I / F) 222 connected to an external terminal connected to an external network or the like (not shown), an input device 224 such as a keyboard and a mouse, and a display device 226 such as a computer monitor. And. The multi-core processing device 210, the RAM 212, the ROM 214, the external storage medium 216, the I / F 222, the input device 224, and the display device 226 are connected to each other via a bus 228.

  A program for causing the multi-core processing device 202 and the CPU 210 of the computer system 200 to operate the tabular data is stored in the CD-ROM 218 and may be read by the CD-ROM driver 220 or stored in the ROM 214 in advance. Also good. Further, what is once read from the CD-ROM 218 may be stored in a predetermined area of the external storage medium 216. Alternatively, the program may be supplied from the outside via a network (not shown), an external terminal, and the I / F 220.

  The multi-core processor system according to an embodiment of the present invention is realized by causing the computer system 200 to execute a program for manipulating tabular data.

  In the computer system 200 shown in FIG. 2, a CPU 210 is provided in addition to the multi-core processing device 202, and controls the entire system and individual components. However, the present invention is not limited to such an embodiment, and in an alternative embodiment, a control unit included in the multi-core processing apparatus 202 controls the entire system and individual components.

[Data management mechanism based on information blocks]
FIG. 3 is a diagram showing an example of tabular data for explaining a data management mechanism as a basis of one embodiment of the present invention. This tabular data is stored as a data structure as shown in FIG. 4 in the computer by using the data management mechanism proposed in the above-mentioned International Publication No. WO00 / 10103. This data structure has been proposed to realize retrieval, sorting, aggregation, etc. of large-scale tabular data using hardware resources of commercially available computers, for example, personal computers, in particular, processors and memories. It should be noted that the data structure of tabular data placed on the computer memory.

  It should be noted that “information record number representing the position in which the record is accommodated in the original tabular data (ie, source record position number)” and “information representing the record order (ie, record order number)” A clear distinction is made in this document. Every record has a source record position number associated with it. This primitive record position number is virtual information used for specifying individual records including item values corresponding to data items. In general, in the tabular data, records are not always arranged in the order of the original record position numbers. For example, when the original tabular data is sorted in ascending order with respect to the item value of a certain item, the order of the records of the tabular data obtained is different from the order of the records of the original tabular data. However, the records in the original tabular data may be arranged in the order of the source record position numbers. In this case, the source record position number and the record sequence number are initially matched. Yes.

  As shown in FIG. 4, the order number (record order number) of each record of the tabular data and the original record position number are abbreviated as a record order designation array 401 (hereinafter, this array is abbreviated as “OrdSet”). .). The record order specification array 401 stores the original record position numbers in the order of the record order numbers. In the example of FIG. 4, the records are arranged in the order of the original record position numbers.

  Here, the notation of the arrangement | sequence in this specification is demonstrated. In general, an array element A can be expressed as A [i], where i is a subscript. However, in the drawing, an array element A [i] is within a region surrounded by a solid line. The boundary between the element A [i] and the element A [i + 1] is indicated by a dotted line. The subscript i of element A [i] is shown on the left side of element A [i]. Further, the subscript i of the array is represented by an integer starting from 0.

  Returning to FIG. 4 again, regarding the gender, it can be seen that the original record position number corresponding to the record order number = 0 of the tabular data is “0” from the array OrdSet [0]. The actual gender value for the record whose source record position number is “0”, ie, “male” or “female” is a value list in which the actual values are sorted according to a predetermined order (eg, ascending or descending order). Item value number array 402 (hereinafter, item value number array, that is, pointer array) that is a pointer array to a certain item value array 403 (hereinafter, item value array, that is, a value list is abbreviated as “VL”). Is abbreviated as “VNo.”). The pointer array 402 stores pointers that point to elements in the actual value list 403 according to the order of the source record position numbers stored in the array OrdSet 401. As a result, the item value of the gender corresponding to the record “0” in the tabular data is (1) the original record position number = 0 corresponding to the record sequence number = 0 is extracted from the array OrdSet 401, and (2) is stored in the value list. The element “1” corresponding to the original record position number = 0 is extracted from the pointer array 402, and (3) the element “female” indicated by the element “1” extracted from the value array 403 from the pointer array 302 to the value list is extracted. Can be obtained by taking out "".

  The item values can be acquired in the same manner for other records and also for age and height.

  In this way, the tabular data is expressed by a combination of the value list VL and the pointer array VNo to the value list, and this combination is particularly referred to as an “information block”. In FIG. 4, information blocks regarding gender, age, and height are shown as information blocks 408, 409, and 410, respectively.

  If a single computer has a single memory (physically multiple, but a single memory in the sense that it is located and accessed in a single address space) May store the ordered set array OrdSet, the value list VL constituting each information block, and the pointer array VNo in the memory. However, in various embodiments of the present invention, the manipulation of tabular data is performed by a multi-core processing device configured by a plurality of arithmetic units with a small capacity dedicated local memory. Therefore, a new mechanism for holding tabular data has been proposed in order to achieve efficient parallel processing.

[Data structure for multi-core processors]
Next, a data structure for a multi-core processing apparatus according to an embodiment of the present invention will be described. 5A to 5D are explanatory diagrams of a data structure according to an embodiment of the present invention. FIG. 5A shows an example of original tabular data. The tabular data 500 shown in FIG. 5A includes item values corresponding to the data item 501 “School” (for example, “West”, “South”, “North”, “East”, and data items “Age”). It is represented as an array of records including item values (for example, “12”, “8”, “11”, “10”, etc.) corresponding to 502. This original tabular data 500 is easy to explain. Therefore, it is assumed that the records are arranged in the order of the source record position numbers, that is, the source record position numbers that specify the records and the record order numbers that indicate the order of the records match. The record 510 located at the head is a record to which the original record position number 0 is assigned.The data item “School” of the record 510 is recorded. Eye value is "West", item value of the data item "Age" is "12".

  In the data structure for a multi-core processing device according to an embodiment of the present invention, this tabular data record includes blocks 520, 521, identified by a block number (in this example, eight block numbers from 0 to 7). ... divided into 527. Initially, this block corresponds to an arithmetic unit of a multi-core type processing apparatus that is responsible for processing the records included in this block.

  The data structure for a multi-core type processing apparatus relates to information (order information) relating to the order in which records are arranged (that is, record order numbers) and storage locations of item values in the data structure, and item values for each data item. Information (item information). The order information functionally corresponds to the record order specification array OrdSet in the data management mechanism that is the basis of the above-described embodiment of the present invention, and the item information similarly corresponds to the information block. Both the order information and the item information are held in the global memory, and a part of them is transferred to the local memory of each arithmetic unit as necessary. FIG. 5B shows the order information 530, and FIGS. 5C and 5D show the item information 531 and 532 of the data item “School” and the data item “Age”, respectively.

  The order information 530 includes a block number array 540 in which the block numbers are stored in the order of the record order numbers. In the data structure of this embodiment, an arithmetic unit responsible for the operation of the record is determined for each record. Therefore, the record (s) is divided into records that each arithmetic unit is in charge of, that is, the records in charge, and a block number is assigned to each record in charge. If the block number array is BlkNo and the record order number is i, BlkNo [i] indicates that the block number of the block to which the record having the record order number i belongs is BlkNo [i]. The block number array 540 is an integer type array having a size equal to the number of records. The block number array 540 is the second type of data. For example, in the examples of FIGS. 5A to 5D, records with record sequence numbers 0 to 3 are included in the block with block number 0, records with record sequence numbers 4 to 7 are included in the block with block number 1, and so on. .

  According to the data structure of the present embodiment, all records are divided into assigned records corresponding to the blocks. Therefore, information for associating each assigned record with the original tabular data record is required for each block. Therefore, the order information 530 includes a record order number array 551-1, 551-2, ..., 551-7 in which the record order numbers of the records in charge are stored in the order of the record order numbers for each block. The record order number array may be referred to as GOrd below. For example, in the examples of FIGS. 5A to 5D, the record sequence numbers of the assigned records belonging to the block of block number 0 are 0, 1, 2, and 3, and the record sequence numbers of the assigned records belonging to the block of block number 1 are 4, 5, 6, 7 and so on. The record sequence number array has the same size as the number of records in charge belonging to each block, and is an integer type array. The record sequence number array is divided into sizes that can be accommodated in the local memory of each arithmetic unit, and is therefore the first type of data. Therefore, the record sequence number array is transferred from the global memory to the local memory in each arithmetic unit as necessary.

  Furthermore, since the item value included in each record is held in the form of item information to be described later, each arithmetic unit has address information for accessing the item value included in the assigned record, that is, item value access information. It is necessary to get Therefore, according to the data structure of the present embodiment, the order information 530 includes the item value access information arrays 552-1 and 552 in which the item value access information of the assigned record is stored in the order of the record order number for each block. 2, ..., 552-7. This item value access information array is an integer type array, and the size of the item value access information array matches the number of records in charge. The item value access information array is also a first type of data, and is transferred from the global memory to the local memory in each arithmetic unit as necessary. The item value access information array may be called by the name LOrd. For example, in the example of FIGS. 5A to 5D, the item value included in the record whose record sequence number is 0 included in the block of block number 0 can be accessed by the item value access information of 0 with respect to this block number 0. The item value included in the record whose record sequence number is 5 included in the block of block number 1 can be accessed by the item value access information of 1 regarding this block number 1.

  Next, according to this embodiment, item information is held as item information for each data item. For example, in the example of FIGS. 5A to 5D, item information 531 relating to the data item “School” and item information 532 relating to the data item “Age” are constructed in the global memory. The item values included in the record in charge for each block are held in the global memory so that each arithmetic unit can access each data item using the item value access information array. To the local memory in each arithmetic unit. The item value itself is constructed on the global memory as a global item value array in which unique item values are stored in a predetermined order (ascending order or descending order) for each data item. For example, in the example of FIGS. 5A to 5D, the item value related to the data item “School” is held in the global memory as the global item value array 570, and the item value related to the data item “Age” is stored as the global item value array 590 in the global memory. Is held in. This global item value array is the second type of data. The global item value array is an array for storing the item value itself, and thus has various data types such as an integer type, a floating point type, and a character string type.

  The item information is configured so that the item value stored in the global item value array can be specified using the item value access information related to the record in charge. Therefore, the item information is represented for each data item by a local item value number array in which local item value numbers for specifying the item values included in the assigned record are stored in the order of the source record position numbers, and the local item value numbers. And an item value designation pointer array in which item value designation pointers for designating positions where the item values to be stored are stored in the global item value array are stored in the order of the local item value numbers. A local item value number array and an item value designation pointer array are provided for each block. The local item value number array is an integer type array having a size that matches the number of records in charge, is the first type of data, and is sometimes referred to as VNo. The item value designation pointer array is an integer type array having the same size as the number of unique item values included in the record in charge, is a first type of data, and is sometimes called LVL. Since both the local item value number array and the item value designation pointer array are data of the first type, they are constructed on the global memory for each block, and from the global memory to the local memory in each arithmetic unit as necessary. Transferred.

  In the example of FIGS. 5A to 5D, regarding the data item “School”, the item information 531 includes local item value number arrays 561-0, 561-1,... 561-7 and an item value designation pointer array 562-0. , 562-1,... 562-7 and a global item value array 590. The local item value number array and the item value designation pointer array are divided for each block. In the figure, for example, the value of the first element of the local item value number array VNo is “1”. This means that the item value number of the item value included in the record specified by the item value access information whose value is “0” is “1”. The item value whose item value number is “1” refers to the second element of the item value designation pointer array LVL, that is, the third element of the global item value array GVL, ie, LVL [1]. It turns out that it is GVL [2]. The same applies to other blocks and other data items.

  As described above, according to the data structure of the present embodiment, the item value included in the record belonging to each block includes the local item value number assigned to each item value in the block, the local item value number, and the global item. This is expressed by an item value designation pointer that associates an item value in the value array with a global item value array.

[Construction of data structure for multi-core processor]
FIG. 6 is a flowchart of a method for constructing a data structure for a multi-core processing device on a global memory according to an embodiment of the present invention. According to this method, the control unit of the multi-core processing device divides the tabular data record into blocks including the record in charge of each arithmetic unit of the multi-core processing device, and the block number corresponding to each record is tabulated. A block number array to be stored in the order of records in the data is created and stored in the global memory (step 602). Next, multiple arithmetic units create a record sequence number array that stores the record sequence numbers of the records in charge in the order of the record sequence numbers in parallel in each arithmetic unit and transfer them to the global memory. (Step 604). This process is easily realized, for example, by notifying the arithmetic unit from the control unit of the record sequence number at the head of the record in charge of each arithmetic unit and the number of records in the record in charge. After that, multiple arithmetic units operate in parallel, and the item value access information array for storing the item value access information for accessing the item values included in the assigned record in the order of the record sequence numbers is stored locally in each arithmetic unit. Step of creating on the memory and transferring to the global memory (step 606). Each arithmetic unit has to store as many sequential numbers starting from 0 in the item value access information array as the number corresponding to the number of records in the assigned record. Finally, field values for each data item are stored locally in each computing unit so that multiple computing units operate in parallel and the field values included in the record in charge are accessed using field value access information. The data is expanded on the memory, and the expanded item value is transferred to the global memory (step 608). For example, the arithmetic unit cooperates, for each data item, a local item value number array for storing local item value numbers attached to each item value in the block in the order of the original record position number, and this local item. An item value specification pointer array that stores the item value specification pointer that associates the value number with the item value in the global item value array in the order of the local item value number is created in the local memory, transferred to the global memory, and all records A global item value array in which the unique item values included in is stored in a predetermined order (ascending or descending order) is created on the global memory.

  Note that the order of step 602 and step 604 can be interchanged. Also, if the number of records in charge is relatively small and each arithmetic unit can store the record sequence number array and the field value access information array at the same time in the local memory, each arithmetic unit creates both arrays at the same time, You may transfer to global memory. Alternatively, the control unit may directly create a record sequence number array and an item value access information array for each block on the global memory.

  In the above description, it is assumed that the source record position number and the record sequence number match, but the source record position number and the record sequence number may not match. For example, it is possible to construct a data structure for a multi-core processing device even if the original tabular data records are sorted and the original record position number does not match the initial record sequence number. Specifically, the control unit divides the tabular data record into blocks including the records in charge of each arithmetic unit, and assigns the block number corresponding to each record in the order of the record order number of the tabular data record. A block number array to be stored may be created and stored in the global memory.

[Item value acquisition processing]
Next, acquisition of item values of tabular data in a data structure for a multi-core processing device according to an embodiment of the present invention will be described. FIG. 7 is a flowchart of an item value acquisition method according to an embodiment of the present invention. As described with reference to FIGS. 5A to 5D, the item value is stored in the global memory in the form of item information for each data item. Therefore, for example, the control unit can easily acquire the item value included in the only record. However, the control unit is not suitable for acquiring item values included in a large number of records at the same time. Therefore, in the present embodiment, a situation is considered in which item values included in a large number of records are acquired simultaneously by a large number of arithmetic units operating simultaneously. Even in such a situation, it will be understood that the basic operation of acquiring the item value is a process in which a specific arithmetic unit acquires an item value included in a certain record in the assigned record. In order to allow a large number of arithmetic units to operate simultaneously, each arithmetic unit transfers information necessary for acquiring item values from the global memory to the local memory, and executes various arithmetic operations on the local memory.

  As shown in FIG. 7, first, the control unit refers to the block number array in the global memory, and the block number of the block including the predetermined record from which the item value is to be acquired, and this block Is determined (step 702). Next, the control unit notifies the record sequence number of a predetermined record to the arithmetic unit identified by the determined block number (step 704). Thereafter, the arithmetic unit transfers the record sequence number array and the item value access information array relating to the record in charge of this arithmetic unit from the global memory to the local memory of this arithmetic unit (step 706). Subsequently, the arithmetic unit specifies the position where the notified record sequence number is stored in the record sequence number array transferred to the local memory (step 708). Thereafter, the arithmetic unit specifies item value access information specified by the specified position in the item value access information array transferred to the local memory (step 710). Finally, for each data item, the arithmetic unit acquires the item value specified by the item value access information specified from the global item value array held in the global memory, and stores the acquired item value in the global memory. (Step 712).

  An example of data acquisition according to the present embodiment will be described in more detail using the data structure shown in FIGS. 5A to 5D. For example, in the record with the record sequence number = 14, FIGS. 5A to 5D, consider obtaining the item value of the record indicated by the reference numeral 511. The control unit reads the value of the element of subscript = 14 in the block number array 540, that is, BlkNo [14] = 3. Thereby, it is understood that the block number corresponding to the target record is 3. Therefore, the control unit notifies 14 which is the record sequence number of the target record to the arithmetic unit in charge of the record included in the block number 3. The correspondence between the block number and the arithmetic unit is managed by the control unit, for example. Thereafter, the arithmetic unit notified of the record sequence number 14 loads the record sequence number array 551-3 relating to the block having the block number = 3 from the global memory to the local memory. In an alternative embodiment, the control unit transfers the record sequence number array 551-3 for the block with block number = 3 from the global memory to the local memory of the arithmetic unit.

  Next, the arithmetic unit searches the position where record sequence number = 14 is stored in the record sequence number array 551-3. This storage position is also referred to as the rank (rank) of the target record in the record in charge of this arithmetic unit. In this embodiment, the record order number array is an ascending order array, so that this storage position can be found efficiently by a well-known two-division method or the like. In this example, the storage position = 2.

  Next, the arithmetic unit transfers the item value access information array 552-3 relating to the block whose block number = 3 from the global memory to the local memory. In an alternative embodiment, the item value access information array 552-3 is transferred from the global memory to the local memory of the arithmetic unit by the control unit. In yet another embodiment, the item value access information array 552-3 is transferred from the global memory to the local memory of the arithmetic unit simultaneously with the record sequence number array 551-3. Although not repeatedly described below, data transfer from the global memory to the local memory may be performed by either the control unit or the arithmetic unit unless otherwise specified. Further, when the local memory capacity of the arithmetic unit is larger than the amount of data to be transferred, two or more pieces of data may be transferred from the global memory to the local memory at once. The arithmetic unit acquires the value stored in the position indicated by the rank of the target record in the item value access information array 552-3, that is, LOrd [2] = 2.

  Next, the arithmetic unit loads the local item value number array 561-3 from the global memory to the local memory with respect to the data item = “School” and the block with the block number = 3. The arithmetic unit uses the previously acquired value, that is, the value of LOrd [2] = 2 as an offset value for acquiring the local item value from the local item value number array 561-3. That is, the arithmetic unit acquires VNo [2] = 0 from the local item value array 561-3.

  Next, the arithmetic unit transfers the item value designation pointer array 562-3 from the global memory to the local memory for the data item = “School” and the block whose block number = 3. The arithmetic unit obtains, from the item value designation pointer array 562-3, the item value designation pointer designated by the previously obtained local item value = 0, that is, LOrd [0] = 1.

  Finally, the arithmetic unit refers to the global item value array 570 stored in the global memory with respect to the data item = “School”, and the item value included in the target record, that is, GVL [1] = “North”. To get. In an alternative embodiment, the arithmetic unit notifies the control unit of the previously acquired item value designation pointer = 1, and the control unit is designated from the global item value array 570 by the notified item value designation pointer. Get item value.

  Subsequently, the arithmetic unit obtains the item value = 9 included in the target record by executing the same operation as the processing executed for the data item = “School” for the data item = “Age”.

[Compile processing of tabular data]
Next, a compiling process for creating data for a multi-core processing device from tabular data according to an embodiment of the present invention will be described. In the following, a compilation process according to an embodiment of the invention will be described in relation to the data structure shown in FIGS. 5A to 5D. FIG. 8 is a schematic flowchart of the compiling process according to an embodiment of the present invention.

  Order information creation: According to this embodiment, first, order information including a block number array, a record order number array, and an item value access information array is created on the global memory (step 802). As described above, the block number array is created by the control unit, and the record sequence number array and the item value access information array are created in parallel by a plurality of arithmetic units and transferred to the global memory.

  In-block compilation: Next, multiple arithmetic units operate in parallel, and for each data item, local item value numbers are stored in the order of the original record position number of the assigned record contained in a single block. A local item value number array is created and transferred to the global memory (step 804). At the same time, the plurality of arithmetic units also create a local item value work array for storing unique values among the item values included in the assigned record in a predetermined order (for example, ascending or descending order), and Forward to.

  Inter-block compilation 1 (merge): Next, a plurality of arithmetic units operate in parallel and hierarchically, and for each data item, a block number work array and a local item value work array related to two blocks. And a pair of local item value designation pointer work arrays in which pointers for designating positions where the item values are stored in the local item value work array are stored in the order of local item value numbers, 2 A merge process for creating a set of a block number work array, a local item value work array, and a local item value designation pointer work array related to the block obtained by merging the blocks is executed. The arithmetic unit repeatedly executes this merging process until it is finally merged into one block, final block number work array, final local item value work array, and final local item value designation The pointer array is transferred to the global memory (step 806). The final local item value work array matches the global item value array.

  Inter-block compilation 2 (distribution): Finally, multiple arithmetic units operate in parallel, and for each data item, the elements in the final local item value designation pointer work array are final block number work arrays. The item value represented by the local item value number is the final local item value work array, that is, the global item value array by distributing the block numbers specified by the corresponding elements in the block and arranging them in a predetermined order. An item value designation pointer array for storing pointers for designating the positions stored therein is created and transferred to the global memory (step 808).

  Through the above steps, from the tabular data shown in FIG. 5A, the block number array, the record sequence number array, and the item value access information array shown in FIG. 5B, and the data item = “School” shown in FIG. 5C are obtained. Local item value number array, item value designation pointer array and global item value array for "", and local item value number array, item value designation pointer array and global item value array for data item = "Age" shown in FIG. 5D Are created in the global memory.

By adopting such a data structure for a multi-core processing apparatus, the following advantages can be obtained.
Advantage 1: Memory saving Since the same value is not redundantly stored in the global item value array, that is, the value list, the memory is saved.
Advantage 2: High-speed calculation In the sorting between blocks, the calculated rank is effective for all blocks.
Advantage 3: Reduction of global memory access by communication between arithmetic units The first phase of sorting between blocks requires the most O (n * log (n)) global memory accesses. The global memory access for other processes is O (n). If inter-block communication is used, the global memory access amount can be exactly ３.

  Hereinafter, the compiling process according to the embodiment of the present invention will be described in more detail with reference to the tabular data shown in FIGS. 5A to 5D. 9A and 9B are explanatory diagrams of order information creation processing according to an embodiment of the present invention. The data shown in FIGS. 9A and 9B is the same as the data shown in FIGS. 5A to 5D, and the order information 530 of FIG. 9B is created from the tabular data 500 of FIG. 9A. The order information creation process is as described above.

  10A to 10C are schematic diagrams of the intra-block compilation processing according to the embodiment of the present invention. According to the intra-block compilation process, the item information related to the data item = “School” and the item information related to the data item = “Age” shown in FIG. 10B and FIG. 10C are created from the tabular data shown in FIG. 10A. The As shown in the figure, the item information includes a local item value number array VNo and a local item value work array wVL. The intra-block compilation process is executed in parallel by each arithmetic unit for each block of Block-0, Block-1,..., Block-7.

Here, an embodiment of the in-block compilation process for one block will be described. FIG. 11 is a schematic diagram of an embodiment of the in-block compilation process. In this example, the local item value number is obtained from the block School (that is, the item value array School in which the item values are stored in the order of the original record position numbers) with respect to the data item = “School” in five rows (number of records = 5). An array VNo and a local item value work array wVL are created. The local item value work array wVL is a list of values in which unique item values extracted from the item values included in the item value array School are stored in a predetermined order (for example, ascending order or descending order). On the other hand, the local item value number array VNo, when i represents the original record position number, is between the Element [1], which is an element of the original item value array School, and the local item value work array wVL [j],
School [i] = wVL [VNo [i]]
It is an array that holds the relationship. The following processing is executed by the arithmetic unit using the local memory of the arithmetic unit.

  First, the working data is initialized. FIG. 12 is an explanatory diagram of the initialization process of the intra-block compilation process. The size of the local item value number array VNo is equal to the number of rows in the block, and is initialized with an integer starting from 0, such as VNo [i] = i (i = 0, 1, 2,...). The conversion array TR is an array for converting the local item value number array and has the same size as the VNo. The pointer array PTR is an array that represents the order in which item values are arranged, and an element of the array PTR represents the position of the item value in the item value array School. A pair of the pointer array PTR and the original item value array School represents a list of item values arranged in a predetermined order. The work pointer array wPTR is used in combination with the original item value array School to represent a list of item values after merging pairs in the block.

  In-block compilation processing is realized by sequentially merging pairs in a block. For example, when there are 5 records in a block, first a pair of the first record and the second record (first pair), a pair of the third record and the fourth record (second pair) ) And the fifth pair of records (there is no third party in the pair, but for convenience, the third pair) is compared in the magnitude relation of the item values. This is the first-stage merge process. Next, a comparison process between the third pair and the second pair and a comparison process between the first pair (there is no pair paired with the first pair) are executed. This is the second-stage merge process. In this way, by repeating the merging process step by step until the last pair of pairs, the local item value number array VNo and the local item value work array wVL merged in the block are finally obtained. .

13A to 13G are explanatory diagrams of the first-stage merge process of the in-block compilation process. FIG. 13A shows the first comparison process of the first pair, that is, the comparison process of School [PTR [0]] and School [PTR [1]]. In this example, the two values (that is, pairs) of the item value array School indicated by the pointer array PTR are compared in size, and the conversion array TR in the same position as the element of the pointer array PTR indicating the smaller one is compared. The subscript of the work pointer array wPTR is stored in the element. More specifically, if we assume that the magnitude relationships of strings are compared alphabetically,
School [PTR [0]] = School [0] = “West”
And School [PTR [1]] = School [1] = “South”
Comparing
“West”> “South”
Because
wPTR [0] = PTR [1] = 1
TR [1] = 0
It becomes. That is, i and j are generally integers starting from 0,
IF School [PTR [i]]> School [PTR [j]]
THEN wPTR [i] = PTR [j]
TR [j] = i
ELSE IF School [PTR [i]] <School [PTR [j]]
THEN wPTR [j] = PTR [i]
TR [i] = j
Can be described. In this process, the smaller one passes the comparison process, and the larger one is compared with the next partner again.

FIG. 13B is an explanatory diagram of the second comparison process of the first pair. The target of the first second comparison process is School [PTR [0]], which is determined not to be smaller in the first comparison process. At this time, since there is no comparison partner, it is determined that School [PTR [0]] is the smaller one,
wPTR [1] = PTR [0] = 0
TR [0] = 1
It becomes.

  Similarly, FIG. 13C shows the first comparison process for the second pair, FIG. 13D shows the second comparison process for the second pair, and FIG. 13E shows the first comparison process for the third pair. ing. Since the third pair does not actually form a pair and there is only one comparison target, it is determined that School [PTR [4]] is the smaller one as it is.

13F and 13G are explanatory diagrams of post-processing of the first-stage merge processing in the in-block compilation processing. In the post-processing, first, the local item value number array VNo is updated. In particular,
VNo [i] = TR [VNo [i]]
Will convert the local field value number. This means that the local item value number corresponding to the item value determined to be small by the pair comparison process is arranged in front in the local item value number array VNo. Next, the pointer array PTR is updated. In particular,
PTR [i] = wPTR [i]
Thus, the pointer array PTR is updated by overwriting the pointer array PTR with the work pointer array wPTR.

  Next, the second stage merge process of the in-block compilation process will be described. In the second-stage merge process, the third pair and the second pair used in the first-stage merge process form the first pair in the second stage, and the first pair in the first stage. A second pair of second stages is formed alone. According to the preferred embodiment of the present invention, the second-stage pairing is performed in the reverse order to the first-stage pairing (that is, from the rear). The reason is to balance the processing amount. Of course, pairing may be performed from the front as in the first stage.

  In the second-stage merge processing, for example, the item values of one pair of the first-stage merge processing are a1 and a2 (a1 <a2), and the item values of the other pair are b1 and b2 (b1 <b2). ) First, a1 and b1 are compared, and if a1 <b1, then a2 and b1 are compared to determine the magnitude relationship between a1, a2, b1, and b2. The Also, if the same value exists such as b1 = a2, duplication is eliminated by selecting the smaller value of the element of the corresponding pointer array PTR.

14A to 14F are explanatory diagrams of the second-stage merge process of the in-block compilation process. FIG. 14A shows the first comparison process of the first pair in the second-stage merge process. In the example shown in the figure, comparison processing of School [PTR [2]] = “South” from the second pair in the first stage and School [PTR [4]] = “South” from the third pair is performed. . In this example, since they are the same value, i, j (i <j) are generally integers starting from 0,
IF School [PTR [i]] = School [PTR [j]]
THEN wPTR [i] = PTR [i]
IF PTR [i] <PTR [j]
THEN TR [i] = PTR [i]
TR [j] = PTR [i]
ELSE TR [i] = PTR [j]
TR [j] = PTR [j]
According to
wPTR [2] = 3
TR [2] = 2
TR [4] = 2
It is set like this.

  FIG. 14B shows the second comparison process of the first pair in the second-stage merge process. Specifically, a single comparison process of School [PTR [3]] = “West” is performed. Since there is no comparison partner, it is determined that School [PTR [3]] is the smaller one, and processing similar to that described with reference to FIG. 13B is performed.

  14C shows the first comparison process of the second pair of the second-stage merge process, and FIG. 14D shows the second comparison process of the second pair of the second-stage merge process. Since no comparison partner exists in any of the comparison processes, it is determined that School [PTR [1]] and School [PTR [0]] are smaller, and the same process as described above is performed.

14E and 14F are explanatory diagrams of post-processing of the second-stage merge processing in the in-block compilation processing. Similar to the post-process of the first-stage merge process, the post-process of the second-stage merge process first updates the local item value number array VNo. In particular,
VNo [i] = TR [VNo [i]]
Will convert the local field value number. next,
PTR [i] = wPTR [i]
Thus, the pointer array PTR is updated by overwriting the pointer array PTR with the work pointer array wPTR.

  Next, the third-stage merge process of the in-block compilation process will be described. In the third-stage merge process, the first pair in the third stage is formed by the first pair and the second pair used in the second-stage merge process. According to the preferred embodiment of the present invention, the third-stage pairing is thus performed in the reverse order (that is, from the rear) as the second-stage pairing. In this example, since only one pair remains, it is not necessary to consider the order of pairing.

  15A to 15D are explanatory diagrams of the merge process at the third stage of the in-block compilation process. In the third-stage merge process, the same process as the first-stage and second-stage merge processes is performed. However, the number of data to be compared generally increases as the number of stages increases. In this example, as shown in FIGS. 15A and 15B, the same value appears frequently, so the comparison process is completed twice even in the third stage. 15C and 15D show the post-process of the third-stage merge process. Thus, the final local item value number array VNo and the final pointer array PTR are obtained. In the pointer array, elements having no stored value are indicated by *.

Finally, according to the in-block compilation process according to the embodiment of the present invention, a final local item value work array wVL, that is, a value list is created. FIG. 16 is an explanatory diagram of a value list creation process in the in-block compilation process. The value list wVL is obtained by reading values from the item value array School using the values of the elements of the pointer array PTR as pointers and storing them in the value list wVL in the order of the elements of the pointer array PTR. Specifically, in this example,
wVL [0] = School [PTR [0]] = School [1] = “South”
wVL [1] = School [PTR [1]] = School [0] = “West”
It is.

  The local item value number array VNo and the final local item value work array wVL are obtained by the above-described in-block compilation process.

  Next, inter-block compilation processing according to an embodiment of the present invention will be described. The inter-block compilation process repeats the process of merging a pair of blocks until multiple arithmetic units operate in parallel and hierarchically and are finally merged into one block for each data item. A merge process for generating a final block number work array, a final local item value work array, and a final local item value designation pointer array, and a final block number work array and a final generated by the merge process. Distribution processing for generating an item value designation pointer array from a local item value designation pointer. The final local item value work array generated by the merge process matches the global item value array.

  In the merge process, each arithmetic unit merges information related to a pair of blocks, and generates information related to one block of the merged higher layer. Therefore, the merge process is realized by a parallel operation of a plurality of arithmetic units. Each arithmetic unit also merges information about merged more block pairs belonging to the same layer and generates information about one block of the merged higher layer. In this way, by repeating the merge processing in parallel and hierarchically, information on one block at the top layer is finally generated. One block in the uppermost layer is a block including the entire record.

For example, ^assuming that there are 2 ⁿ -1 arithmetic units, each arithmetic unit inputs information about two blocks, merges them and outputs information about one block, each arithmetic unit has By executing the merge process once, an n-stage (layer) merge process is realized. In this case, of the total data communication amount by all the arithmetic processors, the ratio occupied by the communication performed by the arithmetic processor with the global memory is 1 / n. The communication amount between the arithmetic units is (n−1) / n of the total data communication amount.

  17A to 17C are schematic diagrams of merge processing in inter-block compilation processing. In the example of the figure, for example, at least seven arithmetic units SPU-0, SPU-1,..., SPU-6 execute a merge process of 128 blocks from block Block-0 to block Block-127. ing.

  The first processing in FIG. 17A is performed by merging from block Block-0 to block Block-7, merging from block Block-8 to block Block-15, and so on from block Block-120 to block Block-127. Up to 16 merges, that is, the process of merging 8 blocks into one block is executed sequentially 16 times. For example, in the first stage, SPU-0 merges block 0 and block 1 to output blocks 0 to 1, and SPU-1 merges Block-2 and Block-3 to output blocks 2-3. . Here, the notations of blocks A to B such as blocks 0 to 1 represent blocks obtained as a result of merging from block A to block B. Next, in the second stage, SPU-4 merges blocks 0-1 merged by SPU-0 and blocks 2-3 merged by SPU-1, and outputs blocks 0-3. Similarly, SPU-4 outputs blocks 4-7. The third-stage SPU-6 merges the blocks 0-3 output by the SPU-4 with the blocks 4-7 output by the SPU-5, and outputs the blocks 0-7. By repeating such a three-stage merge process repeatedly, 16 blocks of blocks 0 to 7, blocks 8 to 15,..., And blocks 120 to 127 are output. In the figure, white arrows indicate input / output between the global memory and the local memory in the arithmetic unit, and black arrows indicate data transfer between the local memory in the arithmetic unit via the intra-chip bus. Represents.

  In the second process of FIG. 17B, among the 16 blocks output by the first process, 8 blocks of blocks 0 to 7,... 63 is output, and the blocks 64 to 71,..., And the blocks 120 to 127 are merged to output the blocks 64 to 127.

  Further, in the third process of FIG. 17C, SPU-0 merges blocks 0-63 and blocks 64-127, and outputs one final block 0-127.

  Next, the merge process in inter-block compilation according to an embodiment of the present invention will be described in more detail. 18A and 18B are explanatory diagrams of results of intra-block compilation processing to which inter-block compilation processing according to an embodiment of the present invention is applied. For example, FIG. 18A shows item information of eight blocks from block 0 to block 7 generated as a result of the in-block compilation processing for data item = “School”, and FIG. 18B shows data item = “ The item information of the block produced | generated regarding "Age" is shown. The item information includes a local item value number array VNo, a block number work array BlkNo, a local item value designation pointer work array LVL, and a local item value work array wVL. In this example, one block 0 to 7 is changed from 8 blocks from block 0 to block 7 shown in FIG. 17A using 7 arithmetic units from SPU-0 to SPU-6. This corresponds to the merge process to be generated.

  First, an example in which SPU-0 merges block 0 and block 1 will be described. 19A to 19C are explanatory diagrams of the first-stage merge process in the inter-block compilation process. SPU-0 transfers item information for block 0 and block 1 from global memory to local memory, and further block number work array BlkNo ′, further local item value designation pointer work array LVL ′, and further local item value work The array wVL ′ is initialized. A pointer for reading from the local item value work array wVL is also initialized. Hereinafter, unless otherwise specified, the operation subject is the arithmetic unit SPU-0.

  Next, the stored values of the local item value work array wVL from both blocks are compared, and the smaller stored value is transferred to the further local item value work array wVL 'in order from the top. The contents of the block number work array BlkNo corresponding to the smaller stored value are transferred to the further block number work array BlkNo '. For example, if the same value is continuously stored in the array LVL, the transfer from the block number work array BlkNo to the further block number work array BlkNo 'is repeated as many times as the same value is stored. Then, the same number of sequence numbers (initial value = 0) as the number of times the value has been written to the further block number work array BlkNo 'is stored in the further local item value designation pointer work array LVL'.

  Finally, the pointer for reading from the local item value number work array wVL in which the smaller value is stored is shifted backward by one.

  If the stored values of the local item value work array wVL from both blocks are the same, any stored value is transferred to the further local item value work array wVL '. Next, among the contents of the block number work array BlkNo from both blocks, the block number having the smaller value is transferred to the further block number work array BlkNo '. Also in this case, if the same value is continuously stored in the array LVL, the transfer from the block number work array BlkNo to the further block number work array BlkNo 'is repeated by that number. Subsequently, among the contents of the block number work array BlkNo from both blocks, the block number having the larger value is also transferred to the further block number work array BlkNo '. If the same value is continuously stored in the local item value designation pointer work array, the transfer of the block number is repeated as many times as the same value. Then, the same number of sequence numbers as the number of times the value is written to the further block number work array BlkNo 'is stored in the further local item value designation pointer work array LVL'. Finally, the pointer for reading from the local item value number work array wVL in which the smaller value is stored is shifted backward by one.

  Returning to FIGS. 19A to 19C, in FIG. 19A, wVL [0] = “South” in block 0 is compared with wVL [0] = “North” in block 1. Since “North” is smaller, the item value is transferred as wVL ′ [0] = wVL [0] = “North”. Then, the content BlkNo [0] = 1 of the block number work array of block 1 is transferred to the further block number work array BlkNo '[0], that is, BlkNo' [0] = 1. The sequence number = 0 is stored in LVL ′ [0]. Thereafter, the wVL read pointer on the block 1 side is shifted backward.

  Similarly, in FIG. 19B, wVL [0] of block 0 and wVL [1] of block 1 are compared. Both values are “South” and match. Therefore, wVL ′ [1] = “South”, BlkNo ′ [1] = 0, and BlkNo ′ [2] = 1. Further, LVL ′ [1] = 1 and LVL ′ [2] = 1 are set. Finally, pointers for reading from wVL on both the block 1 side and the block 2 side are advanced.

  Similarly, in FIG. 19C, wVL [1] of block 0 and wVL [2] of block 1 are compared. Since both values match, the same processing as in FIG. 19B is performed.

  In this way, as shown in FIG. 19B, SPU-0 merges block 0 and block 1 to further block number array BlkNo ′, further item value designation pointer array LVL ′, and further item value work. A set consisting of the array wVL ′ is generated.

  Through the above processing, it can be seen that a pair of BlkNo, LVL, and wVL from the two blocks of block 0 and block 1 has been converted into one set of BlkNo ′, LVL ′, and wVL ′. It is clear that by repeating this process in parallel and hierarchically, multiple sets of BlkNo, LVL and wVL can be merged into one set of BlkNo ', LVL' and wVL '. Here, it should be noted that the operations described with reference to FIGS. 19A to 19C can be realized using only sequential access. As a result, even if the block number array BlkNo, the local item value designation pointer work array LVL, and the local item value work array wVL 'increase in size, they can be processed in the local memory in the arithmetic unit. It should be noted that the further block number array BlkNo 'is always in ascending order as long as the values stored in the further local item value designation pointer work array LVL' are the same.

  This time, the second merging process in which the arithmetic unit SPU-4 merges the blocks 0 to 1 output from the SPU-0 and the blocks 2 to 3 output from the SPU-11 into one block will be described. . 20A to 20D are explanatory diagrams of the second-stage merge process in the inter-block compilation according to the embodiment of the present invention. The second-stage merge process is the same as the first-stage merge process, except that the input information is transferred from the local memory of another arithmetic unit. Briefly describing this processing, first, a pointer for reading wVL from two blocks is set at the head. After the value is read from wVL and compared, the smaller value is set to wVL '. Thereafter, the block number is transferred from BlkNo to BlkNo ′, and the sequence number is written to LVL ′. Finally, the pointer for reading from wVL ′ from which the smaller value is read is advanced. 20A, 20B, 20C and 20D show the process, the further block number array BlkNo ′ obtained, the further local item value designation pointer work array LVL ′, and the local item value work array wVL ′. Has been.

  When the first-stage merge process and the second-stage merge process of the inter-block compilation process are finished, blocks 0 to 3 in which blocks 0 to 3 are merged and block 4 to which blocks 4 to 7 are merged ~ 7 are obtained. The SPU-6 receives the blocks 0 to 3 output by the SPU-4 and the blocks 4 to 7 output by the SPU-5, and similarly executes the merge processing of the two blocks. As a result, Blocks 0 to 7 as one final block are obtained. 21A and 21B are diagrams for explaining the result of the third-stage merge process in the inter-block compilation process according to the embodiment of the present invention. FIG. 21A shows the information for each block before merging regarding the data item “School”, and FIG. 21B shows the result of the merging process by inter-block compilation regarding the data item “School”.

  It should be noted here that the final further local item value work array wVL 'matches the global item value array. The relationship between the block number work array BlkNo ′, the local item value designation pointer array LVL ′, and the local item value work array wVL ′ in FIG. 21B and the information for each block in FIG. 21A will be described. FIG. 21A shows, for example, the value list wVL of the item values included in the block 0 and the rank of the item value in the block 0 when the information of the block 0 is referred to. That is, in block 0, the item value of rank = 0 is “South”, and the item value of rank = 1 is “West”. On the other hand, referring to FIG. 21B, BlkNo ′ [0] = 2, LVL ′ [0] = 0, wVL [LVL [0]] = “East”, the item value = “ It can be seen that “East” is included. Furthermore, since BlkNo ′ [0] is the element in which block number = 2 appears first in the array BlkNo ′, the first item value in the value list in the block with block number = 2 is “East”. I understand that. When LVL ′ is scanned from the top, LVL ′ [0] = LVL ′ [1] = LVL ′ [2] = LVL ′ [3] = 0, and BlkNo ′ [0] = 2 and BlkNo ′ [1] = 4 and BlkNo ′ [2] = 5 and BlkNo ′ [3] = 7, that is, the first item value in the value list in the block 2, block 4, block 5 and block 7 is wVL It can also be seen that “0” = “East”.

  In the inter-block compilation process according to the embodiment of the present invention, the distribution process is executed after the merge process. In the distribution process, multiple arithmetic units operate in parallel, and for each data item, the element in the final local item value specification pointer work array is specified by the corresponding element in the final block number work array. The item values represented by the local item value numbers are stored in the final local item value work array (that is, the global item value array) by distributing each block number and arranging them in a predetermined order. Creates an item value specification pointer array that stores a pointer that specifies the position, and transfers it to the global memory. For example, in the example shown in FIG. 21A, the element number wVL [0] = “South” specified by LVL [0] = 0 in the block 0 is the number element of the global item value array wVL ′. An item value pointer array specifying is obtained.

Therefore, in the distribution in the inter-block compilation process according to the embodiment of the present invention, if the data shown in FIG.
for (i = 0; i <19; i ++) {
index [i] = 0;
}
for (i = 0; i <19; i ++) {
LVL [BlkNo ′ [i]] [index [i]] = LVL ′ [i];
index [i] ++;
}
The local item value designation pointer is distributed for each block number in accordance with the operation described by. The array LVL thus obtained is exactly the item value designation pointer array. The above distribution processing corresponds to an operation when the operation is performed by one processor. However, according to an embodiment of the present invention, the array LVL is preferably generated using a plurality of arithmetic units. Therefore, the block number work array BlkNo ′ and the local item value designation pointer work array LVL ′ are divided and assigned to a plurality of arithmetic units. Next, each arithmetic unit distributes a local item value designation pointer for each block number with respect to BlkNo ′ and LVL ′ in charge. Finally, the local item value designation pointers distributed and held in each arithmetic unit are integrated into one for each block number. Thereby, the item value designation pointer array LVL is obtained.

  FIG. 22 is an explanatory diagram of distribution processing in inter-block compilation processing according to an embodiment of the present invention. As shown in the figure, the array BlkNo 'and the array LVL' are assigned to a plurality of arithmetic units (in this example, eight arithmetic units SPU-0 to SPU-7). For example, the SPU-0 takes charge of the processing of BlkNo ′ [i] and LVL ′ [i] within the range of 0 ≦ i ≦ 2. Since BlkNo ′ [0] = 2 and LVL ′ [0] = 0, LVL ′ [0] = 0 is set in wLVL-2 [0] in the local memory of SPU-0. Similarly, SPU-1 is in charge of processing BlkNo ′ [i] and LVL ′ [i] within a range of 3 ≦ i ≦ 4. Since BlkNo ′ [3] = 7 and LVL ′ [3] = 0, LVL ′ [0] = 0 is set in wLVL-2 [0] in the local memory of SPU-1. All processing units are executed in parallel in this process. In this example, an array wLVL-0 to wLVL-7 from block 0 to block 7 is used for each arithmetic unit. In this case, a work area equal to the number of arithmetic units (= 8) is secured in the local memory and the global memory. In order to make this work area compact, a linked list may be used. The important thing is that as long as the work area can be stored in local memory, the work area is accommodated in local memory, and when it is found that the work area cannot be accommodated in local memory, the entire work area in local memory can be stored. Alternatively, memory access to the global memory can be unified by transferring a part of the data to the global memory.

  FIG. 23 is a diagram showing a result of distribution processing in inter-block compilation processing according to an embodiment of the present invention. For example, for block number 1, value 1 is stored in wLVL-1 of SPU-1, value 2 is stored in wLVL-1 of SPU-4, and value 3 is stored in wLVL-1 of SPU-6 . By combining these into one, an item value designation pointer array LVL-1 relating to block 1 is obtained. Specifically, the values stored in wLVL-1 may be extracted in ascending order of pointer values, that is, in the order of SPU-0 to SPU-7, and the values may be stored in order from the beginning of LVL-1. This operation can be executed by, for example, any one arithmetic unit or control unit, but for each block number, a plurality of arithmetic units extract values from WLVL in parallel and write them to LVL. It may be.

  22 and 23, as the number of blocks increases, the work area on the local memory also increases. Therefore, in an alternative embodiment of the present invention, particularly when the number of blocks is large, a distribution process is performed for each group after a plurality of blocks are grouped in order to improve the processing efficiency. For example, the block number is divided by 4 to separate the upper block number and the lower block number (grouping), and the distribution process is separately applied to the upper block number and the lower block number. Specifically, the block number work array BlkNo ′ for the upper block number and the local item are obtained from the set of the block number work array BlkNo ′ and the local item value designation pointer work array LVL ′ obtained by the merge process in the inter-block compilation process. A set of value designation pointer work array LVL ′ and a set of block number work array BlkNo ′ for the lower block number and local item value designation pointer work array LVL ′ are generated. This processing can also be executed by a plurality of arithmetic units operating in parallel.

  FIG. 24 is an explanatory diagram of block grouping processing in inter-block compilation processing according to an alternative embodiment of the present invention. For example, BlkNo ′ [0] to BlkNo ′ [2] belong to the range in charge of SPU-0. Since BlkNo ′ [0] = 2 is a value of 3 or less and is included in the lower block number, SPU-0 sets BlkNo [0] = BlkNo to BlkNo and LVL-1 for the lower block number in the local memory. Set ｀ [0] and LVL-0 [0] = LVL ′ [0]. On the other hand, the SPU-3 takes charge of BlkNo ′ [13] to BlkNo ′ [14]. Since BlkNo ′ [13] = 6 is a value of 4 or more and is included in the upper block number, SPU-3 assigns BlkNo [0] = BlkNo to BlkNo and LVL-1 for the upper block number in the local memory. Set [13] and LVL-1 [0] = LVL ′ [13]. By continuing such processing, blocks are grouped.

  FIG. 25 is a diagram illustrating a result of the block grouping process in the inter-block compilation process according to an alternative embodiment of the present invention. For example, if BlkNo and LVL-0 elements for lower block numbers created by SPU-0 to SPU-7 are extracted in order and stored sequentially from the beginning of BlkNo ′ and LVL ′, BlkNo ′ for lower block numbers and LVL ′ is obtained. Similarly, if BlkNo and LVL-1 elements for higher block numbers created by SPU-0 to SPU-7 are extracted in order and stored in order from the head of BlkNo ′ and LVL ′, BlkNo ′ for higher block numbers And LVL ′. This combining process may be executed by, for example, the control unit or may be executed by the arithmetic unit. The BlkNo ′ and LVL ′ for the lower block number and the BlkNo ′ and LVL ′ for the upper block number obtained in this way are the blocks according to the embodiment of the present invention described with reference to FIG. Distribution processing in the compilation processing is applied, and an item value designation pointer array is obtained for each block number.

  The present invention is not limited to the above embodiments, and various modifications can be made within the scope of the invention described in the claims, and these are also included in the scope of the present invention. Needless to say.

Claims (11)

Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
In a multi-core processing apparatus comprising:
It is represented as an array of records including item values corresponding to data items, and is a method of constructing tabular data that is shared and operated by the plurality of arithmetic units in the global memory,
A block number array in which the control unit divides the record into blocks including records in charge of each arithmetic unit and stores block numbers corresponding to the records in the order of the original record position numbers in the tabular data. Creating and storing in the global memory;
The plurality of arithmetic units operate in parallel to create a record sequence number array that stores the original record position numbers of the assigned records in the order of record sequence numbers on the local memory in each arithmetic unit, Transferring to the global memory;
Item value access information arrays for storing item value access information for accessing the item values included in the record in charge in the order of the record sequence numbers when the plurality of operation units operate in parallel. Creating on the local memory and transferring to the global memory;
The item values are stored in each operation unit for each data item so that the plurality of operation units operate in parallel and the item values included in the assigned record are accessed using the item value access information. Expanding on the local memory and transferring the expanded item value to the global memory;
A method comprising: The control unit refers to the block number array on the global memory to determine a block number of a block including a predetermined record and the arithmetic unit responsible for the predetermined record;
The control unit notifying the determined arithmetic unit of a record sequence number of the predetermined record;
The arithmetic unit notified of the record sequence number transfers the record sequence number array and the item value access information array related to the record in charge of the arithmetic unit from the global memory to the local memory of the arithmetic unit; ,
The arithmetic unit notified of the record sequence number specifies the position where the notified record sequence number is stored in the transferred record sequence number array; and
The arithmetic unit notified of the record sequence number specifies the item value access information specified by the specified position in the transferred item value access information array;
The arithmetic unit notified of the record sequence number acquires the item value specified by the specified item value access information from the global memory for each data item, and uses the acquired item value as the global value. Transferring to memory;
Further comprising
The method of claim 1. The step of operating the plurality of arithmetic units in parallel, expanding the item value for each data item in the local memory in each arithmetic unit, and storing the expanded item value in the global memory,
The plurality of arithmetic units operate in parallel to transfer, for each data item, the item values included in a single block from the global memory to the local memory, and items included in the single block. Items included in the assigned record in the order of the local field value work array for storing unique values among the values in a predetermined order, and the original record position number of the assigned record included in the single block Creating a local item value number array in the local memory for storing a local item value number specifying a position where a value is stored in the local item value work array, and transferring the local item value number array to the global memory;
The plurality of arithmetic units operate in parallel and store, for each data item, the block number corresponding to a unique value among the item values included in the block, related to a pair of blocks. Block number work array, local item value work array, and local item value designation pointer work array for storing a pointer for designating a position where the item value included in the block is stored in the local item value work array A pair consisting of a further block number work array, a further local item value work array, and a further local item value specification pointer work array associated with the block into which the pair of blocks is merged. Performing a merge process to create
The plurality of arithmetic units operate in a parallel and hierarchical manner, and the above merge processing is repeated until each data item is merged into one final block. And transferring the final local field value work array and the final local field value specification pointer work array to the global memory,
The plurality of arithmetic units operate in parallel, and for each data item, an element in the final local item value designation pointer work array is designated by a corresponding element in the final block number work array. By distributing each block number and arranging in a predetermined order, the item value represented by the local item value number matches the global item value array storing the item values in a predetermined order. Creating an item value specification pointer array for storing a pointer for specifying a position stored in a local item value work array and transferring the pointer to the global memory;
including,
The method of claim 1. Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
With
A multi-core processing device that is represented as an array of records including item values corresponding to data items and constructs tabular data that is shared and operated by the plurality of arithmetic units in the global memory,
A block number array in which the control unit divides the record into blocks including records in charge of each arithmetic unit and stores block numbers corresponding to the records in the order of the original record position numbers in the tabular data. Including means for creating and storing in the global memory;
Each arithmetic unit is
Operate in parallel with other arithmetic units, create a record sequence number array in the local memory in each arithmetic unit to store the original record position number of the record in charge in the order of the record sequence number, and Means for transferring to memory;
An item value access information array that operates in parallel with other arithmetic units and stores the item value access information for accessing the item values included in the record in charge in the order of the record sequence numbers. Means for creating on local memory and transferring to the global memory;
Operate in parallel with other arithmetic units, so that the item values included in the record in charge are accessed using the item value access information, the item values for each data item are Means for expanding on a local memory and transferring the expanded item value to the global memory;
including,
Multi-core processing equipment. The control unit is
Means for determining a block number of a block including a predetermined record and the arithmetic unit in charge of the predetermined record with reference to the block number array on the global memory;
Means for notifying the determined arithmetic unit of the record sequence number of the predetermined record;
Further including
The arithmetic unit notified of the record sequence number is
Means for transferring the record sequence number array and the item value access information array relating to the record in charge of the arithmetic unit from the global memory to the local memory of the arithmetic unit;
Means for identifying the position where the notified record sequence number is stored in the transferred record sequence number array;
Means for specifying item value access information specified by the specified position in the transferred item value access information array;
Means for acquiring the item value specified by the specified item value access information from the global memory for each data item, and transferring the acquired item value to the global memory;
Further including
The multi-core type processing apparatus according to claim 4. Each arithmetic unit is
Means for operating in parallel with other arithmetic units, expanding the item values for each data item in the local memory in each arithmetic unit, and storing the expanded item values in the global memory;
By operating in parallel with other arithmetic units, the item values included in a single block are transferred from the global memory to the local memory for each data item, and the item values included in the single block are transferred. The field values included in the assigned record are in the order of the local field value work array for storing the unique values of them in a predetermined order and the original record position number of the assigned record included in the single block. Means for creating a local item value number array for storing a local item value number for designating a position stored in the local item value work array on the local memory and transferring the local item value number array to the global memory;
A block number that operates in parallel with another arithmetic unit and stores the block number corresponding to a unique value among the item values included in the block, related to a pair of blocks, for each data item A work array, the local item value work array, and a local item value designation pointer work array for storing a pointer for designating a position where the item value included in the block is stored in the local item value work array. From this pair, a pair consisting of a further block number work array, a further local item value work array, and a further local item value specification pointer work array related to the block in which the pair of blocks is merged is created. Means for executing the merge process;
It operates in parallel and hierarchically with other arithmetic units, repeats the merging process until each data item is merged into one final block, and the final block number work array obtained, Means for transferring the final local field value work array and the final local field value specification pointer work array to the global memory;
A block that operates in parallel with another arithmetic unit and for each data item, the element in the final local item value designation pointer work array is designated by the corresponding element in the final block number work array. The final local, wherein the item values represented by the local item value numbers match the global item value array storing the item values in a predetermined order by distributing by number and arranging in a predetermined order Means for creating an item value designation pointer array for storing a pointer for designating a position stored in the item value work array and transferring it to the global memory;
Further including
The multi-core type processing apparatus according to claim 4. Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
A code that is loaded into a computer comprising: A computer readable program to be executed by the computer,
A block number array in which the control unit divides the record into blocks including records in charge of each arithmetic unit and stores block numbers corresponding to the records in the order of the original record position numbers in the tabular data. Code to create and store in the global memory,
The plurality of arithmetic units operate in parallel to create a record sequence number array that stores the original record position numbers of the assigned records in the order of record sequence numbers on the local memory in each arithmetic unit, A code to be transferred to the global memory,
Item value access information arrays for storing item value access information for accessing the item values included in the record in charge in the order of the record sequence numbers when the plurality of operation units operate in parallel. Code created on the local memory and transferred to the global memory,
The item values are stored in each operation unit for each data item so that the plurality of operation units operate in parallel and the item values included in the assigned record are accessed using the item value access information. A code that expands on the local memory and transfers the expanded item value to the global memory;
Including programs. The control unit refers to the block number array on the global memory and determines a block number of a block including a predetermined record and the arithmetic unit in charge of the predetermined record;
A code for the control unit to notify the determined arithmetic unit of a record sequence number of the predetermined record;
The arithmetic unit notified of the record sequence number has a code for transferring the record sequence number array and the item value access information array related to the record in charge of the arithmetic unit from the global memory to the local memory of the arithmetic unit. ,
The arithmetic unit that is notified of the record sequence number has a code that specifies the position where the notified record sequence number is stored in the transferred record sequence number array;
The calculation unit notified of the record sequence number specifies the item value access information specified by the specified position in the transferred item value access information array, and
The arithmetic unit notified of the record sequence number acquires the item value specified by the specified item value access information from the global memory for each data item, and uses the acquired item value as the global value. A code to transfer to the memory,
Further including
The program according to claim 7. The plurality of arithmetic units operate in parallel, expand the item value for each data item in the local memory in each arithmetic unit, and store the expanded item value in the global memory,
The plurality of arithmetic units operate in parallel to transfer, for each data item, the item values included in a single block from the global memory to the local memory, and items included in the single block. Items included in the assigned record in the order of the local field value work array for storing unique values among the values in a predetermined order, and the original record position number of the assigned record included in the single block A code for creating a local item value number array in the local memory for storing a local item value number for specifying a position where a value is stored in the local item value working array, and transferring the local item value number array to the global memory;
The plurality of arithmetic units operate in parallel and store, for each data item, the block number corresponding to a unique value among the item values included in the block, related to a pair of blocks. Block number work array, local item value work array, and local item value designation pointer work array for storing a pointer for designating a position where the item value included in the block is stored in the local item value work array A pair consisting of a further block number work array, a further local item value work array, and a further local item value specification pointer work array associated with the block into which the pair of blocks is merged. Code that performs the merge process to create
The plurality of arithmetic units operate in a parallel and hierarchical manner, and the above merge processing is repeated until each data item is merged into one final block. And a code for transferring the final local field value work array and the final local field value specification pointer work array to the global memory,
The plurality of arithmetic units operate in parallel, and for each data item, an element in the final local item value designation pointer work array is designated by a corresponding element in the final block number work array. By distributing each block number and arranging in a predetermined order, the item value represented by the local item value number matches the global item value array storing the item values in a predetermined order. A code for creating an item value specification pointer array for storing a pointer for specifying a position stored in a local field value work array and transferring it to the global memory;
including,
The program according to claim 7. Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
A tabular data that is loaded into a computer and is represented as an array of records including item values corresponding to data items and is operated in a shared manner by the plurality of arithmetic units is constructed in the global memory. A computer program product for causing the computer to execute the method according to any one of items 1 to 3. Multiple arithmetic units including dedicated local memory,
A global memory connected to the plurality of arithmetic units, and
A bus connecting the plurality of arithmetic units;
At least one control unit connected to the global memory and the plurality of arithmetic units;
A tabular data that is loaded into a computer and is represented as an array of records including item values corresponding to data items and is operated in a shared manner by the plurality of arithmetic units is constructed in the global memory. A storage medium storing a computer program for causing the computer to execute the method according to any one of items 3 to 3.

Images