JPH05101111A - Data processor - Google Patents

Abstract

PURPOSE:To determine an optimum processing system for the data base arithme tic processor by predicting its object data amount and calculating necessary memory capacity when a host device generates a data base calculating com mand. CONSTITUTION:When the host device 10 sends the command such as a file input command to the data base arithmetic processor (DBE), the amount of data is calculated from the file size of an object file, etc., and stored in a result data amount storage part corresponding to said command and when an object process executes an existent command, the data amount of the processing result is stored in the result data amount storage part corresponding to the command. Further, when the host device 10 sends a command where the object processing is present to the DBE16, the necessary memory capacity is calculated from the result data amount stored in the result data amount storage part corresponding to the command used to obtain the object data.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data processing device equipped with a database operation processing device for executing a specified operation process in response to a command given from a host device, and particularly to instructing the database operation process from the host device. When a command to execute is issued, the present invention relates to a data processing device in which a database arithmetic processing device has a prediction function of predicting the target data amount in advance.

[0002]

2. Description of the Related Art A data processing device having a database operation processing device which receives a database operation command given from a host device and executes a specified operation process has been conventionally known. In order to efficiently execute a database operation command with a database operation processing device of this type of data processing device, in order to efficiently execute a database operation command (for example, in the case of semi-join or join processing, the target table and condition table It is necessary to secure a memory area corresponding to the data amount) in the database arithmetic processing unit. In the case of the above semi-join or join processing, this memory area is an input buffer area for temporarily storing the target table and the condition table, and an output for temporarily storing the result data of the semi-join or join processing. This is the buffer area.

In the past, however, it was not possible to know in advance the amount of target data before starting the processing of the database operation command, and therefore it was not possible to predict the memory capacity required for the processing. For this reason, conventionally, either the following first or second method has been applied to the database arithmetic processing in the database arithmetic processing device.

First, on the premise that the required memory capacity cannot be ensured in the database arithmetic processing unit, a system that can always perform processing regardless of the actually required memory capacity (specifically, in the database arithmetic processing unit) Method of using the work file of the secondary storage device instead of the above memory)
Is a method of applying.

The second method is a method that obtains the highest performance (specifically, a method that uses the memory in the database arithmetic processing unit, assuming that the memory capacity required in the database arithmetic processing unit can be secured). It is a method to try from the above, and if it fails, try the next method.

[0006]

As described above, in the conventional data processing device equipped with the database operation processing device, it is not possible to know the target data amount in advance before starting the processing of the database operation command. Since it was not possible to predict the required memory capacity, a method that can be processed in all cases (first method)
Or a method (second method) of trying from the method that gives the best performance and then trying the next method if it fails.

However, the former method (first method) is a general-purpose method, and optimization is not performed when the data amount is small, so there is a problem that the performance is considerably deteriorated when the data amount is small. .. On the other hand, the latter (second method)
Then, if the amount of data is small, optimization can be achieved,
Processing cannot be performed with a large amount of data, and the method will have to be redone. Therefore, in such a case, there is a problem in that the process from the start of the process to the determination that the process is impossible is wasted, and the performance is considerably deteriorated.

The present invention has been made in view of the above circumstances, and an object thereof is to predict in advance a target data amount in a database arithmetic processing device when a command for instructing database arithmetic processing is issued from a host device. Therefore, it is an object of the present invention to provide a data processing device capable of calculating a required memory capacity and determining an optimal database arithmetic processing method in advance based on the calculated memory capacity.

[0009]

SUMMARY OF THE INVENTION The present invention provides a data processing device equipped with a database operation processing device for receiving a database operation command given from a host device and executing a specified operation process. When a command for inputting a file to the device is issued, the host device is provided with means for adding data amount information that directly or indirectly specifies the data amount of the target file, and the data amount of the database operation processing result. Etc., a storage unit used for storing the command, a determination unit that analyzes the command issued from the host device to determine whether the command has a target process, and a command that does not have the target process. Is directly or indirectly specified by the data amount information added to the command. It corresponds to the means for storing the data amount in the storage means as the result data amount corresponding to the command and the preceding command used to obtain the processing target data of the command in the case where the target processing exists. Based on the amount of result data stored in the storage means, the calculation means for calculating the memory capacity to be secured in the database processing device for the same calculation processing, and the optimum according to the calculated required memory capacity The host device is provided with respective means of determining means for determining the database operation processing method and means for storing the data amount of the database operation processing result processed by the determined method in the storage means corresponding to the corresponding command. It is characterized by that.

[0010]

In the above configuration, in the case of a command for which the target process does not exist (for example, a READ command for instructing to read the table data on the secondary storage device to the database arithmetic processing device), it is added to the command. The data amount directly or indirectly designated by the given data amount information is stored in the storage means as the result data amount corresponding to the command. In addition, a command in which the target process exists (for example, a semi-join process targeting the table data obtained by executing the preceding command is instructed).
When a RESTRICT command or the like) is executed, the data amount of the execution result is stored in the storage means corresponding to the command.

Then, a command in which the target process exists, that is, a command (for example, a JOIN command for instructing a join) in which the table data obtained by the execution of the preceding READ command, RESTRICT command, etc. is the target data of the database operation process is When given, the database arithmetic processing unit for the instructed arithmetic processing based on the result data amount stored in the storage means corresponding to the preceding command used to obtain the target data. The memory capacity to be secured is calculated, and the optimum database arithmetic processing method is determined in advance according to the calculated required memory capacity. As a result, the instructed process is efficiently executed in a method suitable for the amount of data. The data amount of the execution result is stored in the storage unit corresponding to the corresponding command (here, JOIN command) so that it can be used in the processing of the subsequent command.

[0012]

Embodiments of the present invention will be described below with reference to the drawings.

FIG. 1 shows the overall system configuration of a data processing apparatus according to an embodiment of the present invention. This data processing device includes a host computer 10 and a database arithmetic processing device (database engine; DBE).
16 and a disk device 17 such as a magnetic disk device. The host computer 10 is C
The CPU 11 includes a PU 11, a main memory device 12, and first and second channel devices 14 and 15.
1, the main storage device 12 and the channel devices 14 and 15 are interconnected via a system bus 13.

The host computer 10 and the database arithmetic processing unit (DBE) 16 are connected by the channel device 14, and the host computer 10 and the disk device 17 are connected by the channel device 15. Further, the database operation processing device (DBE) 16 and the disk device 17 are connected via a dedicated path 18.

The CPU 11 controls the host computer 10 as a whole, and requests the database arithmetic processing unit (DBE) 16 to execute various arithmetic processing such as sorting and relational algebraic arithmetic. In the main storage device 12,
A command group for instructing a calculation target file and calculation contents to the database calculation processing device (DBE) 16 is stored.

Database processing unit (DBE) 16
Is for executing arithmetic processing on the data of the calculation target file based on a command from the CPU 11,
To input the calculation target file and output the calculation result,
The disk device 17 is directly accessed via the path 18.

This DBE (database arithmetic processing unit)
16 is an engine interface processor (EI
P) 161, engine control processor (ECP) 16
2. A large capacity memory (EBDM) 163, a hardware (HW) sorter control processor (ECAM) 164, a parallel sorting module (PSOM) 165, and a parallel relational algebra operation module (PRAM) 166.

EIP (engine interface processor) 161, ECP (engine control processor) 16
2 and ECAM (hardware sorter control processor)
The three processors 164 are interconnected by an internal bus 167, and an EBDM (large capacity memory) 163.
Tightly coupled multi-processor with shared memory is configured. The EBDM 163 is arranged in a common address space of each of the three processors. Further, the functions are distributed to these three processors, that is, the EIP 161, the ECP 162, and the ECAM 164. In this case, in order to efficiently carry out each unique role,
Each of these processors is configured to be tightly coupled and controlled by an independent monitor suitable for each processor. EIP161, ECP162 and ECAM164
The functions are distributed as follows.

That is, the EIP (engine control processor) 161 communicates between the host computer 10 and the DBE 16 and is also connected to the disk controller of the disk device 17 via the path 18 to exchange data with the disk device 17. Control I / O. Also, EI
When P161 outputs data to the disk device 17,
It also reconstructs the output file.

In the communication with the host computer 10, the EIP 161 receives the command sent from the CPU 11 via the first channel device 14 and sends it to the ECP 162. Also, the EIP 161 is
The status as the command result sent from the ECP 162 is received and returned to the CPU 11 via the first channel device 14.

In the data input / output processing with the disk device 17, the EIP 161 receives the input / output request from the ECP 162 and directly transfers the data between the EBDM 163 and the disk device 17. The ECP 162 is
The EIP 161, the EBDM 163, and the ECAM 164 are controlled via the internal bus 167 to perform data processing.

The EBDM (large-capacity memory) 163 is a file data (table data) to be operated, which is read from the disk device 17, ECAM 164, PSOM 16
5 and the PRAM 166 is a shared memory for storing sort processing results, relational algebraic operation results, and the like. This EBDM
The sort processing result, the relational algebraic calculation result, and the like (output results thereof) stored in 163 are basically represented by a pointer (record identification number RID) indicating the start address of the input record on the EBDM 163. this is,
This is to reduce the memory capacity required to store the output result. The ECAM (hardware sorter control processor) 164, based on a command from the ECP 162,
The operation by the PSOM 165 and the PRAM 166 is controlled.

In this case, the ECAM 164 is the EBDM1
The data on 63 is input to PSOM165, and PRAM1
The calculation result output from 66 is stored in the EBDM 163, but when data is input to the PSOM 165, key cutout processing is executed. In this key cut-out processing, the ECAM 164 cuts out only the key necessary for the calculation from each record to be calculated, adds the record identification number RID (a pointer indicating the start address of the record on the EBDM 163) to it, and sends it to the PSOM 165. ..

PSOM (parallel sorting module)
165 is a dedicated hardware circuit that is driven by the ECAM 164 and executes sorting in parallel.
It is connected to M166. The PSOM 165 is called a pipeline merge sorter, and is composed of a plurality of sort cells cascade-connected to perform 2-way merge.

PRAM (parallel relational algebra operation module)
166 is JOIN (join) in relational database
This is a dedicated hardware circuit that executes relational algebraic operations such as RESTRICT (constraints, semi-joins) in parallel.
Input the sorted data from 65 and EC the calculation result
Output to AM164. When only the sorting process is executed, the PRAM 166 functions as the final sort cell.

FIG. 2 shows the CPU of the host computer 10 (
11 shows a command sequence 21 given to DBE (database arithmetic processing unit) 16 from 11). This command sequence 21
Is composed of a command group (commands # 1, # 2 ... #n) representing a series of processing contents to cause the DBE 16 to perform processing from the host computer 10 and the number of commands in the command sequence. Each command can be identified by a command identifier (command ID).

As the command, READ for instructing the reading of the table from the disk device 17 to the EBDM 163.
Command, PROJECTION command for instructing to output the data on the EBDM 163 to the disk device 17 or the EBDM 163 in the specified format, and RESTRI
CT command is prepared. This RESTRICT command is a command to instruct to perform constraint (semi-join) with columns of different tables as conditions, that is, to compare multiple columns of the target table and condition table and select records in the corresponding target table. is there.

As a command, SELECT for instructing selection by columns or constants of the same table
After selecting records according to the condition for the given column for each of the command, target table and condition table, compare the selected record group of the target table and multiple columns of the selected record group of the condition table SELSEL JOIN command (a type of JOIN command that performs a join by the specified key column) for instructing to select the record in the target table to be processed, and WORKFILE for specifying the work file on the disk device 17
Commands are also prepared.

Each command constituting the command sequence 21 is
As shown in the figure, it comprises a command code and various information necessary for processing in the DBE 16. For example, in the case of a READ command, the various information includes file data (table data) to be read from the disk device 17 into the DBE 16 and its size. Further, for example, in the case of a RESTRICT command, a table to be selected (the ID of the command used to obtain it), a condition table (the ID of the command used to obtain it), and the target table These are columns, columns in the condition table, and comparison operators.
If it is a SELSELJOIN command, the table to be combined (the ID of the command used to obtain it),
A condition table (the ID of the command used to obtain it), a selection condition for the target table, a selection condition for the condition table, a column of the target table, a column of the condition table, an operator for joining, and the like.

FIG. 3 shows the structure of the result data amount storage unit 168 secured on the memory (not shown) in the ECP 162. The result data amount storage unit 168 is used to store the data amount of the data obtained by executing the command given from the host computer 10 as the result data amount, corresponding to the ID of the command.

Next, the operation of the embodiment of the present invention will be described. First, the operation of the entire data processing apparatus of FIG. 1 will be described. The CPU 11 in the host computer 10 issues a command string 21 (see FIG. 2) on the main storage device 12 to the DBE (database operation processing device) 16 via the system bus 13 and the channel device 14.

Command sequence 21 issued to the DBE 16
Is input into DBE16 by EIP161,
It is stored in the EBDM 163. ECP162 is EBD
The commands are analyzed in order from the first command of the command sequence 21 (from the host computer 10) stored in M163, and if necessary, the data to be processed is E
From the disk device 17 to the EBDM 16 via the IP 161
No. 3 is read and passed to ECAM 164 for processing. The ECAM 164 sends the target data to the PSO
The specified calculation is passed from the M165 to the PRAM 166, and the specified calculation is performed, and when the processing is completed, the ECP 162 is notified. The EIP 161 outputs the processing result to the disk device 17 or the main storage device 12 or the like if necessary, and returns a response corresponding to the executed command to the CPU 11 via the channel device 14 and the system bus 13.

Next, details of the operation in the DBE 16 are shown in FIG.
This will be described with reference to the flowchart in FIG. First ECP16
It is assumed that 2 retrieves one command from the command sequence 21 stored in the EBDM 163 and analyzes the command. As a result of this analysis, the process is divided into two types depending on the command format (type) (step S1).

One is a case where the command is a command for which a target process does not exist, that is, a READ command or the like (process such as input). In this case, the process of step S2 is performed.

At this step S2, the ECP 162 is
Is the maximum value (upper limit) of the data amount (the number of records) of the target file (table data) on the disk device 17 whose input is specified by the command, and the size of the file (file Size (estimate), etc., and the calculated data amount is used as the result data amount, and the EC corresponding to the ID of the command is calculated.
It is registered in the result data amount storage unit 168 in P162.

On the other hand, if the command is a command in which the target process exists, that is, a command such as a RESTRICT command or a SELSEL JOIN command for performing the database calculation process on the result of another preceding command, step S
Process 3 is performed.

At this step S3, the ECP 162 is
Refers to the result data amount storage unit 168 by the ID of the preceding command (used to obtain the target data) specified by the command, and is registered in the result data amount storage unit 168 corresponding to the ID. The resulting data volume,
That is, the data amount of the processing result of this preceding command (the amount of data obtained by calculation of the target file if the preceding command is a READ command, and if the preceding command is a RESTRICT command or SELSELJOIN command, etc. (Result data amount).

Subsequently, the ECP 162 uses the result data amount taken out to perform EBDM for the target process of the command.
A necessary memory capacity to be secured on the 163 is calculated. That is, the ECP 162 takes, for example, a RESTRICT command instructing a semi-join for the result (generated table) of the join processing by the instruction of the SELSELJOIN command and the file (table) specified by the READ command, as an example. The result data amount registered in the result data amount storage unit 168 corresponding to the ID of the SELSELJOIN command (the data amount of the table generated as a result of the combining process instructed by the SELSELJOIN command) and the above
Result data amount storage unit 16 corresponding to the ID of the READ command
Based on the result data amount registered in 8 (data amount of the table specified by the READ command, calculated based on information such as the size of the file attached to the command) , RESTRICT for both these tables
The maximum value of the data amount of the result of (semi-joining) is calculated (predicted). The maximum value is obtained here because an error occurs only in the data that is excluded from the processing target and further in the data that is dropped by the processing.

Then, the ECP 162 calculates the memory capacity of the buffer area (input buffer) for inputting both tables based on the data amounts of the both tables, and also calculates the data amount of the calculated result. A buffer area for outputting the processing result (output buffer area)
Calculate the memory capacity of. From the above, the total memory capacity required to process the command (here, the RESTRICT command) (only in the DBE 16) is calculated.

When the ECP 162 calculates the required memory capacity, the EB has enough margin to secure the area of the memory capacity.
Whether the data is in the DM 163, that is, the total amount of the data obtained by the above is all E which is the internal memory of the DBE 16.
It is checked whether or not it can be arranged in the empty area of the BDM 163 (step S4). As a result, if the EBDM 163 is out of space, the ECP 162 determines that it is impossible to continue processing only within the DBE 16, and the DBE 16 such as the disk device 17 or the main memory 12 is determined.
The work file on the external storage viewed from above is used (as a buffer), and processing is performed using the ECAM 164 or the like (step 8). Such processing is called extension processing.

On the other hand, when the EBDM 163 has no area shortage and can be processed only within the DBE 16, the ECP
162 is a PSOM 165 and a PRAM, which are dedicated hardware for the processing of the command concerned, for relational algebraic calculation.
Using 166, it is checked whether batch processing is possible (step S5). Depending on whether or not this collective processing is possible, the processing branches into two, step S6 or step S7. That is, PS
The hardware of the OM 165 and the PRAM 166 has an upper limit of the amount of data that can be processed at one time. Depending on whether or not the predicted amount of data obtained above exceeds the amount of data that can be collectively processed, the processing content is different.

First, if batch processing is possible, ECP1
62 requests the ECAM 164 to perform batch processing using the hardware of the PSOM 165 and the PRAM 166 (step S6). Such processing is called internal processing. On the other hand, if batch processing is not possible, ECP1
62 requests ECAM 164 for PSOM 165 and P
The hardware processing of the RAM 166 is repeatedly performed, and the results are combined and integrated (step S7). That is, if batch processing is not possible, P
After the hardware of the SOM 165 and the PRAM 166 has been used many times, post-processing for combining and integrating them (such processing is called external processing) is required.

When the above step S6 (internal processing), step S7 (external processing) or step S8 (extended processing) is completed, the ECP 162 sets the correct data amount of the processing result obtained at that step to the command (here, RESTRI
Result data amount storage unit 16 corresponding to the ID of CT command)
8 is registered (step S9). This makes it possible to use (reference) the information on the data amount of the processing result of the command in the processing of the subsequent command. Here, regarding the processing speed of steps S6 to S8, the internal processing of step S6 is the fastest, the external processing of step S7 is the second fastest, and the expansion processing of step S8 is the slowest.

In the above embodiment, the READ command and the like have been described as being provided with information indicating the size of the target file, etc. (Maximum value) may be attached. This eliminates the need to calculate the data amount of the target file in the ECP 162 when the target command is a READ command or the like.

Further, the contents of the result data amount storage unit 168 may be cleared after the execution of the command sequence 21 is completed. In this case, the area of the result data amount storage unit 168 can be minimized.

[0046]

As described in detail above, according to the present invention,
Data amount information that directly or indirectly specifies the data amount of the target file is added to the command for inputting a file that does not have the target process on the host device side, and on the database arithmetic processing unit side that receives the command , The data amount directly or indirectly designated by the data amount information added to the command is stored in the storage means as the result data amount in correspondence with the command, and the execution result of the command in which the target process exists is stored. The data amount is stored in the storage means corresponding to the command, and the command in which the target process exists, that is, the command in which the table data obtained by the execution of the preceding command is the target data of the database operation process is the host. It is used to obtain the target data when it is given from the device to the database processor. Based on the result data amount stored in the storage means corresponding to the command, the memory capacity to be secured in the database arithmetic processing device for the instructed arithmetic processing is calculated. The optimal database operation processing method can be determined in advance according to the required memory capacity, and the performance of the database operation processing can be improved.

[Brief description of drawings]

FIG. 1 is a block diagram showing a system configuration of a data processing device according to an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a command sequence given from a host computer 10 in FIG. 1 to a database arithmetic processing unit (DBE) 16.

3 is an engine control processor (ECP) 16 of FIG.
2 is a diagram showing a structure of a result data amount storage section secured on a memory in FIG.

FIG. 4 is a database arithmetic processing unit (DBE) 1 of FIG.
6 is a flowchart for explaining the operation of No. 6.

[Explanation of Codes] 10 ... Host computer, 11 ... CPU, 16 ... Database operation processing device (DBE), 17 ... Disk device (secondary storage device), 18 ... Path, 21 ... Command string, 1
61 ... Engine Interface Processor (EIP),
162 ... Engine control processor (ECP), 163 ...
Large capacity memory (EBDM), 164 ... Hardware (H
W) Sorter control processor (ECAM), 165 ... Parallel sorting module (PSOM), 166 ... Parallel relational algebra operation module (PRAM), 168 ... Result data amount storage unit.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Asahiko Yamada 2-9 Suehiro-cho, Ome-shi, Tokyo Incorporated Toshiba Ome Plant (72) Inventor Takao Iwasaki 2-9 Suehiro-cho, Ome-shi, Tokyo Shareholders Inside the Toshiba Ome Plant (72) Inventor Kazunori Shimakawa 2-9 Suehiro-cho, Ome-shi, Tokyo Stock Company Inside the Toshiba Ome Plant (72) Inventor Yuji Sato 1385 Shinmachi, Ome-shi, Tokyo Inside Toshiba Software Engineering Co., Ltd. (72) Inventor Hiroki Soto 1385 Shinmachi, Ome-shi, Tokyo Within Toshiba Soft Engineering Co., Ltd. (72) Inventor Shinichi Amano 1385 Shinmachi, Ome-shi, Tokyo Inside Toshiba Soft Engineering Co., Ltd.

Claims (1)

[Claims] 1. A host device, a secondary storage device for storing file data to be operated, and a path for directly accessing the secondary storage device, wherein the operation is performed in response to a command from the host device. In a data processing device including an arithmetic processing device that executes a predetermined arithmetic process on target file data, a command for inputting a file or the like is issued from the host device to the database arithmetic processing device to the host device. In this case, it is provided with means for adding data amount information that directly or indirectly specifies the data amount of the target file, and is used to store the data amount of the database operation processing result in the database operation processing device. And a storage unit configured to analyze a command issued from the host device to the database arithmetic processing device. When it is determined that the command does not include the target process by the determining unit that determines whether the command has the target process, the data amount information added to the command A means for storing the data amount designated directly or indirectly as the result data amount in the storage means corresponding to the command; and when the discrimination means discriminates that the target process exists. , Secured in the database arithmetic processing unit for the arithmetic processing based on the result data amount stored in the storage means corresponding to the preceding command used to obtain the processing target data of the command Calculation means for calculating the memory capacity to be used, and optimum database calculation processing according to the required memory capacity calculated by this calculation means The database calculation processing is provided with a determining means for determining an expression and a means for storing the data amount of the database operation processing result processed by the method determined by the determining means in the storage means in association with a corresponding command. A data processing device, characterized in that a target data amount is predicted in advance to determine an optimum database operation processing method.