JPH02268368A - Data processor - Google Patents

Abstract

PURPOSE:To efficiently execute the arithmetic processing of high level by generating partial elements of a result file in the record order indicated by identifiers and storing them in an external storage. CONSTITUTION:When a processing object and operation are indicated to a CPU 1, the CPU 1 writes a file of the processing object in a first area constituting an input buffer I-BUF of a main storage 4. A control part 53 in a relation arithmetic processing device 5 extracts a key K1 from each record of file data stored in the first area and adds an identifier R1 to generate input data and writes it an a buffer memory 52. When this processing is set to the mode where the file is taken in with a partial element as the unit because the file cannot be stored in the buffer, the next partial element is taken into the buffer I-BUF at the time of terminating key extraction of each partial element stored in the buffer I-BUF. Thus,the output file according to the result identifier array is generated to quickly execute the data processing without requiring the arithmetic processing.

Description

[Detailed Description of the Invention] [Objective of the Invention] (Industrial Application Field) The present invention is capable of performing complex arithmetic processing on a database that includes sorting and relational operations in a relational database. The present invention relates to a data processing device.

(Prior Art) Conventionally, there is a "selection device" disclosed in Japanese Patent Application No. 62-24751 as a data processing mechanism for databases. This "selection device" is placed in the input/output channel or disk control device, and directly receives data read from a magnetic disk device used as a large-capacity data file, and selects data that satisfies conditions. It realizes an efficient data retrieval mechanism using on-the-fly processing.

However, the above-mentioned device still has the following problems. That is, in the above device configuration, when only a simple selection process is targeted, even if the target files are distributed across multiple magnetic disk devices,
A desired selection result can be obtained by adding up the information on the selection results of each magnetic disk device. However, when the target is complex operations such as sorting or relational operations in a relational database (hereinafter referred to as RDB), processing that simply adds up the information of the results of multiple magnetic disk devices as described above is required. In this case, it is not possible to obtain the desired result information, and it is necessary to perform calculation processing again on each target data.

Furthermore, in the conventional device configuration described above, the magnetic disk device is directly connected to the selection device, and a means is adopted in which data read from the magnetic disk device is directly stored in a buffer within the selection device. Therefore, the extraction processing speed of some keys and the disk read processing speed interact with each other, and due to this, there are restrictions on file format, number of extraction keys (number of keys for multi-key), etc. There was also the problem that processing performance, processing speed, etc. were restricted.

In addition, in a data processing device that has dedicated hardware such as a sorter on the system bus, all files to be processed are stored in the main memory, and information resulting from processing by the dedicated hardware is stored in the main memory. After storing the information in the memory, it was stored in an external memory under the control of the CPU. At this time, if the file to be processed becomes too large to fit in the main memory, the file is divided into a predetermined amount of data that can fit in the main memory, is imported into the main memory, and after performing calculations on each divided partial data, By performing calculations on the calculation result data for each partial data, an output file of the final result can be obtained. However, in such data processing means, as the amount of data to be handled increases, the number of accesses to external storage such as a disk and the number of calculations increase, resulting in a significant reduction in processing speed.

(Problems to be Solved by the Invention) As mentioned above, in the conventional device configuration, when only a simple selection process is targeted, even if the target files are distributed across multiple magnetic disk devices, each The desired selection result can be obtained by summing up the information on the selection results of the magnetic disk drives, but when the target is complex calculations such as sorting or relational calculations in RDB, information obtained from each magnetic disk drive can be obtained. Since it is necessary to perform arithmetic processing on each piece of data again, the processing becomes complicated and the processing performance decreases accordingly, making it impossible to put it into practical use.

In addition, in a data processing device equipped with dedicated hardware such as a sorter for databases on the system bus, when a file to be processed does not fit in the main memory, the file is divided into multiple processing units, If you perform an operation on a unit of partial data and then perform an operation on the operation result data of each partial data to obtain the final result output file, the number of accesses to external storage and the number of operations will be reduced. There was a problem in that the processing speed was significantly reduced.

The present invention was developed in view of the above circumstances, and is capable of efficiently executing advanced data processing for complex operations such as sorting or relational operations in RDB with simple control. The purpose of the present invention is to provide a database processing mechanism that can efficiently perform the above-mentioned high-level arithmetic processing when the amount of data to be processed exceeds the main memory capacity, etc. shall be.

[Structure of the Invention] (Means and Effects for Solving the Problems) When the amount of data of a file to be processed exceeds the main memory capacity, the present invention extracts a key and extracts the same key for each partial element of the file to be processed. A process of adding an identifier is performed, and arithmetic processes such as sorting and merging are performed on the data string that is bare of the identifiers and keys of all the files to be processed obtained by repeating this process. As a result, from the obtained identifier string and the logical information and physical information of the target file, multiple data blocks containing records indicated by the above identifier string are read into the main memory, and from this data, the records indicated by the above identifier string are Create partial elements of the result file and output them to external storage. By performing this process on all the identifiers in the above result identifier string, even if the amount of data in the target file exceeds the main memory capacity, sorting or R
Processing such as relational calculations in the DB can be executed quickly and efficiently.

(Example) The present invention will be described in detail below with reference to the drawings.

First example FIG. 1 is a block diagram showing a first embodiment of the present invention.

In Figure 1, ■ is the CPU that controls the entire system.
Here, input/output channels 2, 2 . . . are connected to a system bus (S-Bus) 6. . . . When the relational calculation processing device 5 and the like are placed under control and data processing is performed for sorting or relational calculations in RDB, etc.,
Input/output channels 2, 2. The magnetic disk devices 3, 3 . . . . . . has a control function of reading the target file and writing it into the first area which becomes the input buffer (1-BUF) on the main memory 4. At this time, if the file to be processed exceeds the size of the human buffer (1-BUF), the file to be processed is divided into a plurality of partial elements whose unit is the input buffer size in the main memory 4, and -BUF), and every time the key extraction process described below is completed, the next remaining partial element is read. Also, CPUI
is a control function that sends information regarding the file to be processed (file information including file format, block length, record length, etc.) and calculation instructions for the file to the relational processing unit 5, and The result identifier string (R1゜R
j,...) and the logical information and physical information of the target file above, calculate the in-disk position information string of the operation result record (
DAi, DAj, . . .), and a processing means for obtaining the same disc position information sequence (DAi, DAj, . . .);
) and a magnetic disk device 3.3゜ that follows the same address.
. . . processing means for extracting record data having the above-mentioned in-disk position information as the leading position from the read data, and storing it in a predetermined third area of the main memory 4 in the order of the above-mentioned position information string; When the record string of the calculation results stored in the area 3 reaches a predetermined amount of data, procedures are performed to store it in the output file on the predetermined magnetic disk device 3 and to the input/output channel 2 for storing it in the area. and a processing means.

2 is an input/output channel (l) connected to the system bus 6.
1O-C1(), which controls access to the input/output device numbers under the control target under the control of the CPUI.Here, multiple magnetic disk devices 3, 3... are under the access control target. put.

3.3. ... are magnetic disk devices (DISK) placed under the control of the input/output channel 2, and data groups of various file structures constituting the database are stored. In the figure, A and B indicate files to be processed (input files), and R indicates an output file stored in the output file storage area.

4 is a main memory (HEM) connected to the system bus 8, which includes a first area serving as an input buffer (1-BUF) for storing a file to be processed or a partial element thereof; Information about the file to be processed stored in the area (file information including file format, block length, record length, etc.), information such as calculation instructions, calculation results, result identifier string according to the calculation result of the above file to be processed, etc. A second area for storing an in-disk position information sequence according to a resultant identifier sequence, and a third area for storing partial output data consisting of a record sequence as a result of an operation are provided.

5 is a relational processing unit (
ROPU) and is stored in the first area that becomes the input buffer (I-BUF) on the main memory 4 according to the information and calculation instructions regarding the file to be processed stored in the second area of the main memory 4. Extract the key (key field) Kl to be processed from the record of the file to be processed (or its partial element), and set the serial number (record number) within the file of the record with that key to 1 as the identifier R1 in the above key. a processing means for adding data, a key consisting of a pair of 1 and an identifier R1, and means for storing data regarding all target records of a file to be processed in an internal buffer (buffer memory 52) as input data (old); An arithmetic processing function that executes an arithmetic operation according to the above arithmetic instruction on the key value of the input data (old) consisting of a pair of key 1 and identifier R1 stored in this buffer (buffer memory 52), and obtains the result of the arithmetic operation. The identifier Ri is extracted in order from the data string, and the identifier string (result identifier string) is
It has a processing function of storing . buffer memory 52,
It consists of components such as a control section 53. Among these components, the control unit 53 controls the processing target file stored in the first area of the main memory 4 or Key K to be processed from each record of the subelement
l is extracted, and the serial number (record number) of the record with that key in the file is added to the above key as an identifier Rt, and 1 and the identifier 1? are added to a part of the key. Input data (DI) consisting of a set of I is generated, and manual data (DI) of all target records of the processing target file are configured on the buffer memory 52. Further, output data (DO) consisting of a result identifier string received from the arithmetic unit 51 is stored in a second area on the main memory 4. The calculation unit 51 has a buffer memory 52
The calculation according to the above calculation instruction is performed on the key value of the input data (DI) read from the input data (DI). Further, the identifier R1 is sequentially extracted from the data string on the buffer memory 52 obtained as a result of the calculation, and sent to the control unit 53 as output data (Do). This output data (Do) is
3 and is stored in the second area on the main memory 4.

The operation of the first embodiment of the present invention will now be described with reference to FIG. 1 above.

When the CPU receives processing targets and calculation instructions such as sorting (ascending/descending) or relational calculations from an input mechanism on the terminal side (not shown) via the system bus 6, the CPU receives the processing targets via the system bus 6 and the input/output channel 2. A file is read from the magnetic disk devices 3, 3°, . . . and written into the first area constituting the input buffer (!-BUF) on the main memory 4 (see Sl in FIG. 1).

At this time, the file to be processed is the manual buffer (1-B
When the memory size exceeds the memory size of the input buffer (1-BUF), the data is divided into partial elements each having an amount of data corresponding to the buffer size and taken into the input buffer (1-BUF). Further, the files to be processed are connected to a plurality of input/output channels 2, 2 . When the magnetic disk drives 3, 3゜, etc. under the control of... are distributed, the CP
U1 selectively specifies an input/output channel according to the file structure of the file to be processed and issues an access instruction, and the data of the read target file is stored in the main memory 4 under the control of the input/output channel 2.

When the CPUI reads the file to be processed or its partial elements into the first area constituting the input buffer (1-BUF) on the main memory 4 as described above, the CPU reads information regarding the file (file format, block length, etc.). (file information including record length, etc.), calculation instructions for the file, result identifier string according to the calculation result, and each area designation information in the main memory 4 for storing partial output data consisting of the record string of the calculation result, etc. It sends various types of information including information to the relational arithmetic processing device 5, and activates the relational arithmetic processing device 5.

When the relational calculation processing unit 5 receives the above information from the CPUI, it sets up the operation mode of the calculation unit 51 according to the instructions, and performs sorting or relational calculation processing in the RDB according to the above instructions, including key extraction processing. Execute.

That is, the control unit 53 in the relational arithmetic processing device 5 controls the main memory 4
Extract the key (key field) K1 according to the specified key from each record of the file data stored in the first area constituting the input buffer (1-BUF) above, and write one key field to each of the extracted keys. The serial number (record number) of the record in the file with the file name is added as an identifier R1, input data (DI) is generated in which the extracted key Kl and identifier R1 are combined, and the same data is written to the buffer memory 52. (See Figure 1, 82).

At this time, the control unit 53 synchronizes with the sending of data to the buffer memory 52 and checks whether the data is the key Kl or not using the identifier ^
A flag indicating whether it is 1 is added to each data.

In this key extraction process, if the processing target file does not fit into the input buffer (1-BUF) at once and is in the processing mode where it is imported into the input buffer (I-BUF) in units of partial elements, the above-mentioned manual buffer (1-BUF) -BUF) Each time the key extraction process for the partial element stored in the input buffer (1-BUF) is completed, the next partial element to be processed is taken into the input buffer (1-BUF).

Such extraction processing for each partial element is repeatedly executed until the end of the file to be processed.

By writing the input data (DI), which is a pair of extraction key Kl and identifier R1, into the buffer memory 52 through the key extraction process as described above for all target records of the processing target file, the buffer memory 52, for all files to be processed,
A data string is configured in which the key is a pair of 1 and the identifier R1.

Input data (D
When the data string consisting of I) is constructed on the buffer memory 52, the control section 53 activates the calculation section 51.

The calculation unit 51 receives input data (old) from the buffer memory 52.
are sequentially read together with the above-mentioned flags, and the calculations according to the calculation instructions from the CPUI are executed for each extraction key. Then, identifiers ^1. A2. . . . and output data (Do) consisting of the resulting identifier string is sent to the control unit 53.

The control unit 53 stores the output data (DO) consisting of the result identifier string received from the calculation unit 51 in a prespecified second area on the main memory 4, and performs calculations on all files to be processed. When the process is finished and the resultant identifier string is constructed on the second area, this is notified to the CPUI (see FIG. 1, 83).

When the CPUI receives the notification of the completion of the calculation from the control unit 53 of the relational calculation processing device 5, it stores the result identifier string (R1, Rj) in the second area of the main memory 4.

...) and the logical and physical information of the file to be processed, the in-disk location information string (D
Ai, DAj,...), and based on this disk position information sequence (DAI, oAj,...), obtain the disk address that includes the record storage area of the calculation result, and then perform magnetic recording according to the same address. Disk device 3゜3,
. . , and extracts record data (record of the calculation result) whose head position is the disc location information from the read data, and reads the disc location information string (DAI, oA, +, . . . ) from the read data. ) are sequentially stored in the third area constituting a predetermined output buffer of the main memory 4 (see FIG. 1 84).

Then, when the record string of the operation results stored in the third area constituting the output buffer reaches a predetermined data amount determined by the buffer size, the CPU In order to store the output file in the specified output file storage area (R) on the magnetic disk device 3,
Perform the storage procedure to the corresponding input/output channel 2. As a result, under the control of the human output channel 2, the partial output data consisting of the record string of the calculation results stored in the third area constituting the output buffer is transferred to the output file storage area (R) on the magnetic disk device 3. (Fig. 1 85)
.

Such processing generates an in-disk location information sequence (DAl, DAj) for all files to be processed.

) is executed repeatedly.

In this way, by creating an output file according to the result identifier string, it is possible to sort or RD a large number of processing target files that cannot be stored in the main memory 4 at once.
Complex arithmetic processing such as relational arithmetic operations in B can be executed quickly and efficiently without the need for partial arithmetic operations based on the amount of data determined in the work area of the main memory 4, or even arithmetic processing such as merging. .

In the above embodiment, the CPU, upon receiving the above calculation completion notification from the control unit 53 of the relational calculation processing unit 5, stores the result identifier string (R1, R
j, ...), an output file consisting of a record string arranged in the order of the calculation results was constructed in the output file storage area (R) on the magnetic disk device 3. By providing the channel 2 or the relational arithmetic processing unit 5, the CPU load can be reduced.

With the configuration of the embodiment described above, advanced data processing for complex operations such as sorting or relational operations in RDB can be efficiently executed with simple control, and especially when the target files are independent of each other. When the amount of data to be processed exceeds the work area size of the main memory 4, the high-level arithmetic processing described above can be efficiently executed. can.

Second Embodiment Next, a second embodiment of the present invention will be described with reference to FIG.

This second embodiment is different from the first embodiment in that the result identifier string (
R1, Rj, . . . ), the magnetic disk drives 3, 3 . ...The record of the calculation result is cut out and stored in the output buffer of the main memory 4, whereas in this case, the disk position information string (DAI,
DAJ, . . . . and obtains a record string as a calculation result from the same data. This reduces the number of accesses to the magnetic disk drives 3, 3, . . . compared to the first embodiment. We are trying to reduce the number and speed up the processing.

In the second embodiment shown in FIG. 2, a result identifier string (R1゜Rj, . . .
) to the disc position information string (DAI, DAj
.

) is the same as in the first embodiment described above.

In this second embodiment, the CPUI, as shown in FIG.

DAj,...) to the magnetic disk devices 3, 3.
゜... Check the distribution state of the target records above, determine an efficient data input unit (sector length of data read in one access), and according to this input input unit, the magnetic disk device 3, 3. ..., and stores the same data in a specific area of the main memory 4 (in this second embodiment, the third area), and the start address string of each record in the main memory 4. generate. Then, based on the start address string of this target record, one target record according to the calculation result is extracted from the third area on the main memory 4, and the fourth record that constitutes the output buffer on the main memory 4 is extracted. Store the calculation results in the area in order. Partial output data consisting of a record string according to the calculation result stored in this output buffer is stored in the output file storage area (R) on the magnetic disk device 3 in units of output buffer size.

Such processing generates in-disk location information sequences (DAI, DAj) for all files to be processed.

) is executed repeatedly.

In this way, by creating an output file according to the result identifier string, it is possible to perform complicated relational operations such as sorting or RDB related operations on a large number of processing target files that cannot be stored in the main memory 4 at once. calculation processing,
The frequency of access to the magnetic disk devices 3, 3, . . . can be suppressed and execution can be performed efficiently and at high speed.

Note that also in this second embodiment, the result identifier string (R1,
Rj, . . . ) to the input/output channel 2 or the relational arithmetic processing unit 5, the CPU load can be reduced.

Furthermore, in each of the above embodiments, a specified key (key field) is extracted from a record of a file to be processed, and the serial number of the record with that extraction key in the file is added as an identifier to the key, and the key is Input data (old) consisting of a pair of identifiers was obtained, but this identifier is not limited to a serial number within the file to be processed.In short, it is necessary to know the position of the target record in external storage as a result. Any information that can be used is sufficient.

Further, in each of the above embodiments, the input buffer (first
area) has a single buffer configuration, but this manual buffer (first area) is configured with a double buffer, and the magnetic disk drives 3, 3 . . . . and key extraction processing may be performed in parallel.

[Effects of the Invention] As detailed above, according to the present invention, a database that includes sorting and relational operations in a relational database as processing targets for processing target files placed in a single or multiple external memories. In a data processing device that executes predetermined arithmetic processing for a file, when the amount of data in the file to be processed exceeds the main memory capacity, a key is extracted and an identifier is assigned to the same key for each partial element of the file to be processed. Perform 'am' processing such as sorting and merging on the data string with the bare identifiers and keys of all the files to be processed obtained by repeating this process, and as a result, the obtained identifier Read multiple data blocks containing records indicated by the above identifier column into the main memory from the logical information and physical information of the column and the target file, and from this data write partial elements of the result file to the records indicated by the above identifier column. By creating a configuration in which the file is created, outputted to external storage, and this process is performed for all identifiers in the result identifier column, even if the amount of data in the target file exceeds the capacity of the main memory, sorting or RD processing is possible.
Processing such as relational calculations in B can be executed quickly and efficiently.

[Brief explanation of drawings]

FIG. 1 is a block diagram showing a first embodiment of the invention, and FIG. 2 is a block diagram showing a second embodiment of the invention. ■... CPU, 2.2... Input/output channel (l1O-CI) 3, 3,... Magnetic disk device (DISK) 4... Main memory (HEM
) 5...Relational arithmetic processing unit (1?0PU), 6
. . . system bus (S-BLIS), 51 . . . arithmetic unit, 52 .

Claims (2)

[Claims] (1) Data processing that executes predetermined arithmetic processing on databases that include sorting and relational operations in relational databases on files to be processed stored in single or multiple external storages. In the device, when the amount of data of the file to be processed exceeds the data size that can be processed at once in the main memory, the file to be processed is divided into a plurality of partial elements, and each partial element is stored in the first part of the main memory. In response to receiving a calculation instruction including a means for importing into an area, logical information including block length, record length, key position, key length, etc., and an operation type regarding the file, the data is stored in the first memory in the main memory according to the logical information. means for extracting a key to be processed from a record of a file to be processed stored in an area of and adding a serial number or relative position information within the file of a record having the same key as an identifier to the key; The process is executed on all of the files to be processed above, and as a result, using the obtained key and identifier pair as input data, the operation according to the above operation instruction is executed on the key value of the same data, and the operation result is means for storing, in a second area of the main memory, an identifier string arranged in an order according to the above; means for obtaining a positional information string on an external storage of a calculation result record from the information; and a read access to the external storage according to the positional information string to transfer a record according to the positional information string from the external storage to a third area of the main memory. and means for storing in a predetermined output file storage area of the external storage in units of a predetermined amount of captured data, and stores corresponding records for all of the position information strings in the output file storage area of the external storage. Data processing equipment. (2) Data processing that executes predetermined arithmetic processing on databases that include sorting and relational operations in relational databases on files to be processed stored in single or multiple external storages. In the device, when the amount of data of the file to be processed exceeds the data size that can be processed at once in the main memory, the file to be processed is divided into a plurality of partial elements, and each partial element is stored in the first part of the main memory. In response to receiving a calculation instruction including a means for importing into an area, logical information including block length, record length, key position, key length, etc., and an operation type regarding the file, the data is stored in the first memory in the main memory according to the logical information. means for extracting a key to be processed from a record of a file to be processed stored in an area of and adding a serial number or relative position information within the file of a record having the same key as an identifier to the key; The process is executed on all of the files to be processed above, and as a result, using the obtained key and identifier pair as input data, the operation according to the above operation instruction is executed on the key value of the same data, and the operation result is means for storing in a second area of the main memory an identifier string arranged in an order according to the following; and a calculation result record from the identifier string stored in the second area of the main memory and the logical information and physical information of the target file. means for obtaining a positional information string on an external storage; and determining an input unit of external storage data that includes one or more target records from the positional information string;
means for reading the data of each input unit from the external storage into the third area of the main memory in the order according to the position information string; and the beginning of each record of the data placed in the third area of the main memory. means for obtaining addresses in the order of the result identifier string, extracting record data from the third area of the main memory in the order of the address string, and outputting data to a fourth area of the main memory according to logical information of the result file; and means for outputting the output data configured in the fourth area of the main memory to a predetermined output file storage area of the external storage according to physical information including an external storage address of the result file. However, by storing the output data for all of the above-mentioned processing target files in the output file storage area of external storage, it is possible to obtain an output file in which arithmetic processing is performed on the processing target files that exceed the amount of data that can be processed in the main memory. Featured data processing equipment.