JP2004288213A - Data processing system and data processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To improve throughput of searching processing by reducing the frequency of read from an external storage unit, for a database system in which all item search is frequently requested for the same data. <P>SOLUTION: In a database system of parallel execution type comprising a plurality of processing units, a dictionary administration part, a buffer administration part, and a search execution control part are provided. The buffer administration part, accompanied by reading of a new data block from the external storage unit, transfers the read data block to other processing units. The dictionary administration part administrates recording of the address information for locating the data block once read and transferred to each processing unit. The search execution control part executes and controls search on the data block in each processing unit according to the above address information. Thus, the frequency of reading from the external storage unit is reduced to improve the performance of all item search of the system. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a database system, and more particularly to a database system suitable for a case where an atypical search (all-item search) frequently occurs.

  In a conventional database system, when a search for all data occurs in response to a data search request from a certain user, data is sequentially stored in a data buffer on a main memory in a processing device in order from the top of data constituting a database in a large-capacity external storage device. When reading is performed, and furthermore, a full search is performed in response to a data search request from another user in succession, in the same manner as described above, the data constituting the database in the external storage device is sequentially stored in the main storage in the processing device from the beginning. Is read into the data buffer (). That is, in the above database system, when the next all-item search occurs after the first data read into the data buffer in the first all-item search is evicted, the first data remains in the data buffer on the main memory (there is no first data). Since some data cannot be reused, all data is read from the external storage device to the data buffer again.

Conventionally, a method of narrowing the search range by creating an index for the target data at the time of data search has been used, but in the use form where it is unknown in advance what search form will occur, It is difficult to create an index, and consequently it is necessary to read all the data. Therefore, a method of pre-reading when reading data from an external storage device to shorten an apparent search time is often used. Furthermore, by dividing a series of processes involved in a search and executing the processes in parallel by a plurality of processing devices, the load on each processing device can be reduced and equalized to improve the performance of the entire search process. A distributed processing method has been proposed in Patent Document 1.
An Introduction to Database Systems, CJ Date, ADDISON-WESLEY, 1986, pp45-pp58 JP-A-3-41550

  Generally, the capacity of a data buffer in a processing device is smaller than the capacity of an external storage device, and data read from the external storage device to the data buffer is discarded when the buffer memory becomes full after use of the data. Therefore, when the same data is used again in another search, the data must be read from the external storage device again. For this reason, in a database system in which all-item searches frequently occur, all data is read from the external storage device each time. As a result, the frequency of reading data from the external storage device increases, and the throughput of the search processing increases significantly. There was a problem that it was lowered.

  Also, in a conventional database system in which prefetching of data and parallel distributed processing described above are performed, when a search for all records frequently occurs due to a search request from a large number of users, an external storage device is used in the same manner as described above. The frequency of reading the data increases, and the throughput of the search processing decreases. When a plurality of search requests during the parallel processing are mixed, the reading position in the external storage device fluctuates greatly, and the reading operation from the external storage device itself is not performed. There is a problem that the throughput may be reduced.

  Accordingly, an object of the present invention is to solve the above-described problem, reduce the frequency of reading data from the external storage device when all-item search is frequently performed, and simultaneously execute different all-item search for the same data. It is an object of the present invention to provide a database system capable of executing searches in parallel to improve the throughput of search processing.

  (1) In order to achieve the above object, the database system of the present invention is a database system that performs a data search for a database in an external storage device by a plurality of processing devices that can operate in parallel, with the data search. For each data block once read from the database, use of a dictionary management unit for recording management of address information specifying a processing device having a data buffer holding the data block, and use of a data buffer in each processing device Depending on the situation, a buffer management unit that transfers a data block that satisfies a predetermined condition to another data buffer, and a search execution control unit that performs execution of data search based on the address information in the dictionary management unit Is provided.

(2) In the configuration of (1), the data search is an all-item search,
The buffer management unit transfers a plurality of data blocks read from the external storage device in the first all-item search to a data buffer in each processing device for each predetermined number of blocks, and the search execution control unit The execution control of the subsequent all-item search is performed for the data blocks held in the data buffers in the respective processing devices for each block number.

  (3) In the configuration of (2), a transfer destination of the data block by the buffer management unit is a data buffer in an unused processing device or a data buffer in a designated processing device.

  The operation based on the above configuration will be described as follows.

  In the database system of the present invention, in a database system in which a data search for a database in an external storage device is performed by a plurality of processing devices capable of operating in parallel, each data block read from the database once along with the data search A dictionary management unit that performs recording management of address information that specifies a processing device having a data buffer that holds the data block, and data that satisfies a predetermined condition according to a use state of a data buffer in each processing device. Firstly, by having a configuration including a buffer management unit that transfers a block to another data buffer, and a search execution control unit that controls execution of data search based on the address information in the dictionary management unit. External storage device for data retrieval The read data block is transferred to a data buffer in another processing device without being discarded, and is recorded and managed by the dictionary management unit when a search request for the data block for the second time or later is made. Based on the address information, the data block held in the data buffer in one of the processing devices can be reused.

  That is, in the second and subsequent data searches for the same data, the data having a high access speed (about 100 MB / sec) is read without reading the data from an external storage device having a low access speed (about 1 MB / sec). Since the search process can be performed immediately on the data in the buffer, the frequency of reading data from the external storage device is reduced particularly when all-item search is frequently performed, thereby improving the search process throughput. be able to. Further, it is possible to further improve the throughput of the search process by executing different all-item searches for the same data in parallel.

  According to the database system of the present invention, the data block that is first read from the external storage device as a target of data search is transferred to the data buffer in another processing device without being discarded, Upon a search request for a data block, it is possible to reuse the data block held in the data buffer in any of the processing devices based on the address information recorded and managed in the dictionary management unit. It becomes possible.

  That is, in the second and subsequent data searches for the same data, the data having a high access speed (about 100 MB / sec) is read without reading the data from an external storage device having a low access speed (about 1 MB / sec). Since the search process can be performed immediately on the data in the buffer, the frequency of reading data from the external storage device is reduced particularly when all-item search is frequently performed, thereby improving the search process throughput. The effect that it can be obtained is obtained. Further, there is an effect that different all-item searches for the same data can be executed in parallel, and the throughput of the search process can be further improved.

  Hereinafter, an embodiment of a database system according to the present invention will be described in detail with reference to FIGS.

  FIG. 1 is a block diagram showing an example of the overall configuration of the database system of the present invention. In the figure, a database system is a plurality of processing units 102, each of which is hardware for executing data search / update processing, which can operate in parallel, and the respective processing units 102 are mutually connected to enable data transfer. , And a database 111 stored in an external storage device such as a large-capacity magnetic disk device. The processing device 102 has a search / update request input unit 101, which is a program for receiving a search request from a user, as a basic component of the database system, analyzes the input search request from the user, and analyzes the entire search request. A search processing request analysis unit 103 that determines specific search processing contents such as whether or not a search should be performed, a search execution control unit 104 that controls execution of the determined search processing, A dictionary management unit 106 for recording management of address information indicating whether the data is stored, a database access control unit 108 for managing access to the database 111 in the external storage device, and a main unit for storing data read from the database 111. Data buffer 110 on the storage device, management of data buffer 110 and other processing devices A buffer management unit 109 that performs data transfer control is provided with 102.

  FIG. 2 is a block diagram showing details of the dictionary management unit in FIG. In the figure, a dictionary control unit 106 is provided with a control block 107 for management, and a processing device having a data buffer 110 in which a head data block is held by a table name index 202 managed by a data management block 201. A head data block address 203 for specifying 102 and a general data block address 204 for specifying the processing device 102 having the data buffer 110 holding other data blocks are stored for each table.

  FIG. 3 is a block diagram illustrating a specific application example (data division type) of the database system according to the present invention. In a configuration in which the purpose of use of some of the plurality of processing devices 102 is undecided (floating). When a necessary data block is read from the external storage device having the database 111 in the first search of all records, the data buffer 110 in the processing device 102 to which the external storage device is connected is filled with the read data block. Then, by transferring the data block to the data buffer 110 in the other processing device 102, it is possible to obtain desired data from the other processing device 102 in the second and subsequent all-item search. To reduce the frequency of reading from.

  In an atypical business, there are many cases where an index corresponding to a search condition does not exist, and even when an index exists, it is impossible to narrow down the search condition. In such a case, as a method of searching the database, a full search must be performed. That is, not only a search request having no search condition but also a search request in which various search conditions are specified (not limited to the same search request), it is necessary to perform a search for all data. Many occur.

  In FIG. 3, by placing all of the data blocks read from the external storage device in response to the first all-item search request in the data buffer 110 in each processing device 102, the subsequent all-item search request In this case, it is possible to execute the all-item search and respond to the request without performing any read operation from the external storage device. Here, the dictionary management unit 106 provided in another processing device 102 performs recording management of address information that specifies the processing device 102 having the data buffer 110 in which each data block is arranged. That is, when the all-item search request 302 of the user B occurs after the all-item search is performed in response to the all-item search request 301 of the user A, the desired data is inquired to the dictionary management unit 106. Since all of the processing devices 102 having the data buffers 110 in which are stored are known, by performing a search process on these processing devices 102 in parallel, it is possible to easily perform a full search. In updating the data block, the update is performed using another area other than the transferred data block, so that there is no adverse effect on the simultaneous search of all records. In the example of FIG. 3, a search is performed on the data buffers 110 in the plurality of processing devices PB and the PC in response to the all-item search request 302 of the user B, and one search is performed in response to the update request of the user C. Is updated using the data buffer 110 in the processing device PA.

  FIG. 4 is a flowchart showing the flow of the process in FIG. In the figure, when performing all-item search processing, an inquiry is made to the dictionary management unit 106 to obtain whether or not desired data is stored in any of the processing devices 102 (step 401). ). If there is a processing device 102 having the desired data, the process branches to step 410, and a search request is issued to all of the corresponding processing devices 102 at the same time to perform an all-item search. On the other hand, if the desired data is not stored in any of the processing devices 102, the process branches to step 402, where the data buffer 110 may be released into the processing device 102 to which the external storage device is connected. It is determined whether or not there is a data buffer (the "data buffer that may be released" is, for example, data of another table which is not referred to by any currently executing retrieval processing, etc. is stored). Data buffer area.) If it is determined in step 402 that there is a data buffer 110 that can be released, desired data is read from the external storage device into the data buffer 110 (step 403), and the dictionary management unit 106 is updated (step 403). Step 404). If it is determined in step 402 that there is no data buffer 110 that can be released (that is, if the data buffer 110 is full), the process branches to step 406 to float (unused). A data buffer 110 that may be released in another processing device 102 is found, and data that has already been used is transferred to the data buffer 110 in the other processing device 102 and expelled (step 407). The management unit 106 is updated (Step 408). Then, desired data is read into the data buffer 110 in the processing device 102 to which the external storage device vacated by the transfer is connected (step 409). As described above, the used data block can be transferred or reused when the data buffer overflows to another processing device, or at regular time intervals or at regular access amounts. .

  Further, according to the above, since the search is performed by the processing device 102 different from the processing device 102 to which the external storage device is connected, the load on each processing device used for input / output at the time of all-item search is reduced. Can be done. Further, in order to support the capacity of such a processing device, the data in the data buffer in the processing device 102 to which the external storage device is connected is transferred in order to effectively utilize the capabilities of the other processing devices. When a search is performed in the transfer destination processing device 102, the transfer destination processing device 102 is used exclusively for reference, so that exclusive control is not required and reference performance can be improved. Since the search is performed only by the original processing device 102 (to which the external storage device is connected), there is no need to interrupt the search of all the records being executed in consideration of the influence of the update.

  FIG. 5 is a block diagram for explaining a specific application example (pipeline type) of the database system according to the present invention. By sequentially transferring each of the completed data blocks to a vacant data buffer 110 in another designated processing device 102, the data blocks can be reused in the second and all subsequent searches. It is possible.

  In FIG. 5, a case is considered in which the data blocks 1 and 2 are read from the database 111 in the external storage device to the data buffer 110 of the processing device PA in response to the all-item search request 301 of the user A. In general, the capacity of the data buffer 110 is smaller than the capacity of the external storage device, and the next data block 2 cannot be read while retaining the data block 1 for which the search has been completed. Was thrown away. On the other hand, in this application example, the used data block 1 is transferred to another designated processing device 102 having the data buffer 110 having a sufficient remaining capacity, so that the data block 1 is not read from the external storage device. It is possible to perform a search according to the next all-user search request 302 of the user B. On the other hand, the update request 303 from the user C is updated in the data buffer 110 in the processing device 102 to which the external storage device is directly connected, separately from the data block transferred to the other processing device 102 as described above. Perform processing. As a result, a plurality of all-item search processing can be executed by a processing device different from the processing device in which the update process is performed, and the data blocks sequentially read from the external storage device can be reused on another processing device 120. it can.

  FIG. 6 is a flowchart showing the flow of the process in FIG. In the figure, when performing an all-item search process, an inquiry is made to the dictionary management unit 106 to specify the processing device 102 having the data buffer 110 in which the first data block of the data block group to be searched is stored. Address information 203 to be found is found (step 601). If there is a head data block, the search is performed on the head data block, and then the flow waits for the next data block to flow (step 611). On the other hand, if none of the processing devices 102 has the data buffer 110 in which the first data block to be searched is stored, the data buffer 110 in the processing device 102 directly connected to the external storage device has a free space. It is checked (step 602), and if there is a vacancy, a desired data block is read from the database 111 in the external storage device into the data buffer 110 (step 603), and the dictionary management unit 106 is updated. (Step 604), a search is performed using the read data block (step 605). If there is no free space in step 602, it is checked whether or not there is free space in the data buffer 110 in the designated next processing device 102 (step 607). Is transferred (step 608), the dictionary management unit 106 is updated (step 609), and then the next data block is read from the database 111 in the external storage device into the data buffer 110 having an empty space due to the transfer (step 608). 610), a search is performed using the read data block (step 605). If there is no free space in the data buffer 110 in the next processing device 102 specified in step 607, the data in the data buffer 110 in the next specified processing device 102 is further linked in a chain (such as Tokoroten). After reading out, a desired data block is read (step 612), and a search is performed using the read data block (step 605). As described above, the used data block can be transferred or reused when the data buffer overflows to another processing device, or at regular time intervals or at regular access amounts. .

  Further, in any of the data division type and the pipeline type, by providing a sufficient processing device 102, all data blocks read from the external storage device can be held in the data buffer 110 in the processing device 102. Then, it is also possible to perform a full search without making an input request to the external storage medium 111 at all. The update processing of the data block is executed in the data buffer 110 in the processing device 102 to which the external storage device is directly connected, and the updated data block is transferred to the data storage in the other processing device 102 by normal data transfer. This is reflected in the buffer 110.

  As described above in detail, according to the database system according to the embodiment of the present invention, in a database system in which a plurality of processing devices capable of operating in parallel perform data search for a database in an external storage device, For each data block once read from the database, a dictionary management unit that performs recording management of address information that specifies a processing device having a data buffer holding the data block, and a data buffer in each processing device. A buffer management unit that transfers a data block satisfying a predetermined condition to another data buffer according to a use condition, and a search execution control unit that controls execution of a data search based on the address information in the dictionary management unit With the configuration having The data block first read from the external storage device as a data search target is transferred to a data buffer in another processing device without being discarded, and a search request for the second or subsequent data block is performed. In this case, the data block held in the data buffer in any of the processing devices can be reused based on the address information recorded and managed in the dictionary management unit.

  That is, in the second and subsequent data searches for the same data, the data having a high access speed (about 100 MB / sec) is read without reading the data from an external storage device having a low access speed (about 1 MB / sec). Since the search process can be performed immediately on the data in the buffer, the frequency of reading data from the external storage device is reduced particularly when all-item search is frequently performed, thereby improving the search process throughput. The effect that it can be obtained is obtained. Further, there is an effect that different all-item searches for the same data can be executed in parallel, and the throughput of the search process can be further improved.

FIG. 1 is a block diagram illustrating an example of an overall configuration of a database system according to the present invention. FIG. 2 is a block diagram illustrating details of a dictionary management unit in FIG. 1. It is a block diagram explaining the specific application example (data division type) of the database system of this invention. 4 is a flowchart illustrating a flow of processing in FIG. 3. It is a block diagram explaining the specific application example (pipeline type) of the database system of this invention. 6 is a flowchart illustrating a flow of processing in FIG. 5.

Explanation of reference numerals

101 search / update request input unit 102 processing unit (CPU)
103 Search processing request analysis unit 104 Search execution control unit 106 Dictionary management unit 107 Control block 108 Database access control unit 109 Buffer management unit 110 Data buffer 111 Database in external storage device 112 Between processing devices High-speed transfer path 201 Data management block 202 Table name index 203 First data block address 204 General data block address

Claims (3)

In a database system in which a data search targeting a database in an external storage device is performed by a plurality of processing devices capable of operating in parallel,
For each data block once read from the database with the data search, a dictionary management unit for recording management of address information specifying a processing device having a data buffer holding the data block,
A buffer management unit that transfers a data block that satisfies a predetermined condition to another data buffer according to a use state of a data buffer in each processing device;
A database system comprising: a search execution control unit that controls execution of data search based on the address information in the dictionary management unit. The data search is a full search,
The buffer management unit transfers a plurality of data blocks read from the external storage device in the first all-item search to the data buffer in each processing device for each predetermined number of blocks,
2. The search execution control unit performs execution control of a subsequent search for all data for the data blocks held in a data buffer in each processing device for each of the number of blocks. The database system described. 3. The database system according to claim 2, wherein a transfer destination of the data block by the buffer management unit is a data buffer in an unused processing device or a data buffer in a designated processing device.

Images