JP2002182953A - Distributed file management method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accelerate parallel disk device access at the time of I/O requests in a method of distributing and arranging a plurality of logic files to disk units by a striping file mechanism. SOLUTION: In this method of distributing and arranging the logic file 603 to sub-filing systems 614-617, stride groups SG0-SG3 which are the sets of file blocks are generated so as to arrange the file blocks FB0-FB15 of the same file in the batch area of the disk devices 610-613 and they are written altogether to the respective sub-filing systems by the stride group unit. Also, when accessing the file block, the successive file blocks FB4-FB15 are looked ahead to the buffer cache 604 of a master filing system 605, the number of times of requesting read to the sub-filing systems is reduced and file access is accelerated further.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of dividing a logical file virtually existing in a master file system into a plurality of disk devices and storing the divided logical files so that the file blocks of the plurality of disk devices can be accessed in parallel. This is related to a file operation method called a striping file mechanism.

[0002]

2. Description of the Related Art Conventionally, a method of dividing a logical file into a plurality of disk devices and arranging the logical file on a plurality of disk devices by a striping file mechanism as disclosed in Japanese Patent Application Laid-Open No. 9-223049 has been proposed when an I / O request to the logical file is made. It is known that a plurality of disk devices can be accessed in parallel, and this file access technology is widely used in large-capacity file management in a parallel computer.

[0003]

In the above-mentioned prior art, parallel access is enabled by distributing logical files to a plurality of disk devices, and the read request for a large-capacity file is speeded up. However, when writing to the disk device,
Since each file block is stored individually in the disk device without recognizing the data group to be stored in the same disk device, the arrangement of the file blocks on the disk device varies, and when reading a file, the disk device waits for rotation. However, the number of seeks (searching for file blocks) increases, which causes performance degradation.

[0004] It is an object of the present invention to provide a method for distributing logical files to a plurality of disk devices by a striping file mechanism, to improve the efficiency of parallel reading to a disk device when reading a file, and to reduce the time required to read a large capacity file. Is to shorten the time.

[0005]

In order to achieve the above object, in a method for managing logical files in a distributed manner by a striping file mechanism, when a logical file is distributed to a plurality of disk units, the same file allocated to each disk unit is allocated to the logical unit. By writing the file blocks in a lump, the number of seeks when reading the file blocks from the disk device is reduced, and the access time to the file is reduced. In addition, by writing the file blocks to the disk device all at once, it becomes possible to read ahead the file block to be accessed in the buffer cache of the master file system at the time of reading the file, and to make a read request to each disk device. In order to reduce the number of times, the file reading time is shortened.

[0006]

Embodiments of the present invention will be described below. FIG. 1 shows the overall structure of an embodiment of the present invention. 101 is a master file, 102 is a configuration file, 103 is a logical file, FB0 to FB15 are file blocks obtained by dividing the logical file into a certain size, and S
G0 to SG3 are stride groups which are sets of file blocks, 104 is a master file system for storing 101, 102, and 103, 105 to 108 are subfiles A to D, 109 to 112 are disk devices A to D, and 113 are Reference numeral 116 denotes a subfile and subfile systems A to D for storing disk devices.

[0007] The master file has information on which sub file is stored in the file block of the logical file, and manages the distributed logical file. The configuration definition file defines information to be set when using the striping file mechanism. These pieces of information include the sub-file system used by the striping file mechanism and the size of a file block when dividing a file. A logical file is a file to be divided and managed, and is logically created in the master file system. A file block is a file block obtained by dividing a logical file into a certain size, and is a minimum unit that is distributed and managed as a subfile. The stride group is a grouping of file blocks distributed to the same file system, and is a unit used for sub-file division. The master file system is a file system that uses a striping file mechanism and manages logical files and master files. Each subfile is a file in which actual data on the subfile system is stored. Each disk device physically stores a file, and one sub-file system creates a file system by mounting one disk device. Each sub file system is a file system in which a sub file is stored.

According to the present invention, a file block obtained by dividing a logical file in a master file system into a predetermined size is divided and arranged in a plurality of disk devices of a sub file system, and an I / O request from an application is transmitted to each of the disk devices. By executing the sub-files in parallel, the speed of file access is increased. In FIG. 1, the logical file is composed of file blocks 0 to 15 divided into a certain size. In the master file system in which these file blocks are arranged in a plurality of disk devices, the configuration file defines in advance the sub-file systems to which the file blocks are to be distributed. In FIG. 1, it is assumed that there are four disk devices mounted for the sub file system. At this time, file blocks 0, 4, 8, 12 of the logical file
To the disk device A of the sub-file system A, and the file blocks 1, 5, 9, and 13 to the sub-file system B.
File blocks 2, 6, 10,
14 is distributed to the disk device C of the sub-file system C, and the file blocks 3, 7, 11, 15 are distributed to the disk device D of the sub-file system D. Determined by number. The master file that manages the relationship between a logical file and a subfile is a file block that is first distributed to the same disk device without writing to the disk device in file block units when allocating the file block to the disk device. Is grouped. This group of file blocks is called a stride group. File blocks 0, 4, 8, and 12 are stride group 0, file blocks 1, 5, 9, and 13 are stride group 1, file blocks 2, 6, 10, and 14 are stride group 2, and file blocks 3, 7, 11,. 15 as a stride group 3 and collectively write to the disk device of the target sub-file system for each stride group.

FIG. 2 illustrates details of one file distribution process in the present method. FIG. 2 shows a process performed by the master file system when dividing a logical file in order to realize the present distributed file management method. S2
01, from the configuration file, the number of sub file systems that can be used for the striping file
Also, the size of the file block for dividing the file is read. In S202, the logical file is divided into file block sizes, and the total number of the divided file blocks is set to n. The divided file blocks are assigned block numbers 0 to n-1 in order from the beginning. At this time, even when the last file block is smaller than the defined file block size, it is regarded as one file block and a block number is defined. After the block numbers are defined for the file blocks, in S203, file blocks having the same remainder as the division of the block number by the number of sub-file systems are collected to form a stride group. The remainder is used as a stride group number. In S204, the stride groups generated in S203 are assigned to the sub file systems in the order of the stride group numbers.

FIG. 3 illustrates the arrangement of file blocks when a plurality of logical files are distributed to the same disk device using the above-described distributed file management method. 3, reference numeral 301 denotes a master file, 302 denotes a configuration file, 303 denotes a logical file 1, 304 denotes a logical file, FB0 to FB15 denote file blocks obtained by dividing the logical file 1 into a certain size, and SG0 to SG3 denote logical files. 1, FB16 to FB27 are file blocks obtained by dividing the logical file 2 into a predetermined size, Sg0 to Sg3 are stride groups each being a set of file blocks of the logical file 2, and 305 is 301 and 302. , 303, and 304, 306-3
Reference numeral 09 denotes the sub-files 1A to 1D in which the logical file 1 is divided and arranged in the sub-file system, and 310 to 313 denote the sub-files 2A to 2D in which the logical file 2 is divided and arranged in the sub-file system. D, 318 to 321 are two subfiles and subfile systems A to D for storing disk devices.
Is shown. In the disk device A of the sub file system A, when the writing of the stride group SG0 of the logical file 1 and the writing of the stride group Sg0 of the logical file 2 conflict with each other, the respective file block groups are collectively written to the disk device A as a stride group. . As a result, file blocks belonging to two different logical files are prevented from being alternately (or mixedly) written to the disk device, and file blocks of the same logical file are continuously arranged as much as possible in the free space in the disk device. This makes it possible to reduce the number of rotation waits and seeks when reading a file block, thereby improving the efficiency of a file read request.

FIG. 4 shows a sub-file system.
A method of allocating a stride group assigned by the master file system to a disk device will be described. In step S401, a file block group for each stride group is received from the master file system. S4
In 02, the file blocks received from the master file system are collectively allocated to the free space of the disk device. When the file block group of two or more logical files is written to the same disk device by this processing method, the writing to the file block group of the next logical file is started after the writing to the file block group of the same logical file is completed. Therefore, it is ensured that file blocks belonging to different logical files are not mixed and written to the disk device.

Next, a description will be given of a file access method for reading a logical file in the distributed file management method. FIG. 5 illustrates a file access method when a read request is issued to a logical file in FIG. 1, where 501 is a logical file, FB0 to FB15
Is a file block obtained by dividing a logical file into a certain size, SG0 to SG3 are stride groups each being a set of file blocks, and 502 to 505 are subfiles A.
To D and 506 to 509 indicate disk devices A to D. In FIG. 5, file blocks 0 to 4 of the logical file
, It is possible to read file blocks 0 to 3 in parallel, and the disk device A immediately executes a read request for file block 0 and then immediately executes a read request for file block 4. I can do it. Previously, when writing to a file,
Each block was separately written to the disk device without recognizing that it was stored on the same disk device for the divided file block group, so the file block arrangement on the disk device varied when reading the file. In this case, the number of times of waiting for the rotation of the disk device and the number of seeks increased, resulting in performance deterioration. On the other hand, when the file blocks belonging to the same logical file are collectively written, it is possible to efficiently read the file block group of the same logical file by reducing the rotation waiting and the number of seeks at the time of reading.

In the following embodiment, in this file management method, when reading out a sub-file, a method of pre-reading file blocks from information managed by a master file and storing them in a buffer cache of the master file system. Is shown. According to the file management method, when a read request is issued, a file block to which a read request is likely to occur next in the same disk device is written together with a currently read file block from a file block accessed first. , File blocks can be read ahead and stored in the buffer cache of the master file system.

FIG. 6 is a diagram for explaining a method in which a buffer cache is provided in a master file system in the present distributed file management method.
2 is a configuration definition file, 603 is a logical file, 604
Is a buffer cache, FB0 to FB15 are file blocks obtained by dividing a logical file into a certain size, SG0
SG3 is a stride group which is a set of file blocks, and 605 is 601, 602, 603, and 604
606 to 609 indicate subfiles A to D, 610 to 613 indicate disk devices A to D, and 614 to 617 indicate subfile systems A to D storing subfiles and disk devices.

The buffer cache is provided in the master file system and is used as an area for storing a file block read ahead from the sub file system.

In FIG. 6, when the file block 0 of the sub file A is accessed, the file block 4,
Since it is possible that 8 and 12 are accessed,
By pre-reading these areas to the buffer cache on the master file system when a read request for file block 0 is issued, the master file system can directly access the sub-file system without issuing a read request again from the master file system. It is possible to respond to a logical file read request. Therefore, the speed of file access can be further increased.

FIG. 7 illustrates a file access method when a read request is made using a master file system equipped with a buffer cache.
Is a logical file, 702 is a buffer cache, FB0
FB15 are file blocks obtained by dividing a logical file into a certain size, SG0 to SG3 are stride groups each of which is a set of file blocks, 703 to 706 are subfiles A to D, and 707 to 710 are disk devices A to D. In FIG. 7, when a read request is issued to file blocks 0 to 8 of the logical file, in this method, when a read request is issued to file block 0, the stride group to which file block 0 belongs is uploaded to the buffer cache. When a read request is issued to the file blocks 4, 8, and 12, it is not necessary to access the disk device A of the sub-file system A, and instead it is possible to access a file block in the buffer cache. Similarly, the stride groups SG1, SG2, and SG3 can be uploaded to the buffer cache. For this reason, when the range of a read request for a logical file is large, a file block is read in parallel from a plurality of disk devices using a striping file mechanism, and the file block exists in the disk device by one read request to the disk device. Since the subsequent file block can be uploaded, it is not necessary to issue a read request to the sub file system thereafter, and the file access can be speeded up.

Referring to FIGS. 8 and 9, the processing when a buffer cache is provided by the present file access method will be described with reference to flowcharts. The flowchart in FIG. 8 shows processing in the master file system, and the flowchart in FIG. 9 shows processing in each sub file system.

In the process of FIG. 8, when the master file system receives a read request from the application in S801, the master file system accesses the first file block by the read request in S802. At this time, first, in S803, it is checked whether the file block for which the read request has occurred exists in the buffer cache of the master file system. The first file block accessed is
Since it does not exist in the cache, the process moves to S805.
In step S805, a request to read the file block and the subsequent file block is issued to the allocated sub file system based on the information managed by the master file. In the access of the second and subsequent file blocks, the file block may be stored in the buffer cache.
Move to 804. In S804, the file block requested to be read is read from the buffer cache of the master file system. In step S806, it is confirmed whether access to all the file blocks for which the read request has been made is completed. If the read for the read request has been completed, the process ends. If the reading for the read request has not been completed, the next file block for which the read request has been issued is accessed in S807. At this time, the process returns to S803 to check whether the file block exists in the buffer cache. As described above, in the master file system, the file blocks that have been sequentially requested to be read are accessed, and when the file blocks are present in the buffer cache, the file blocks in the buffer cache are used to respond to the read request, and are not present in the buffer cache. In this case, a read request is issued to the sub-file system in which the file block is located.

Referring to FIG. 9, a description will be given of the processing on the sub file system side in the method of equipping the buffer cache. First, in step S901, a read request is received from the master file system, and in step S902, the file block from which the read request was issued is read, and the file block is transmitted to the master file system. In S903,
If there is a file block of the same stride group for which a read request has been made, the process of S902 is repeated to read the block. In step S903, if the reading of all the file blocks belonging to the same stride group has been completed, the process ends. At this time, in the present method, the file blocks of the same logical file are collectively arranged in a continuous area of the disk device by using the stride group at the time of writing the file. It becomes possible.

[0021]

According to the present invention, when file blocks of a plurality of logical files are arranged on the same disk device, it is possible to avoid arranging these file blocks in a mixed manner in the disk device and to store each file block. By collectively writing, it is possible to reduce the time required to search for a file block requested to be read when reading a logical file. Further, by pre-reading the file block, the number of read requests from the master file system to the sub file system can be reduced, and a large-capacity file access time can be reduced.

[Brief description of the drawings]

FIG. 1 is an overall view showing a configuration of a master file system and a sub file system according to an embodiment of the present invention.

FIG. 2 is a flowchart showing processing of a logical file division method in the embodiment of FIG.

FIG. 3 is a diagram showing a case where a plurality of logical files share the same sub-file system in the embodiment of FIG. 1;

FIG. 4 is a flowchart showing a file block writing process on the sub-file system side for writing in the embodiment of FIG. 1;

FIG. 5 is a diagram showing a file access method when a read request is issued to a logical file after the stride group is divided into respective sub-file systems in the embodiment of FIG.

FIG. 6 is a diagram illustrating a distributed file management system according to an embodiment of the present invention, in which a buffer cache is provided in a master file system, and file blocks belonging to the same stride group as a file block from which a file has been read are transferred to the master file system. FIG. 3 is an overall configuration diagram showing a prefetching method.

FIG. 7 is a diagram illustrating processing of a file access method and a file block prefetch method when a read request is issued to a logical file in the embodiment equipped with the buffer cache of FIG. 6;

FIG. 8 is a flowchart showing processing in a case where a read request is issued to a master file system in the embodiment for pre-reading the file block of FIG. 6;

FIG. 9 is a flowchart showing processing when the sub-file system receives a read request from the master file system in the embodiment for pre-reading the file block in FIG. 6;

[Explanation of symbols]

602: Configuration file, 603: Logical file, 6
04: Buffer cache, 605: Master file system, 606: Subfile A, 610: Disk device A, 614: Subfile system A

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Kazuhito Sugiyama 5030 Totsuka-cho, Totsuka-ku, Yokohama-shi, Kanagawa Prefecture Inside Software Development Division, Hitachi, Ltd. 81-chome Hitachi Software Engineering Co., Ltd. In-house F-term (reference) 5B065 BA01 CH02 5B082 CA18 CA20 FA12 5D110 AA13 DA11 DB05 DC03 DC16 DE01

Claims (2)

[Claims] In a file system including a plurality of disk devices, a logical file is divided into a plurality of file blocks of a fixed size, and these file blocks are distributed and arranged on the plurality of disk devices so that a plurality of files can be read at the time of reading a file. In a file system that can be accessed in parallel with disk units, file blocks of the same file are physically located in a non-contiguous area within one disk unit by distributing file blocks. A file file block is provided with management information so that the file blocks are continuously arranged, and can be read / written by a single I / O operation. 2. The distributed file management method according to claim 1, wherein the management information of the logical file is maintained on the file system, and when a read request is issued to a file block of the logical file, each disk device performs the following according to the information. A file block to be read in advance, and by saving these file blocks in a buffer cache on a file system, the number of times of reading to a disk is reduced.