JP4743444B2 - Data transfer method - Google Patents

Description

  The present invention relates to a data transfer method, and more particularly to a data transfer method capable of efficiently performing file transfer in a shared file system or the like.

  In recent High Performance Computing (HPC), the amount of input / output data is also increasing due to the speeding up of the CPU (Central Processing Unit), the increase in memory capacity and the improvement in memory bandwidth. Therefore, particularly in the field of HPC, a high-speed file sharing system based on a large-scale SAN (Storage Area Network) environment is used. One characteristic of HPC is that one file is large. However, not only that, but also a large number of files with a small size up to several hundred kilobytes such as environment setting files and log files are often handled at the same time. These files are used not only in the job execution environment but also in an environment where preparation and post-processing are performed. Therefore, file transfer is required between these environments. Examples of transferring a large number of these files include HPC staging / destaging and general copy / move commands.

  In a SAN shared file system, a control packet is communicated from a client node to a server node via a LAN (Local Area Network). When transferring a large file by this communication, a high-speed file transfer function utilizing the performance of the disk device is realized by directly inputting / outputting data to / from the disk device via the SAN. However, when a large number of small files are transferred, the overhead increases, so that sufficient performance cannot be obtained.

  One of the functions for executing jobs in the HPC cluster is staging / destaging with the configuration shown in FIG. Staging / destaging is one technique for separating the disk load for job execution from the disk load for normal access. Prior to job execution, files necessary for job execution are deployed to the job execution disk device of the compute node (staging), and after job execution, the job execution result file is saved to the normal access disk device (destage). In staging / destaging, a shared file system is created in the disk device in order to perform job execution and file transfer independently. Therefore, the computation node is devoted to job execution on the shared file system, and the file transfer node transfers the file on the shared file system instead of the computation node.

  As a shared file system used in staging / destaging, there is a shared file system using NFS (Network File System) or SAN. NFS is a widely used shared file system. As shown in FIG. 2, both control packet communication and data input / output are performed by a LAN (Local Area Network). Therefore, there is a problem that the load on the LAN is high and all input / output data passes through the NFS server, so the load on the server becomes high and the performance of the disk device cannot be exhibited.

  A shared file system using a SAN is a file sharing method that solves the above-mentioned NFS problems and is generally used in HPC. As one of file sharing methods using SAN, there is a method disclosed in Japanese Patent Laid-Open No. 11-338792 (Patent Document 1). As shown in FIG. 3, the client communicates control packets with the server via the LAN, and directly inputs / outputs data to / from the disk device via the SAN. The method of Patent Document 1 has a smaller load on the LAN and server than NFS, and enables high-speed data input / output utilizing the performance of the disk device.

  Furthermore, an improvement example of the method of Patent Document 1 is described in Japanese Patent Laid-Open No. 2003-345698 (Patent Document 2). As shown in FIG. 4, even when the files requested for access are distributed and stored in the disk device, the data storage location information is grouped into one file (striping disk unit). By transmitting, the number of control packet communications is reduced. Compared with the method of Patent Document 1, the method of Patent Document 2 can further reduce the load on the LAN. In HPC, such staging / destaging in a shared file system using SAN is generally performed.

Japanese Patent Laid-Open No. 11-338792 JP 2003-345698 A JP-A-6-231016

  As described above, there are various sizes of files transferred by the file transfer node in staging / destaging. The file to be transferred is not limited to a huge file of giga (G) or tera (T) byte order in which a huge calculation result such as a job execution result is stored. There are many small-sized files of up to several hundred kilobytes, such as environment setting files and log files. The methods of Patent Documents 1 and 2 are effective methods for transferring a large file, but there is a problem that the performance cannot be sufficiently exhibited by transferring a small file.

  In the method of Patent Document 1, the ratio of the communication time of the control packet to the time required for file transfer becomes large for a small file. Therefore, even if data input / output is performed at a high speed, the communication time of the control packet becomes a bottleneck, and sufficient performance cannot be obtained. As shown in FIG. 5, the method of Patent Document 2 is small in that small files are not distributed and stored in the disk device. Thus, the effect cannot be obtained sufficiently.

  Due to these problems, in the methods of Patent Documents 1 and 2, when a large amount of small files are included in staging / destaging, the performance of the entire staging / destaging is seriously affected. As a method for transferring a large number of small files, there are general copy / move commands and the like in addition to staging / destaging of HPC.

  In addition, as a method for efficiently handling a plurality of files, Japanese Patent Laid-Open No. 6-231016 (Patent Document 3) describes a method for combining files into one. However, Patent Document 3 does not consider that the disk storing the file can handle data only in units of sectors (512 bytes). Therefore, even if the files are combined by this method, the created combined file is not stored. It cannot be used. Furthermore, since it is not considered to return the combined file to a plurality of original files at the transfer destination, it cannot be used for file transfer. Therefore, even when such a file transfer includes a large number of small files, it is a problem to provide a function for improving the file transfer performance by solving the above-mentioned problems of Patent Documents 1, 2, and 3. It has become.

  An object of the present invention is to provide a data transfer method that solves the above-described problems.

  In order to solve the above-described problems, the data transfer method of the present invention corrects the size of a large number of small files in the transfer source storage device from block units to sector units, and concatenates these size corrected files. A single virtual file is collected and transferred to the transfer destination storage device as a single file.

  In the data transfer method of the present invention, the file name list transferred by the transfer source server is received, the virtual file creation function reduces the file size of the plurality of files from the block unit to the sector unit, and concatenates these files. As a single virtual file, in accordance with read access, the created virtual file is written to the file transfer memory area, the transfer destination server receives the file name list to be transferred, and the data area securing function is stored in the transfer destination disk device. It is also possible to secure a data area and write a virtual file in the file transfer memory area to the data area of the transfer destination disk device according to the access request.

  Further, the data transfer system of the present invention is characterized by using these data transfer methods described above.

In the present invention, by transferring a virtual file in which a plurality of files are combined into one, file transfer can be performed at a speed higher than that of a file unit.
According to the present invention, a data transfer method capable of efficiently performing data transfer is obtained.

  Embodiments of the present invention will be described in detail with reference to the accompanying drawings. First, the basic principle and basic configuration of the invention will be described, and then more detailed description including a specific data structure and a flowchart will be given.

  The present invention adds a function (virtual file function) for handling a large number of small files as one large file to the shared file system. That is, high-speed file (data) transfer is realized by handling a large number of small files as one large file. FIG. 6 shows the configuration of the virtual file. FIG. 7 shows a system configuration diagram of the shared file system. FIG. 8 is a file management table, FIG. 9 is an explanatory diagram for explaining the data structure of a normal file and a virtual file, and FIG. 10 is a data arrangement diagram before and after correcting a gap between virtual files.

  A virtual file is a file in which a large number of small files are collected as one large file. This virtual file is an image obtained by concatenating a plurality of files as shown in FIG. 6, and is a readable / writable virtual file similar to the file. In order to realize the virtual file, as shown in FIG. 7, a virtual file creation function (206) and a virtual file deletion function (207) are prepared in the server (200). The virtual file is managed in the same manner as a general file by the file management function (205) and the file management table (209) of the server (200). As shown in FIG. 8, the file management table (209) includes, for each file, a file management number, a file type such as “normal file” and “directory”, a list of data storage location information where the file is stored, and the like. Is stored as one entry.

  For the virtual file, a “virtual file” is newly added to the file type of the file management table (209), and the data storage location information is shared with the existing file. In FIG. 9, the virtual file 1 has the file type “virtual file” and has data storage position information of all of the files A, B, and C. Accordingly, by accessing the virtual file 1, the file A, the file B, and the file C can be input / output as one connected file.

  File transfer using virtual files is performed as follows. First, processing is started from the virtual file transfer program (101). The virtual file transfer program (101) is a user mode application that transfers a virtual file. However, in the present invention, the virtual file transfer function (virtual file transfer means) controlled based on the virtual file transfer program is simply described as a virtual file transfer program.

  The virtual file transfer program (101) designates a plurality of files, requests the virtual file creation request function (104) to create a virtual file, and uses the virtual file creation function (206) of the transfer source server to Create The virtual file transfer program (101) designates the created virtual file, makes a read request from the shared file system client function (103), and stores the virtual file in the file transfer memory area (108), that is, a plurality of files. Are loaded at once. When the virtual file has been read, the virtual file deletion request function (105) requests deletion of the virtual file. Next, the virtual file transfer program (101) secures a data storage area for each file by the data area securing function (208) of the transfer destination server, and creates a virtual file in which the data storage areas are collected. After that, write in the same way as read and delete the virtual file.

  The virtual file data input / output from the client (100) to the disk can be accessed only by a multiple of the sector unit. The file system generally manages a data storage area for each block size. However, since the file size is in units of bytes, when reading a virtual file, it must be taken into account that an indefinite length gap occurs between individual files, as in the arrangement of data before correction in FIG. .

  Therefore, in the case of inputting / outputting data with any file system of any block size as in the corrected data arrangement of FIG. 10, the data arrangement is always corrected after the sector arrangement is corrected. By preparing the disk access conversion function (107) and the disk access conversion table (108), even if the block size differs depending on the file system, the gap position is correctly recognized, and unnecessary input / output to the disk device is performed as much as possible. Make it possible not to. In this way, the block unit file is converted into the sector unit. For this reason, the gap between files becomes 1 sector or less, and the file size is reduced.

  Regarding writing to the transfer destination server, before transferring the file to the transfer destination server, the data storage area (208) secures a data storage area of each file in the disk device of the transfer destination server. Thereafter, the virtual file creation function (206) creates a virtual file concatenating these files at the transfer destination. Finally, data is written to the virtual file on the destination server. A plurality of small-sized files are grouped into a virtual file at the transfer source, but by creating a virtual file in the transfer destination in the same way, it is the same as each file in the disk device of the transfer source server. As described above, each file data can be written to the transfer destination disk.

  With the above functions, the client inputs / outputs not a single file but a virtual file that combines them. The number of input / output operations can be reduced from individual file units to virtual file units, and the increase in control packets in a small file, which was a problem of Patent Document 1, can be reduced. Furthermore, by storing virtual files on the server using the method described above, a state in which many files are physically distributed and stored is changed to a state in which one virtual file is physically distributed and stored. Can be replaced. For this reason, the function which collects the data storage position information of patent document 2 per file can be used more effectively.

  Next, an embodiment of the present invention will be described in detail including a more specific data structure and a flowchart with reference to FIGS. 11 to 25 in addition to FIGS. 11 and 12 show a file structure diagram of the disk access conversion table and an example thereof. FIGS. 13 and 14 show a file structure diagram of the disk access information table and an example thereof. FIG. 15 shows a flowchart of the file transfer program for file transfer. 16 to 22 show flowcharts in the respective file transfer processes. 23 to 25 show data arrangement diagrams before and after correction of each disk access information table.

  Referring to FIG. 7, the system of this embodiment includes a client (100), a server (200), and a disk device (300). The client (100) and server (200) are connected via a LAN, and the client (100), server (200) and disk device (300) are connected via a SAN.

  The relationship between the server having the system configuration shown in FIG. 7 and the disk device is the same as that shown in FIG. Assume that the server (200) includes two servers, a transfer source server (file server 1 in FIG. 1) and a transfer destination server (file server 2 in FIG. 1). Assume that the disk device (300) includes two disk devices: a transfer source disk device (job execution data disk device in FIG. 1) and a transfer destination disk device (normal data disk device in FIG. 1). Furthermore, it is assumed that these servers and disk devices are connected to each other. These servers and disk devices may have the roles of transfer source and transfer destination reversed.

  The client (100) includes a virtual file transfer program (101), a shared file system client (102), a LAN access function (111), and a disk driver function (110).

  The virtual file transfer program (101) is a user mode application for transferring a virtual file, and represents a virtual file transfer function (virtual file transfer means) itself controlled based on the virtual file transfer program. The virtual file to be transferred is read from the transfer source server to the file transfer memory area (109) and then written to the transfer destination server.

  The shared file system client (102) includes a shared file system client function (103), a virtual file creation request function (104), a virtual file deletion request function (105), a data area reservation request function (106), a disk access conversion function ( 107) and a disk access conversion table (108). Various functions for manipulating files and virtual files provided by the shared file system (102) are disclosed to user lands as system calls.

  The shared file system client function (103) is a function (means) for requesting the shared file system server (201) to perform normal operations such as files and directories. The virtual file creation request function (104) performs a process of requesting the shared file system server (201) to create a virtual file and a process of adding a record to the disk access conversion table (108). The virtual file deletion request function (105) performs a process of requesting the shared file system server (201) to delete a virtual file and a process of deleting a record from the disk access conversion table (108). The data area securing request function (106) requests the shared file system server (201) to secure a file data storage area. The disk access conversion function (107) is a function for correcting the disk access information table received by the shared file system client (102) based on the disk access conversion table (108).

  The disk access conversion table (108) is a table for correcting gaps between files included in a disk access information table described later in units of sectors. The conversion table has information before and after correction of the offset and data size of the data transfer memory area (109) of each file constituting the virtual file. As shown in FIG. 11, in the disk access conversion table (108), the file size, the file correction size, the file offset, and the file correction offset are collected as one line for each file constituting the virtual file. The file correction size is a size obtained by correcting the file size in units of sectors. The file correction offset is obtained by correcting the read position of the file offset memory, which is the read position of the memory, in units of sectors.

  These lines are collected and stored in one record by the file handler of the virtual file. One disk access conversion table (108) is generated for each process of the virtual file transfer program (101). FIG. 10 is a data layout diagram before and after the gap correction. FIG. 12 is an example of a disk access conversion table showing the data arrangement after the correction. With the file correction size and file correction offset, the size and offset of each file are corrected to an integral multiple of the sector size.

  The disk driver function (110) inputs / outputs data between the file transfer memory area (109) and the disk device (300) using a disk access information table described later. The disk access information table is a table for disk access showing the correspondence between the file storage location in the disk device (300) and the memory offset of the file transfer memory area (109). As shown in FIG. 13, the disk access information table includes a memory offset, which is an offset of the file transfer memory area (109), a data size for input / output (an integral multiple of the sector), and a file name in the disk device (300) The disk position representing the storage position is stored in one line. FIG. 14 is an example of a disk access information table corresponding to the data arrangement before correction in FIG. For example, in the data arrangement before correction in FIG. 10, when the data of the last file in the virtual file is distributed and stored in different positions in the disk device, the row of the disk access information table is 3 in FIG. Created separately as in lines 4 and 5.

  The LAN access function (111) performs communication of control information via the LAN between the shared file system client (102) and the shared file system server (201).

  The server (200) includes a shared file system server (201), a file system (204), a disk driver function (210), and a LAN access function (211).

  The shared file system server (201) includes a shared file system server function (202) and a disk access information table integration function (203). The shared file system server function (202) is a function for notifying the file system (204) of an operation request from the shared file system client (102) and returning the result to the client. The disk access information table integration function (203) has a function of collecting disk access table information tables described in Patent Document 2 in units of disks.

  The file system (204) includes a file management function (205), a virtual file creation function (206), a virtual file deletion function (207), a data area securing function (208), and a file management table (209). The file management function (205) is a function for executing normal operations such as files and directories based on the file management table (209). The virtual file creation function (206) is a function for registering a virtual file in the file management table (209). The virtual file deletion function (207) is a function for deleting a virtual file from the file management table (209). The data area securing function (208) is a function for securing a data storage position for a specified size in a file of the file management table (209). As shown in FIG. 8, the file management table (209) manages a list of various attributes and data storage positions of each file.

  The disk driver function (210) and the LAN access function (211) have the same functions as those of the client (100) having the same name.

  The disk device (300) is accessed from the disk driver function (110, 211), and inputs and outputs data at the memory offset and disk position specified in the disk access information table.

  Next, the operation of the embodiment of the present invention will be described in detail with reference to the system configuration diagram of FIG. 7 and the flowcharts of FIGS. In the following, numbers (A1 to H13) starting with alphabets represent processing steps of the flowchart. 15 is a file transfer program, FIG. 16 is a virtual file transfer process, FIG. 17 is a virtual file creation request function, FIG. 18 is a virtual access request function, FIG. 19 is a virtual file deletion request function, and FIG. 20 is a virtual file creation. 21 shows a disk virtual access conversion table record creation request process, and FIG. 22 shows a flowchart of the disk access conversion function.

(A, virtual file transfer program (FIG. 15))
FIG. 15 shows a processing flow by the virtual file transfer program. The virtual file transfer program (101) receives the name list of files to be transferred as a virtual file and the size of the file transfer memory area (109) and starts processing. The designated files are collected into virtual files for each number that fits in the file transfer memory area (109) and transferred in order.

  1. Perform initialization. The transfer list is a name list of files to be transferred as one virtual file, and the size list is a list of file sizes of these files. The transfer list and size list are initialized to be empty (step A2).

2. The following processing is performed on all elements of the name list received as an argument (step A3).
2-1. Extract one file name from the name list (step A4) and obtain the file size from the file system (204) of the server (200) via the shared file system client function (103) (step A5). .

  2-2. The file size obtained above, the total value of the elements of the size list, and the size of the file transfer memory area (109) are compared (step A6). If the total value of the file size and the size list element is smaller than the file transfer memory area (109), the file size is added to the size list and the file name is added to the transfer list (step A9). Further, returning to step A3, steps A4, 5, and 6 are repeated. If the total value of the file size and the size list element is larger than the file transfer memory area (109), a virtual file transfer process (detailed flowchart B) is performed using the held size list and transfer list as arguments (step B). A7). Thereafter, the size list and transfer list are reset, and the size list and transfer list are initialized with the current file size and file name (step A8). Further, returning to step A3, steps A4, 5, and 6 are repeated.

  It is most efficient when the size of the virtual file to be transferred is slightly smaller than the size of the file transfer memory area (109). Therefore, the step of extracting the file name from the name list and adding the file size is repeated. The size added in this repetition step becomes larger than the size of the file transfer memory area (109). In this case, the files accumulated up to the previous one are connected as one virtual file and transferred. In this way, data can be efficiently transferred as a virtual file having a size slightly smaller than the size of the file transfer memory area (109).

  3. If the transfer list is not empty (step A10), the virtual file transfer process (detailed flowchart B) is performed using the transfer list and the size list as arguments (step A11). This step is a transfer process when there is no name list in step A3 but a transfer list remains in step A9. If the transfer list is empty, the processing step by the virtual file transfer program is terminated.

(B, virtual file transfer process (FIG. 16))
The virtual file transfer process shown in FIG. 16 is a subroutine of the file transfer program (101), and is called with the transfer list and size list as arguments in the file transfer program flowchart (steps A7 and A11). The virtual file is read from the transfer source server into the file transfer memory area (109) (steps B1 to B4), and the area of each file is secured in the transfer destination server (steps B5 to B8). A virtual file is written (steps B9 to B11).

  1. Using the transfer list as an argument, the virtual file creation request function (104) (details are flowchart C) is executed for the transfer source server, and the file descriptor of the virtual file is received (step B2).

  2. Using the file descriptor of the virtual file received from the transfer source server as an argument, execute an access request (details flowchart D) of the shared file system client function (103) to the transfer source server, and execute the file transfer memory The virtual file data is read into the area (109) (step B3).

  3. Using the file descriptor of the virtual file received from the transfer source server as an argument, execute the virtual file deletion request function (105) (details are flowchart E) to the transfer source server to delete the virtual file (step B4).

4. The following processing is performed on all elements of the transfer list (step B5).
4-1. Extract a file name from the transfer list and one file size from the size list (step B6).
4-2. Create a file with the above file name on the transfer destination server via the shared file system client function (103), and obtain a file descriptor (step B7).
4-3, using the file descriptor and file size as arguments, execute the data area securing request function (106) to the transfer destination server (step B8). The shared file system server function (202) of the transfer destination server executes the data area securing function (208) to secure a data storage area for the designated file size (step B8 '). Here, only the area is reserved and no data is written. Steps B5 to B8 are repeated for all elements in the transfer list.

  5. Similarly to step B2, the virtual file creation request function (104) (details are flowchart C) is executed on the transfer destination server with the transfer list as an argument, and the file descriptor of the virtual file is received (step B9).

  6. In the same manner as in step B3, the shared file system client function (103) access request (details are flowchart D) to the transfer destination server using the file descriptor of the virtual file received from the transfer destination server as an argument. Execute and write the virtual file data to the transfer destination server (step B10).

  7. In the same manner as in step B4, the virtual file deletion request function (105) is executed on the transfer destination server with the file descriptor of the virtual file received from the transfer destination server as an argument. The virtual file is deleted (step B11).

(C, virtual file creation request function (FIG. 17))
The virtual file creation request function (104) is called with the transfer list as an argument in the virtual file transfer flowchart (step B2) and (step B9).
1. Specify the transfer list received as an argument, and send a virtual file creation request to the shared file system server function (202) (step C2). The shared file system server function (202) executes the virtual file creation function (206) (details are flowchart F) to create a virtual file, and lists the file handler corresponding to the virtual file and the size of the file included in the transfer list Is returned (step C2 ′).
2. The disk access conversion table entry creation process (details are flowchart G) is executed with the file handler of the virtual file and the list of file sizes as arguments (step C3).
3. Return the file descriptor for the file handler of the virtual file to the caller (step C4).

(D, file access request function (FIG. 18))
The file access request function is a function obtained by extending the disk access function of Patent Document 1 so as to correct the disk access information table. The file access request function is called as a part of the shared file system client function (103) by using the virtual file transfer process flowchart (steps B3 and B10) with the file descriptor of the virtual file as an argument.

  1. A file handler corresponding to the file descriptor received as an argument is designated, and a virtual file access request is transmitted to the shared file system server function (202) (step D2). The shared file system server function (202) acquires the file management number from the file handler, and acquires the data storage location information of the virtual file from the file management function (205) using the file management number as a key (step D3). The disk access information table integration function (203) summarizes the data storage location information in units of disks, creates a disk access information table, and returns it to the client (100) (step D4).

  2. Using the file handler corresponding to the file descriptor received as an argument as a key, obtain a record from the disk access conversion table (108). The disk access conversion function (107) is executed with the disk access information table and this record as arguments, and the disk access information table is corrected (details are in flowchart H) (step D5).

  3. The disk driver function (110) is executed with the corrected disk access information table as an argument, and data is input / output between the disk device (300) and the file transfer memory area (109) (step D6).

(E, virtual file deletion request function (FIG. 19))
The virtual file deletion request function (105) is called with the file descriptor of the virtual file as an argument in the flowchart of the virtual file transfer process (steps B4 and B11).
1. A file handler corresponding to the file descriptor received as an argument is designated, and a virtual file deletion request is transmitted to the shared file system server function (202) (step E2). The shared file system server function (202) executes the virtual file deletion function (207) to delete the virtual file (step E2 ′).
2. Using the file handler as a key, delete the record from the disk access conversion table (108) (step E3).

(F, virtual file creation function (FIG. 20))
The virtual file creation function (206) is called with the transfer list as an argument in the flowchart (step C2 ′) of the virtual file creation request function.
1. One entry with “virtual file” set as the file type is created (step F2), and the size list is initialized to be empty (step F3).

2. The following processing is performed on all elements of the transfer list (step F4).
2-1. One file name is extracted from the transfer list (step F5), the data storage position information of the file is acquired from the file management table (209), and added to the entry (step F6).
2-2. The file size is acquired from the file management table (209) with the file name and added to the size list (step F7). Steps F4 to F7 are repeated for all elements in the transfer list.
3. The total size of the data storage location information is set to the file size of the entry (step F8).
4. An entry is added to the file management table (209) (step F9).
5. Return the virtual file and the size list to the caller (step F10).

(G, disk access conversion table record creation process (FIG. 21))
The disk access conversion table record creation process is a subroutine of the virtual file creation request function (104), and is called with the file handler and size list of the virtual file as arguments in the flowchart (step C3) of the virtual file creation request function. In the disk access conversion table record creation process, a record for correcting the disk access information table from the data arrangement before correction in FIG. 10 to the data arrangement after correction is created.

  In the pre-correction disk access information table, as shown in the pre-correction data arrangement of FIG. 10, the memory offset is specified so that each file constituting the virtual file is arranged with an offset rounded up in units of blocks. . For this reason, the offset (file offset) in the memory of each file before correction is obtained by sequentially adding values obtained by rounding up the file size by the block size (step G9). Further, the file offset of each file after correction (file correction offset) is set so that each file is arranged with an offset obtained by rounding up each file as shown in the data arrangement after correction in FIG. A value obtained by rounding up the size by the sector size is sequentially added and obtained (step G9). Also, the file correction size is calculated by rounding up the file size by the sector size (step G7). As described above, the file size, the file correction size, the file offset, and the file correction offset are calculated for each file, and stored together with the file handler of the virtual file (step G10).

  1. Queries the block size of the file system (204) of the server (200) via the shared file system client function (103) (step G2). The shared file system server function (202) acquires the block size of the file system from the file management function (205) and returns it to the client (100) (step G2 ').

2. Create one record in the disk access conversion table (step G3).
3. The file offset and file correction offset are initialized to 0 (step G4).

4. The following processing is performed on all elements of the size list (step G5).
4-1. One file size is extracted from the size list (step G6), and a value obtained by rounding up the file size by the sector size is set as the file correction size (step G7).
4-2: A file size, a file correction size, a file offset, and a file correction offset are added to the above record (step G8).
4-3: Add the value obtained by rounding up the file size to the block size to the file offset, and add the value obtained by rounding the file size up to the sector size to the file correction offset (step G9). Steps G5 to G9 are repeated for all elements in the size list.

  5. The file handler of the virtual file received as an argument is set in the above record and registered in the disk access conversion table (step G10).

(H, disk access conversion function (Fig. 22))
The disk access conversion function (107) is called with the records of the disk access information table and the disk access conversion table (109) as arguments in the flowchart of the virtual file access request function (step D5). The disk access conversion function performs the following four processes for each row of the disk access information table, and a new memory offset and data size corrected in units of sectors as in the corrected data arrangement of FIG. A disk access information table (hereinafter referred to as a post-conversion table) is created.

As the creation of the post-conversion table, the following four processes are largely performed.
(Process 1) A file to be accessed is determined in association with each row of the record of the disk access conversion table (108) (step H5).
(Process 2) If the file data does not exist, the line is deleted (step H6).
(Process 3) If no data exists behind (step H8), the data size is reduced in units of sectors (step H9).
(Process 4) The memory offset is corrected with the file correction offset in the disk access conversion table (step H10).

  Specifically, referring to FIGS. 12, 14, 23, 24, and 25, the last row of the disk access information table (the fifth row in FIG. 14, hereinafter referred to as the pre-conversion row) is the disk access conversion table. An example of conversion using the record (108) (FIG. 12, hereinafter referred to as a conversion record) will be described. As shown in FIG. 23, in the last row of the disk access information table, the memory offset is moved forward, and the data size is corrected to a value obtained by rounding up the size including a valid file in units of sectors (hereinafter referred to as correction). Called the converted line).

  First, (Process 1) is performed. When accessing a file having a pre-conversion line, the memory offset of the pre-conversion line indicates a memory area into which the file is read. The start position of the memory area from which the file is read is equal to the file offset value of the conversion record. Therefore, each line of the conversion record is examined in order from the smallest file offset, and the line immediately before the file offset larger than the memory offset of the line before conversion becomes the line of the conversion record corresponding to the file accessed by the line before conversion. (Step H5). As shown in FIG. 24, the current pre-conversion line corresponds to the last line in FIG. 12 because the memory offset of the pre-conversion line is larger than the file offset of the last line of the conversion record. Hereinafter, this corresponding line is referred to as a conversion line.

  Next, (Process 2) is performed. As shown in FIG. 24, the effective range of the file is up to “file offset of conversion line + file size” (1002). If the memory offset of the pre-conversion line is larger than (1002), the pre-conversion line does not include file data, so the conversion process for this line is stopped (yes in step H6). Since the memory offset (20480 bytes) of the row before conversion this time is equal to or less than the value of (1002) (12288 + 10000 = 22288 bytes), the processing is continued because the row includes file data.

  Next, (Process 3) is performed. If the file data does not exist after the pre-conversion line, the pre-conversion line is in a state of being accessed beyond the effective range of the file, and therefore, from “file offset of conversion line + file size” (1002) in FIG. In addition, “the memory offset of the row before conversion + the data size” (1005) is increased (yes in step H8). Since the value before (1005) (20480 + 4096 = 24576 bytes) is larger than the value of (1002) (12288 + 10000 = 22288 bytes), the pre-conversion line this time is a line in which no file data exists. Therefore, the work of truncating the portion where the data of the backward file does not exist in units of sectors is performed.

  As shown in FIG. 25, the memory offset of the line before conversion corresponds to which position in the file can be calculated by “memory offset of the line before conversion−file offset of the line of conversion” (1102). Further, since the file correction offset of the conversion line is a value obtained by rounding up the file size in units of sectors, the value obtained by correcting the data size of the line before conversion is “file correction size of conversion line− (memory of conversion line before conversion). (Offset-file offset of conversion line) "(1105) (step H9). A value obtained by correcting the data size of the previous conversion row is changed from (1105) to 2048 bytes (= 10240− (20480-12288)).

  Finally, (Process 4) is performed. The memory offset of the line before conversion is corrected by the file correction offset of the conversion line. As shown in FIG. 25, the head offset of the file is “file correction offset of converted line” (1103). Since the position in the file where the pre-conversion line is accessed can be calculated in (1102) as in (Process 3), the value obtained by correcting the memory offset of the pre-conversion line is “file correction offset of conversion line + (conversion (Memory offset of previous line−file offset of converted line) ”(1106) (step H10). The value obtained by correcting the memory offset of the previous conversion row is changed from (1106) to 15072 bytes (= 7680 + (20480-12288)).

  As described above, a post-conversion row in which the pre-conversion row is corrected is created and registered in the post-conversion table (step H11). In the case of the pre-conversion line this time, a post-conversion line of memory offset 15072 bytes, file size 2048 bytes, and disk position 5 is registered. The above processing is performed on all the pre-conversion rows, and a post-conversion table is returned (step H12).

  As described above, the present invention includes a function (called a virtual file function) that can handle a large number of small files as if they were one large file. When a large number of small files are transferred, first, the large number of files are set as one virtual file. By treating this virtual file as one large file and transferring it at once, the overhead of the control packet can be reduced and high-speed file transfer can be realized. By transferring a virtual file in which a plurality of files are combined into one in this way, input / output that is not separated in units of files can be realized, and file transfer can be performed at a higher speed than in units of files. This is effective when a large number of small files are transferred in a shared file system using SAN. It is also effective in file systems that can execute large-sized input / output.

  According to the present invention, the sizes of a large number of small files in the transfer source storage device are corrected in size from block units to sector units, and these size corrected files are combined into one virtual file that is concatenated and transferred to the transfer destination storage device. A data transfer method is obtained in which the apparatus transfers a file as a single file.

  In the transfer destination storage device of the present invention, the transferred virtual file can be restored to a small size file in units of blocks before correction, and can be divided and written.

  In the virtual file according to the present invention, the gap of the file in units of blocks before being concatenated so that the file size is in units of sectors can be within one sector. In addition, the virtual file can be managed by a file management table in which a file management number, a file type, a file size, and a list of data storage position information for each file before concatenation are saved as one entry. The virtual file can also correct the offset of the file before concatenation in units of sectors. Furthermore, the virtual file can be deleted after it is written to the file transfer memory area.

  According to the present invention, the file name list to be transferred is received by the transfer source server, and the virtual file creation function reduces the file size of the plurality of files from the block unit to the sector unit, and one virtual file obtained by concatenating these files. As a file, in accordance with read access, the created virtual file is written to the file transfer memory area, the transfer destination server receives the file name list to be transferred, and the data area allocation function stores the data area of the transfer destination disk device. A data transfer method is obtained in which a virtual file in the memory area for file transfer is written to the data area of the transfer destination disk device in accordance with the access request.

  Writing to the data area of the transfer destination disk device of the present invention can be restored to the transfer source file size and divided and written. Further, after the virtual file is written in the file transfer memory area, the virtual file created by the transfer source virtual file creation function can be deleted using the virtual file deletion function.

  As described above, the present invention has been described with reference to the embodiment, but the present invention is not limited to the above embodiment. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

  In the present invention, a large number of files are handled as one large virtual file and transferred at a time, thereby reducing the overhead of the control packet and realizing high-speed file transfer. This is effective when a large number of small files are transferred in a shared file system using SAN, or in a file system that can execute input / output of a large size. The present invention can be suitably used in a shared file system that performs high-speed file transfer via SAN in HPC.

It is a system configuration diagram for explaining staging / destaging. It is a block diagram of a network file system (NFS). It is a block diagram of a storage area network (SAN). It is a block diagram of a storage area network (SAN) that collects data storage locations. It is explanatory drawing which shows the effect of FIG. It is a block diagram of the virtual file which concerns on this invention. 1 is a system configuration diagram of a shared file system according to the present invention. FIG. 3 is a file management table according to the present invention. It is explanatory drawing explaining the data structure of a normal file and a virtual file. It is a data arrangement | positioning figure before and behind correction | amendment of the gap of a virtual file. It is a file block diagram of a disk access conversion table. It is an example of a disk access conversion table. It is a file block diagram of a disk access information table. It is an example of a disk access information table. It is a flowchart in the file transfer program of file transfer. It is a flowchart in the virtual file transfer process of file transfer. It is a flowchart in the virtual file creation request function of file transfer. It is a flowchart in the virtual file access request function of file transfer. It is a flowchart in the virtual file deletion request function of file transfer. It is a flowchart in the virtual file creation function of file transfer. It is a flowchart in the disk virtual access conversion table record creation process of file transfer. It is a flowchart in the disk access conversion function of file transfer. It is a data arrangement | positioning figure before and behind correction | amendment of the entry of a disk access information table. It is a data arrangement | positioning figure before and behind correction | amendment of the access position of a disk access information table. It is a data arrangement | positioning figure before and after correction | amendment of offset and size of a disk access information table.

Explanation of symbols

100 Client 101 Virtual file transfer program 102 Shared file system client 103 Shared file system client function 104 Virtual file creation request function 105 Virtual file deletion request function 106 Data area reservation request function 107 Disk access conversion function 108 Disk access conversion table 109 For file transfer Memory area 110 Disk driver function 111 LAN access function 200 Server 201 Shared file system server 202 Shared file system server function 203 Disk access information table integration function 204 File system 205 File management function 206 Virtual file creation function 207 Virtual file deletion function 208 Data area Secure function 209 File management table 210 Disk driver function 11 LAN access function 300 disk device

Claims (10)

  The size of a large number of small files in the transfer source storage device is corrected in size from block units to sector units, and these size corrected files are combined into one virtual file concatenated and stored in the transfer destination storage device as one file. A data transfer method comprising transferring a file.   2. The data transfer method according to claim 1, wherein in the transfer destination storage device, the transferred virtual file is restored to a small size file in units of blocks before correction, and is divided and written.   3. The data transfer method according to claim 1 or 2, wherein the gap of the file of the block unit before the virtual file is concatenated so that the file size becomes a sector unit is reduced within one sector. .   The virtual file is managed by a file management table in which a file management number, a file type, a file size, and a data storage location information list for each file before concatenation are stored as one entry. Item 4. The data transfer method according to any one of Items 1 to 3.   5. The data transfer method according to claim 1, wherein the virtual file is corrected in units of sectors with an offset of the file before being concatenated.   6. The data transfer method according to claim 1, wherein the virtual file is deleted after being written in the file transfer memory area. The transfer source server receives the list of file names to be transferred, and the virtual file creation function reduces the file size of multiple files from block units to sector units, consolidates these files into one virtual file, and follows read access , Write the created virtual file to the file transfer memory area,
The transfer destination server receives the list of file names to be transferred, the data area securing function secures the data area of the transfer destination disk device, and the virtual file in the memory area for file transfer is transferred to the transfer destination disk device according to the access request. A data transfer method comprising writing to a data area.   8. The data transfer method according to claim 7, wherein the writing to the data area of the transfer destination disk device is restored to the transfer source file size and is divided and written.   9. The virtual file created by the transfer source virtual file creation function after the virtual file is written in the file transfer memory area is deleted using the virtual file deletion function. The data transfer method described in 1.   A data transfer method using the data transfer method according to claim 1.

Images