JP5664318B2 - Virtual file system - Google Patents

Description

  The present invention relates to a virtual file system, and more particularly to a virtual file system that does not require physical storage.

  In addition to being used as a permanent storage means for data, the file system is also widely used for temporarily using data. An example of temporary use is a test file used when evaluating a storage system.

  In the evaluation of the storage system, backup performance and restore performance of the evaluation target storage system are evaluated using backup software. The outline is shown in FIG.

  First, before performing the evaluation, a test file 711 is created in the existing storage 71 using the test file creation software 75 of the evaluation machine 70. In the backup performance evaluation, the test file 711 created on the existing storage 71 is backed up as a test file 74 to the evaluation target storage system 73 using the backup software 72 of the evaluation machine 70. To do. Further, in the evaluation of the restore performance, the test file 74 backed up in the evaluation target storage system 73 is restored as the test file 712 in the existing storage 71 using the backup software 72 of the evaluation machine 70, and the evaluation machine This is performed by comparing the test file 711 before backup and the test file 712 after restoration using the file comparison software 76 of 70.

  On the other hand, there is a technique described in Patent Document 1 as a technique related to the present invention. In the technique described in Patent Document 1, a video signal output from a video signal generator is transferred to a control device and compressed by a compression encoding method such as JPEG to form compressed video data. Transfer and write to storage device. Also, the compressed video data is read from the external storage device, transferred to the control device, decompressed and decoded, and the reproduced image is displayed on the image display device. Since a reproduced image based on video data that has been written to and read from the external storage device is displayed on the image display device 4, the random data can be verified intuitively.

  In addition, Linux has a file that can read random data called / dev / urandom. This eliminates the need for a physical storage system and eliminates the time for file generation.

JP-A-7-210990

  The storage system evaluation method shown in FIG. 7 requires a storage 71 that stores a test file 711 that is data before backup and a test file 712 that is data after restoration. Since the capacity of the storage system is increasing year by year, the capacity of the test files 711 and 712 is also increased. Further, when the evaluation target storage system 73 has a data compression function and a duplicate data deletion function, it is necessary to generate a large number of various test files 711 in order to perform the evaluation. For this reason, a large capacity system is required as the existing storage 71.

  On the other hand, the evaluation method described in Patent Document 1 does not require an existing storage for storing test data, but cannot store original data for comparison with data read from an external storage device to be evaluated. Even if the verification of random data can be performed intuitively, it cannot be performed objectively.

  In addition, the file / dev / urandom that can read random data of Linux is different from a normal file, and data is read infinitely, so it cannot be backed up by backup software. In addition, since the random number data is not reproducible, the restored files cannot be compared.

  An object of the present invention is to provide a virtual file system that solves the above-mentioned problem, that is, the problem that a large-capacity storage is required in order to be able to read a large amount of the same data at any time. is there.

A virtual file system according to an aspect of the present invention includes:
Storage means for storing the file type;
Data generation means for generating file data using a pseudo-random number generation algorithm from the file seed;
The system includes a file system access unit that receives a read request for a file and returns an access response including the data generated by the data generation unit.

  By using the virtual file system of the present invention, a large amount of the same data can be read at any time even without a large-capacity existing storage.

It is a block diagram of a 1st embodiment of the present invention. It is a block diagram of the 2nd Embodiment of this invention. It is operation | movement explanatory drawing at the time of the test file creation in the 2nd Embodiment of this invention. It is operation | movement explanatory drawing at the time of backing up a test file to evaluation object storage in the 2nd Embodiment of this invention. It is operation | movement explanatory drawing at the time of restoring backup data in the 2nd Embodiment of this invention. It is explanatory drawing of the data generation by the data generation means in the 2nd Embodiment of this invention. It is a block diagram of the general system which evaluates storage.

Next, embodiments of the present invention will be described in detail with reference to the drawings.
[First embodiment]
Referring to FIG. 1, the virtual file system 10 according to the first exemplary embodiment of the present invention includes a storage unit 11, a file system access unit 12, and a data generation unit 13.

  The storage unit 11 stores a file type 111. The file type 111 is small-capacity data such as several sequences.

  The file system access means 12 is an interface between the virtual file system 10 and the outside. The file system access means 12 accepts a read request 14 from an external program (not shown) and returns a response 15 in response thereto.

  The data generation means 13 has a function of generating file data from the file seed 111 using a pseudo-random number generation algorithm. More specifically, the data generation unit 13 includes a block information calculation unit 131, a block seed generation unit 132, a block data generation unit 133, and an access target data generation unit 134.

  When the block information calculating unit 131 receives the offset specified by the read request 14 from the file system access unit 12, the block information calculating unit 131 determines the access target by the read request 14 from the offset and the fixed-length block size defined in advance by the system. A function of calculating the file block number and the intra-block offset.

  The block seed generation unit 132 uses the non-recursive pseudo-random number generation algorithm from the file seed 111 and the file block number calculated by the block information calculation unit 131, to generate a block corresponding to the file block number. Has the function of generating seeds. The block seed is composed of a small amount of data like the file seed 111.

  The block data generation means 133 uses the recursive pseudo random number generation algorithm from the block seeds generated by the block seed generation means 132 to obtain the data that is the contents of the file block corresponding to the block number of the file. It has a function to generate the block size.

  The access target data generating unit 134 generates, as access target data, data determined by the intra-block offset and the data size specified by the access request 14 among the data of the file block generated by the block data generating unit 133 Have

  Next, a virtual file method according to the present embodiment, that is, an operation for emulating a file will be described.

  When the file system access unit 12 receives a read request 14 for a file from an external program (not shown), the file system access unit 12 notifies the data generation unit 13 of the file type 111 and the offset and data size specified by the read request 14.

  The block information calculation unit 131 of the data generation unit 13 uses the offset received from the file system access unit 12 and the fixed-length block size defined in advance by the system to determine the file block number and block to be accessed by the read request 14. The inner offset is calculated. Next, the block seed generation unit 132 uses the non-recursive pseudo-random number generation algorithm to calculate the file block number from the file seed 111 and the file block number calculated by the block information calculation unit 131. Generate the corresponding block seed. Next, the block data generation unit 133 uses the recursive pseudo-random number generation algorithm from the block seed generated by the block seed generation unit 132, using the contents of the file block corresponding to the block number of the file. Generate some data for the block size. Next, the access target data generation unit 134 sets the data determined by the offset within the block and the data size specified by the access request 14 among the data of the file block generated by the block data generation unit 133 as the access target data. Generate and notify the file system access means 12.

  The file system access unit 12 returns the response 15 including the access target data notified from the data generation unit 13 to the request source of the read request 14.

  As described above, the virtual file system 10 of this embodiment can read a large amount of the same data at any time without a large-capacity storage. The reason is that the data generation means 13 for generating data of the same file each time using the pseudo random number generation algorithm from the file seed 111 is provided.

  In addition, the virtual file system 10 of the present embodiment can read random data. The reason is that the data generation unit 13 calculates the number of the file block to be accessed and the offset within the block from the offset specified by the read request 14 and the block size, and calculates the type of the block from the file block number. This is because block data is calculated from this block seed, and access target data is extracted from the calculated block data based on the intra-block offset and the data size.

  Furthermore, in the virtual file system 10 of this embodiment, generation of block seeds is performed using a non-recursive pseudo-random number generation algorithm, but when generating block data from block seeds, a recursive pseudo-random number generation algorithm is used. Therefore, it is possible to speed up the reading of random data compared to the case of using a non-recursive pseudorandom number generation algorithm.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.

[Overview of this embodiment]
The virtual file system according to the present embodiment does not store the actual data of the file in the physical storage, but stores the “file type” necessary for reproducing the data instead of the file data. FIG. 2 shows a configuration diagram of the evaluation environment of the storage system using the virtual file system of this embodiment. Although described in detail later with reference to FIG. 2, the following operations are generally performed.
-When creating a file, create only the file type, not write the actual data to physical storage.
・ When reading file data, instead of reading data from physical storage, calculate the file type and reproduce the data.
• When writing data to a file, do not write the data to physical storage, but compare it with the data that is calculated and reproduced from the file type.
-Other operations (such as directory operations and file attribute operations) are processed in the same manner as a normal file system.

  The outline of each file operation will be described below.

  First, test file creation will be described. When creating a test file, the test file is generally created by writing random number data into the file using test file creation software, etc., but this method takes a lot of time as the file size increases. Will start. In the present embodiment, the file data itself is not written. Instead, it holds a “file type” for reproducing the file data by calculation. The file seed is used as a seed for the pseudo-random algorithm, for example, to generate data. This file type is given to the virtual file system using the file name at the time of file creation. Also, the size of the file to be created is transmitted to the virtual file system in the same manner. As an example, if a file is created with the file name “randomfile_128GB_12345”, a file with a file type of 12345 and a file size of 128 GB is created. The file is 128GB at the moment of creation, even if nothing is written.

  Next, a case where file data is read will be described. In a normal storage, when reading a file, data is read from the storage. On the other hand, in the present embodiment, data is calculated from the file type using, for example, a pseudo random number algorithm. The virtual file system calculates data and sends it to the request source each time there is a read request. In addition, the virtual file system needs to be capable of random access in the same way as a normal file system. For example, if a pseudo-random algorithm is used for data generation, a non-recursive pseudo-random algorithm is required to perform random access. Here, recursive means that it depends on the previous state. That is, recursive pseudo-random numbers can only be accessed sequentially from the beginning, and non-recursive pseudo-random numbers can be accessed randomly. However, since the nonrecursive pseudorandom algorithm is generally slower than the recursive pseudorandom algorithm, it is preferable to use the recursive pseudorandom algorithm in terms of calculation cost. Therefore, the data generation method is devised as described below so that random access is possible even with a recursive pseudorandom algorithm.

  The file is divided into fixed-length blocks, and the block data is calculated using “block seed” as a pseudo-random seed. The block seed is calculated from the file seed and the block serial number using a non-recursive pseudo-random number. Since only the blocks containing the necessary data need only be calculated by this method, only the block seed calculation is required for the slow non-recursive pseudorandom number, and the data itself is calculated using the fast recursive pseudorandom number. Since it can be used, calculation cost is improved.

  Next, a case where data is written to a file will be described. In normal storage, when writing a file, data is written to the storage. On the other hand, in the present embodiment, since the file data is not stored on the storage and the file data is calculated from the file type, the data is not written to the storage. On the other hand, in the evaluation of the storage system, when the test file is restored to the virtual file system by the backup software, the file is written, and it is confirmed that this file will not be garbled later compared to the test file before backup. Used for. Instead, in this embodiment, when data is written to the storage, the data created at the time of backup from the “file type” (= data to be written by restoration) and the data actually written are compared. Therefore, a file comparison operation is also performed simultaneously with the restore operation.

  Finally, other operations will be described. Operations such as file deletion, name change, attribute change, and directory creation all function in the same way as a normal file system. However, regarding file deletion, instead of deleting data from the storage, the file type is deleted. In addition, regarding the file name change, the file size and the file type are not affected by the changed name. Thus, when creating a file, a special name is given to specify the file size, file type, etc., but after creation, it can be changed to a free file name.

[Description of configuration]
The configuration of this embodiment will be described in detail with reference to FIG. FIG. 2 shows a configuration example when an evaluation equivalent to FIG. 7 is performed in this embodiment.

  The evaluation machine 20 is a computer that operates backup software 22 that evaluates the evaluation target storage system 23, and is also a computer that operates the virtual file system 21 according to the present embodiment.

  The virtual file system 21 is a virtual file system that can generate and verify random data, and includes a file system access unit 210, a file creation unit 211, a data generation unit 212, a data verification unit 213, and a storage area 215.

  The backup software 22 is software that constitutes an evaluation unit that performs backup and restore for the evaluation target storage system 23.

  The evaluation target storage system 23 is a storage system to be evaluated.

  The test file creation software 25 is software for creating a test file. The test file creation software 25 may be any software that can create a file by specifying a file name. For example, the POSIX touch command can be used.

  The file system access unit 210 is a unit serving as a window for accessing the virtual file system 21. The backup software 22 accesses the virtual file system 21 through the file system access means 210.

  The file creation unit 211 is a unit that receives a file name from the file system access unit 210 and creates corresponding file information (file name, file size, file type).

  The data generation unit 212 is a unit that receives the file type, the size and offset of the data to be generated from the file system access unit 210, generates data using a data generation algorithm, and returns the generated data.

  The data verification unit 213 receives the file type, data, its size, and offset from the file system access unit 210, compares the data generated using the data generation unit 212 with the received data, and returns the comparison result. Means.

  The storage area 215 is a storage area for storing the directory structure, file type, attribute, size, and the like. This may be main memory or auxiliary memory.

[Description of operation]
Next, the operation of this embodiment will be described with reference to FIGS. 3 to 6, the same reference numerals as those in FIG. 2 denote the same parts.

  First, test file creation (file creation operation in the virtual file system) will be described in detail with reference to FIG.

  First, the test file creation software 25 issues a file creation request to the file system access means 210 (S1 in FIG. 3). At this time, the file name of the file to be created is passed. This file name includes file size and file type information.

  Next, the file system access unit 210 passes the received file name to the file creation unit 211.

  The file creation unit 211 extracts the file size and the file type from the received file name as follows. From the beginning of the file name to the first underscore (_) represents the creation method of the file data, the next underscore is a character string representing the file size, and the rest is a character string representing the file type. For example, if the file name is “ramdomfile_128GB_12345”, a pseudo random number is used as the file data creation method, the file size is 128 GB, and the file type is 12345. Information on these files is returned to the file system access means 210.

  The file system access unit 210 associates the returned file information with the file name and stores them in the storage area 215.

  The file system access unit 210 performs processing for opening the file, and returns a file handle to the test file creation software 25. The file opening process and the file handle here are the same as those in a general file system, and thus description thereof is omitted.

  The test file creation software 25 confirms that the file has been created normally and closes the file. In a normal file system, the size at the time of file creation is 0, but in the present embodiment, a file of a specified size (for example, 128 GB) is created. The process for closing the file is the same as that of a general file system, and thus the description thereof is omitted.

  Next, a backup (reading from the virtual file system) operation will be described in detail with reference to FIG. It is assumed that a file for backup is previously opened on the virtual file system 21 (having a file handle). The process for opening a file and the file handle are the same as those of a normal file system.

  First, the backup software 22 issues a data read request to the file system access unit 210 (S11 in FIG. 4). At this time, in order to identify the data to be read, the file handle, the data offset, and the data size are passed. Data offset is the length from the beginning of the file to the data you want to read.

  Next, the file system access unit 210 reads the file information (file type, file size) and the like corresponding to the received file handle from the storage area 215 (S12 in FIG. 4).

  The file system access unit 210 passes the read file information and the offset and size of the data received from the backup software 22 to the data generation unit 212 (S13 in FIG. 4).

  The data generation unit 212 generates the requested data from the received file type, offset, and size, and returns it to the file system access unit 210.

  The file system access unit 210 returns the returned data to the backup software 22.

  The backup software 22 writes the data returned to the evaluation target storage system 23 for backup as a test file 24.

  Next, the operation of the restore (write to virtual file system) operation will be described in detail with reference to FIG. It is assumed that a file for restoration is opened in advance on the virtual file system 21 (having a file handle).

  First, the backup software 22 reads data from the test file 24 of the evaluation target storage system 23 for restoration (S21 in FIG. 5).

  Next, the backup software 22 issues a write request for the read data to the file system access unit 210 (S22 in FIG. 5). At this time, the file handle, the data to be written, the data offset, and the data size are passed.

  Next, the file system access unit 210 reads the file information (file type, file size) and the like corresponding to the received file handle from the storage area 215 (S23 in FIG. 5).

  The file system access unit 210 passes the read file information, the data received from the backup software 22, the offset, and the size to the data verification unit 213 (S24 in FIG. 5).

  The data verification unit 213 passes the received file type, offset, and size to the data generation unit 212 (S25 in FIG. 5).

  The data generation unit 212 generates file data from the received file type, offset, and size, and returns them to the data verification unit 213.

  The data verification unit 213 compares the data received from the file system access unit 210 with the data returned from the data generation unit 212, and returns the comparison result to the file system access unit 210.

  The file system access unit 210 returns the returned data to the backup software 22.

  Since the backup software 22 receives the verification result as writing success or failure, if the verification result data is different, the restoration fails.

  Next, data generation will be described with reference to FIG.

  FIG. 6 is a diagram showing the relationship between file types, blocks, and data. Here, the block is a piece of data obtained by dividing the file data at a fixed size, and the block number is a block number represented by a serial number with the first block of the file being 0. When the file offset is divided by the block size, the quotient becomes the block number, and the remainder becomes the offset in the block.

  The block seed is calculated from the file seed and block number using a non-recursive pseudorandom number generation algorithm. For example, the block seed 612 is calculated from the file seed 61 and the block number (here 2).

  The block data is calculated from the seed of the block using a recursive pseudo random number generation algorithm which is a high speed pseudo random number algorithm.

[Description of effects]
The effect of this embodiment will be described.

  The first effect is that a storage for storing test file data becomes unnecessary. The reason is that the file holds only the file type, does not store the data, and is generated from the file type as needed. Thus, since it is not necessary to store file data, expensive high-performance storage for storing huge test file data becomes unnecessary.

  The second effect is that the time required for creating the test file can be greatly shortened. The reason is that when creating a file, only the file type is extracted from the file name and saved, the data itself is not created, and the data is not written to the storage.

  The third effect is that high-speed reading / writing is possible, and high-speed (and expensive) storage is not required even when high-speed access is required. This is because the data is not stored on the storage, the calculation cost for generating the data is sufficiently low, and the data can be generated at high speed.

  The fourth effect is that the comparison operation of the test file after the restoration is not necessary in the restoration evaluation of the storage system. The reason is that at the time of restoration, it is confirmed whether or not the data is garbled when data is written to the virtual file system of this embodiment.

[Other embodiments]
The present invention is not limited to the above-described embodiments, and various additions and modifications as exemplified below are possible.

  As another embodiment of data generation, when generating random data using pseudo-random numbers, not only pseudo-random seeds as “file seeds” but also “file compressibility ratio” or “file deduplication” “Ratio” or the like may be designated. Originally, pseudo-random data can hardly be compressed, but when creating a block, the data is not filled with random data, but mixed with data padded with zeros according to the compressible ratio, and the file with the specified compression rate Can be generated. Further, by mixing blocks having the same data as other blocks in accordance with the deduplication ratio, data that can be deduplicated at a specified deduplication rate can be generated. These are particularly effective for evaluating high-function storage systems having functions such as compression and deduplication.

  Although the method of generating random data has been described as an embodiment of data generation, an algorithm for generating a test file that reproduces a certain data format (for example, an image file, a text file, mail data, etc.) is used. You can also. As an example of an image file, an algorithm for drawing a reproducible figure from a file type and creating the image in accordance with an image data format can be considered. As another example, as an example of a text file, an algorithm for extracting and concatenating a character string corresponding to a random number from a character string dictionary for various languages or the like to a random number sequence generated from a file seed can be considered. These are effective for evaluation according to the actual usage of the storage system.

  When multiple file creation algorithms are possible as described above, an algorithm can be selected at the time of file creation using a character string ("randomfile" in the above-described embodiment) for identifying the data creation method at the beginning of the file name. . For example, “compressableramdomfile” for a compressible random number file, “pngfile” for a png image file, and the like. In a system that can use the file extension, an algorithm can be selected using the file extension. For example, “.txt” for a text file.

  As another embodiment of the file creation, a method may be used in which the file creation means does not extract the file type or the like from the file name but sets the file type when special data is written in the file. For example, when data with the content “randomfile: size = 128 GB, seed = 12345” is written at the beginning of the file, the size of the file is changed to 128 GB and the seed is changed to 12345. This method has the advantage that the file size and file type can be set without depending on the file name at the time of creation.

  As another embodiment of creating a file, it may be possible to specify the number of files to be created as a file name. For example, when a file “randomfile_128GB_12345_15000” is created, 15000 files from “randomfile_128GB_12345_00001” to “randomfile_128GB_12345_15000” are created. This is the case when creating many files such as tens of thousands of files, and when it is not necessary to specify detailed parameters (file size and seed) for each file, it is necessary to call the file system file creation operation only once. There is an advantage of disappearing.

  As another embodiment of creating a file, an example of reproducing a specific directory structure by creating a directory with a specific name can be considered. For example, when a directory called “maildir_1000_200” is created, directories “Inbox” and “Sent” are automatically created in it, and 1000 text files simulating the contents of the mail in each directory are created in “Inbox”. , "Sent" is created 200 pieces. This has an advantage that a data file group (a pseudo file) used by a specific system can be generated at a time.

  Regarding the change of the file name, the above embodiment has been described as being the same as the normal file system. This means that even if the file name is changed, the file information (file size and file type) is retained without being changed. However, as another embodiment, in changing the file name, it is conceivable to call the file creation unit again and change the file information so as to match the new file name. In this case, for example, when the file “randomfile_128GB_12345” (file size 128 GB, file type 12345) is changed to the name “randomfile_256GB_67890”, the file size is automatically changed to 256 GB and the file type is changed to 67890.

  As another embodiment relating to the restore (write to the file system) operation, it is conceivable that the received data is simply discarded as it is and the write success is returned without performing operations such as data comparison. This can be used when a comparison operation is unnecessary and more performance is required, such as when reading performance from an evaluation target storage system is desired.

  The present invention can be applied to uses such as evaluation of storage systems and backup software.

DESCRIPTION OF SYMBOLS 10 ... Virtual file system 11 ... Storage means 12 ... File system access means 13 ... Data generation means

Claims (6)

Storage means for storing the file type;
Data generation means for generating file data using a pseudo-random number generation algorithm from the file seed;
A file system access unit that receives a read request for a file and returns an access response including the data generated by the data generation unit ;
The data generation means includes
The file block number to be accessed and the offset in the block are calculated from the offset specified in the read request and the fixed-length block size,
Generating a block seed corresponding to the file block using a non-recursive pseudo-random number generation algorithm from the file seed and the file block number;
Generating data for the file block from the seed of the block using a recursive pseudo-random number generation algorithm;
Of the data of the generated file block, data determined by the offset in the block and the data size specified by the read request is generated as data to be accessed.
Virtual file system. A data verification unit that compares the data to be accessed generated by the data generation unit with the write data;
The file system access means receives a write request for the file that specifies an offset from the head of the file, a data size, and write data, and returns an access response including a comparison result by the data verification means. The virtual file system according to claim 1 . Virtual file system of claim 1 or 2, characterized in that it comprises a file creating means for storing in said memory means from the file name to extract the files species. Virtual file system according to any one of claims 1 to 3 wherein the file species, characterized in that stored in the storage means corresponding to the file name. A virtual file method executed by a computer having storage means for storing a file type, data generation means, and file system access means,
The data generation means generates file data using a pseudo-random number generation algorithm from the file seed,
The file system access means accepts a read request for a file, returns an access response including the generated data ,
In generating the data of the file, the data generating means calculates the number of the file block to be accessed and the offset within the block from the offset specified in the read request and the fixed-length block size, and the file A non-recursive pseudo-random number generation algorithm is used to generate a block seed corresponding to the file block from the seed and the file block number, and the file is generated from the block seed using a recursive pseudo-random number generation algorithm. Block data is generated, and data determined by the offset in the block and the data size specified by the read request is generated as data to be accessed among the generated data of the file block.
Virtual file method. Executed by a computer having a virtual file system having storage means for storing file types, data generation means, data verification means, and file system access means, and evaluation means for evaluating storage using the virtual file system Storage evaluation method,
The evaluation means sends a read request to the virtual file system;
The data generation means of the virtual file system generates file data using a pseudo-random number generation algorithm from the file seed,
The file system access means of the virtual file system returns an access response including the data generated by the data generation means to the evaluation means;
The evaluation means backs up data included in the access response to the storage,
The evaluation means reads the data backed up in the storage,
The evaluator sends a write request to the virtual file system to restore the read data;
The data generation means of the virtual file system generates file data using a pseudo-random number generation algorithm from the file seed,
The data verification unit of the virtual file system compares the data generated by the data generation unit with the data included in the write request;
The file system access unit of the virtual file system returns a response including the comparison result to the evaluation unit ,
In generating the data of the file, the data generating means calculates the number of the file block to be accessed and the offset within the block from the offset specified in the read request and the fixed-length block size, and the file A non-recursive pseudo-random number generation algorithm is used to generate a block seed corresponding to the file block from the seed and the file block number, and the file is generated from the block seed using a recursive pseudo-random number generation algorithm. Block data is generated, and data determined by the offset in the block and the data size specified by the read request is generated as data to be accessed among the generated data of the file block.
Storage evaluation method.

Images