JP6327028B2 - Object storage system, control method thereof, and control program thereof - Google Patents

Description

  The present invention relates to an object storage system, a control method thereof, and a control program thereof.

  An object storage system is a system that handles a series of data as objects. An operating system (OS) of a file-based object storage has a file system for controlling data recorded in a storage device. The file system divides the storage space in the storage device and creates a file as a data storage destination in the divided storage space. And the file system manages how files and other storage spaces are used. In the case of a large-capacity object, the data is divided into data having a size of a predetermined unit or less, and each data is stored in a file. Some of these file systems record a processing log in order to quickly recover from a failure. Such a file system is called a journaling file system. A general description of the journaling file system is described in Non-Patent Document 1, for example.

  In a general storage system, for example, as described in Non-Patent Document 2, a file verification program is resident and periodically verifies the integrity of all stored files.

Wikipedia (registered trademark), “Journaling File System”, [March 25, 2014 search], Internet <URL: http://en.wikipedia.org/wiki/%E3%82%B8%E3%83%A3 % E3% 83% BC% E3% 83% 8A% E3% 83% AA% E3% 83% B3% E3% 82% B0% E3% 83% 95% E3% 82% A1% E3% 82% A4% E3 % 83% AB% E3% 82% B7% E3% 82% B9% E3% 83% 86% E3% 83% A0> VNA Linux Systems Japan Co., Ltd. Website, “OpenStack: Distributed Object Storage Switch”, [Search March 25, 2014], Internet <URL: http://www.valinux.co.jp/technologylibrary/document / cloud / openstack0001>

  However, as in Non-Patent Document 2, a method in which a file verification program is resident and performs data verification of all files has a problem that the load on the system increases as the amount of data to be stored increases. Specifically, one million objects consume several tens of percent of CPU load and input / output load.

  The present invention has been made in view of the above problems, and provides a data verification method that does not require the residence of a file verification program and the verification of all files, and an object thereof is to reduce the system load due to data verification.

  In order to solve the above problems, an object storage system of the present invention is an object storage system that stores data constituting an object as a file, and a storage device that stores the file, and an operating system that controls the storage system And an object management unit that manages the file in association with the object, the storage device has a journal that holds an update history of the file, and the object management unit stores the object in the file. Object identifier setting means for setting an object identifier indicating the correspondence between the storage system, the operating system, the file system that controls the storage device and the file, and the recording when the operating system is mounted. A mount file update means for verifying the file updated by the apparatus, and the file system stores the damaged file found by the file verification means in association with the object identifier. Have.

  The effect of the present invention is that the system load for data verification is reduced. In addition, the attribute of the damaged file can be quickly identified and repaired.

It is a block diagram which shows the 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the 1st Embodiment of this invention. It is a block diagram which shows the 2nd Embodiment of this invention. It is a flowchart which shows the operation | movement of the 2nd Embodiment of this invention. It is a block diagram which shows the metadata of a general technique. It is a block diagram which shows the metadata of the 3rd Embodiment of this invention. It is a sequence diagram which shows the file storage operation | movement of the 4th Embodiment of this invention. It is an example of the directory block diagram which shows the 5th Embodiment of this invention. It is an example of the directory structure figure of a general technique.

Hereinafter, the present invention will be described in detail with reference to the drawings.
(First embodiment)
The object storage system of the present invention relates to a file-based object storage system that stores data constituting an object as a file.

  FIG. 1 is a block diagram showing an object storage system 100 of this embodiment. The object storage system 100 includes a storage device 10 that stores a file 11, an operating system 20, and object management means 30 that manages the file 11 in association with the object 1. The operating system 20 controls connection devices including the storage device 10 and performs intermediation and control between application programs, devices, and application programs. Note that the object management means 30 may be implemented in the operating system 20. Hereinafter, the operating system 20 is also referred to as OS 20.

  The storage device 10 includes a journal 12 that holds an update history of the file 11 in addition to the storage area of the file 11.

  The object management unit 30 includes an object identifier setting unit 31 that sets an object identifier 2 indicating the correspondence with the object 1 in the file 11. In a general format, the file 11 includes data 3 and metadata 4, but the object identifier 2 is set as one element of the metadata 4, for example.

  The operating system 20 controls the operation of the object storage system 100. And it has the file system 40 and the update file verification means 50 at the time of mounting.

  The file system 40 controls the file 11. Here, the control of the file 11 includes, for example, dividing the storage space of the storage device 10 to arrange the file 11 and controlling the hierarchical structure of the directory to which the file 11 belongs.

  The mount update file verification means 50 verifies the file 11 updated in the storage device 10 when the file system 40 is mounted. The updated file 11 is specified by referring to the journal 12.

  The file system 40 also includes a damaged file storage unit 41 with an object identifier. The corrupted file with identifier storage unit 41 stores the corrupted file with object identifier 5 passed from the update file verification unit 50 at the time of mounting.

  According to the configuration described above, the damaged file is stored in the damaged file storage unit with object identifier 41 as the damaged file 5 with object identifier. For this reason, the damaged object can be easily identified. Further, since only the updated file is verified when the file system 40 is mounted, a verification program is resident, and the load on the system can be greatly reduced as compared with the method of verifying all the files. Since the update files have already been verified, there is no need for verification. The object storage system 100 may include an application program 60 that uses the OS 20.

  Next, the detail of each part is demonstrated.

  The object management unit 30 handles a series of data as one object, and internally inputs and outputs the file 11 corresponding to the object. The object identifier setting unit 31 sets the object identifier 2 in the metadata 4 when the object is stored in the storage device 10 as the file 11. Then, the object management means 30 discriminates and manages each object 1 with the object identifier 2.

  The file system 40 divides the storage space of the storage device 10 to generate and control the file 11 that is the storage destination of the data 3. Data is input / output between the object management means 30 or the application program 60 and the storage device 10. The file system 40 also includes a damaged file storage unit 41 with an object identifier for storing the damaged file 5 with an object identifier received from the update file verification unit 50 at the time of mounting.

  As described above, the mount-time updated file verification unit 50 verifies the file 11 updated in the storage device 10 when the file system 40 is mounted. The updated file 11 is specified by referring to the journal 12. The file in which the damage is found by the verification is transferred to the file system 40 as the damaged file 5 with the object identifier, and stored in the damaged file storing means 41 with the object identifier. A damaged file 6 in which the metadata 4 is damaged and the object identifier 2 cannot be specified is stored in another folder. This folder can be, for example, LOST + FOUND80 used in general technology.

  Next, before explaining the operation, the reason why this mechanism is necessary will be briefly described. In general, in a storage system, a record indicating that the file system has ended normally is stored when the storage system ends normally. When the storage system ends abnormally, a record indicating that the file system has ended normally cannot be recorded, so that it is understood that the storage system has not ended normally at the next startup. If the storage system has terminated abnormally, check the normality of the file system at the next mount, and if an abnormality is found, restore it. In a journaling file system using a journal, an update record after being synchronized with a storage device for storing the file is recorded in the journal. When the process ends abnormally, it is only necessary to restore the update recorded in the journal. Therefore, the amount of processing can be greatly reduced as compared with the method of verifying and restoring the whole. The present invention is based on the journaling file system technique described above. Since the details of the journaling file system are well-known techniques, the description thereof is omitted.

  By the way, a damaged file may occur in the process of recovery. This is due to the following reason. The purpose of the journaling file system is not to protect the data itself, but to protect the entire file system. In the recovery of the journaling file system, the log after the last synchronized log is read, the completed operations and committed transactions are re-executed in order, and the file system is reconstructed. When reconstruction is performed, the consistency of the entire file system is guaranteed, but it is not always in a correct state for an upper layer such as an application. This causes file corruption problems.

  Returning to the invention, the operation will be described. FIG. 2 is a flowchart showing the operation of the present embodiment.

  First, the OS 20 is activated when the system is activated, and the connected device such as the storage device 10 is validated. Next, the file system 40 is activated (S1). The file system makes it possible to handle the mount-designated area as a file system. The following processing is a part of processing performed at the time of mounting. Then, the on-mount update file verification means 30 is activated (S2). Next, the mounted update file verification means 30 refers to the journal 12 (S3), and specifies the file 11 that has been updated since the last synchronization as a verification target (S4). Then, the transaction recorded in the journal and committed but not reflected is executed again (S5).

  Next, the data stored in the updated file 11 is verified (S6). As a specific verification method, whether the checksum value of the data stored in the metadata area and the checksum value calculated from the data indicate the same value can be considered. The checksum value stored in the metadata 4 is calculated by the storage management unit 30 when data is stored (when a file is created or changed) and stored in the metadata 4.

  As a result of the verification by the update file verification unit 30 at the time of mounting, if it is determined that the data is invalid (S7_yes), the corresponding file is passed to the damaged file storage unit with object identifier 41. The damaged file storage means with object identifier 41 stores the file with the object identifier 2 under a dedicated directory (S8). If it is determined that there is no fraud (S7_no), the verification of the file is terminated, and it is confirmed whether there is another unverified file (S9). If there is an unverified file (S9_yes), the process returns to S6, and data verification is performed on this file. This is sequentially repeated for those recorded in the journal. When there are no unverified files (S9_no), that is, when the verification of all the files specified in S4 is completed, the process ends.

  After the file verification is completed, the file system publishes the corresponding area as a file system. (Completes the file system mount). This processing may be performed by a generally known mount processing method. After mounting, the file can be used from the application program.

As described above, according to the present embodiment, the damaged file is stored in the damaged file storing unit 41 with the object identifier as the damaged file 5 with the object identifier. For this reason, the damaged object can be easily identified. In addition, since only the update file is verified, the consumption of system resources can be greatly reduced as compared with the method in which all files are always verified.
(Second Embodiment)
FIG. 3 is a block diagram showing a second embodiment of the present invention. The object management means 30 of this embodiment has an object repair means 32.

  Next, a method for repairing the object 1 will be described. FIG. 4 is a flowchart showing this operation.

  First, the file system 40 is activated (S101). Next, the object management means 30 is activated (S102). When the object management unit 30 is activated, the object restoration unit 32 refers to the damaged file storage unit with object identifier 41 and identifies the object 1 in which the file 11 has been damaged (S103). The object stored in the damaged file storage means with object identifier 41 can be easily identified by the object identifier 2 to which object 1 the file belongs. Next, the object restoration means 32 copies the backup object 1 stored in the other object storage (S104), generates the file 11, and stores it in the storage device 10 (S105). Then, the object identifier damaged file 5 is deleted from the object identifier damaged file storage means 41. (S106). This completes the restoration of the stored object.

With the above operation, the damaged object 1 can be quickly repaired.
(Third embodiment)
The present embodiment relates to the configuration of the metadata 4. Therefore, the configuration of the metadata 4 will be described in comparison with a general technique.

  FIG. 5 shows an example of metadata 4 in a general technique. The i-node number 4-1, the creation date 4-2, the update date 4-3, the data length 4-4, and the address of the corresponding data storage location (position stored on the storage device) are described. With existing technologies such as UNIX (registered trademark) and Linux (registered trademark), when a file is damaged, the damaged file is stored in a specific damaged file storage directory called “lost + found” with the i-node number 4-1 as the file name. . Here, the i-node number 4-1 does not include object information. The metadata 4 is normally automatically created and changed by the file system, but can be registered from the outside.

FIG. 6 shows an example of metadata 4 in the present embodiment. Similar to a general technique, an i-node number 4-1, a creation date 4-2, an update date 4-3, a data length 4-4, and a corresponding data storage location address are used. Further, a checksum 4-5 for verifying data may be added. Further, in the present embodiment, the object identifier 2 is added to the metadata 4. The object identifier 2 is used for storing the data 3 of the file 11 in association with the object 1. According to the present embodiment, since the attribute of the damaged file can be specified by introducing the object identifier 2, the object 1 can be easily restored.
(Fourth embodiment)
In this embodiment, file creation will be described. First, an operation of adding the object identifier 2 to the metadata 4 and storing the data 3 in the file 11 will be described.

  FIG. 7 is a sequence diagram showing this operation. First, the object management unit 30 uses the object identifier setting unit 31 to determine the object identifier 2 of the data 3 to be stored as the file 11 (S201).

  Next, a file name as a storage destination is designated, and a write request for data 3 is issued to the file system 40 (S202).

  The file system 40 writes the data 3 to the storage area (S211). When writing to the storage device 10 is completed, a write completion notification is returned from the storage device 10 (S212).

  Next, the file system 40 writes the metadata 4 corresponding to the i-node number 4-1 of the data 3 to the file 11 (S213). The metadata 4 is also stored in metadata storage means (not shown) provided in the file system 40. This corresponds to writing to the journal 12 in the existing technology (journaling file system).

  When the writing to the storage device 10 is completed, a write completion notification is returned from the storage device 10 (S214).

  When the data can be written to the storage device 10, the file system 40 notifies the object management means 30 of the completion of writing (S203). Next, the object management means 30 registers the object identifier 2 for the written data 3 in the file metadata 4. The object management means 30 issues this registration request to the file system 40 (S204).

  The file system 40 that has received the request adds the object identifier 2 to the metadata 4 of i-node corresponding to the data 3 (S215). When the metadata 4 can be written, a write completion notice is received (S216).

  Upon receiving the write completion notification, the file system 40 notifies the object management means 30 of the write completion (S205).

As described above, the data 3 can be stored with the object identifier 2 added thereto.
(Fifth embodiment)
In this embodiment, a configuration example of the damaged file storage means 41 with an object identifier for effectively using the present invention is shown. FIG. 8 is a directory configuration diagram showing an example of the damaged file storage means 41 with an object identifier. In the present invention, for example, a directory called LOSTFILES 41 a is created as a damaged file storage unit 41 with an object identifier immediately under the root directory of the file system 40. Then, under the LOSTFILES 41a, for example, a subdirectory 42a having the directory name of the lower three characters of the object identifier 2 is created. In addition, you may divide | segment by a method other than lower 3 characters. If the object identifier 2 (= file name) does not collide in the name space, a single directory may be used. However, if a large number of files are expected, it may cause performance problems. Then, a damaged file 5 with an object identifier having the object identifier 2 as a file name is arranged under the subdirectory 42. As long as the object identifier 2 can be recognized, the file name may not be the object identifier 2 itself.

  Thus, when arranged, the object management means 30 can easily identify the object 1 having a damaged file. If the object 1 can be identified, it can be directly repaired using the backup data, so that the repair time can be greatly shortened compared to the method of performing all file verification and repair.

  Next, for comparison, storage of a damaged file 6 without an identifier in a general technique will be described. FIG. 9 is a directory configuration diagram showing this storage example. In a general technique, the damaged files 6 are put together in a directory such as LOST + FOUND70, for example. For this reason, in some cases, a very large number of damaged files 6 are stored in one directory. Further, since the file name is an i-node number, it is very difficult for the user to specify the relationship with the original file name or object.

  As described above, according to the present embodiment, the relationship between a damaged file and an object that occurs when the file system 40 is restored can be easily identified and linked to quick repair.

DESCRIPTION OF SYMBOLS 1 Object 2 Object identifier 3 Data 4 Metadata 5 Corrupt file with an object identifier 6 Corrupted file 10 Storage device 11 File 12 Journal 20 Operating system 30 Object management means 31 Object identifier setting means 32 Object repair means 40 File system 41 Corruption with an object identifier File storage means 50 Mounted file verification means 60 Application program 100 Object storage system

Claims (10)

  An object storage system for storing data constituting an object as a file, a storage device for storing the file, an operating system for controlling the storage system, and object management means for managing the file in association with the object And the storage device has a journal that holds the update history of the file, and the object management means has an object identifier setting means for setting an object identifier indicating a correspondence with the object in the file. The operating system includes a file system that controls the storage device and the file, and a mount-time update file verification unit that verifies the file updated in the storage device when the file system is mounted. The file system object storage system, wherein said file verification means having with object identifiers corrupted file storage means for storing object identifiers with corrupted files discovered it.   The object storage system according to claim 1, wherein the object management unit includes an object repair unit that replicates the object from another storage system and repairs the damaged file with the object identifier.   The object storage system according to claim 1 or 2, wherein the damaged file storage unit with an object identifier stores the damaged file with an object identifier by a file name that can be identified by the object identifier.   4. The object according to claim 3, wherein the damaged file storage unit with an object identifier includes a damaged file storage folder generation unit with an object identifier that generates a folder having a name that allows the object identifier to be identified. Storage system.   An object storage system control method for storing data constituting an object as a file, wherein an object management means sets an object identifier indicating a correspondence with the object in the file, and the file system stores the file in a storage device And the file system stores the update history of the file in a journal in the storage device, and the mount-time update file verification means verifies the updated file with reference to the journal when the file system is mounted, A method for controlling an object storage system, comprising: storing a damaged file with an object identifier discovered by the update file verification unit at the time of mounting in a damaged file storing unit with an object identifier.   6. The storage system control method according to claim 5, wherein the object is copied from another storage system to repair the damaged file with the object identifier.   The object storage system control according to claim 5 or 6, wherein the object identifier damaged file storage means stores the object identifier damaged file with a file name that can be identified by the object identifier. Method.   The damaged file storage unit with an object identifier generates a folder having a name that can identify the object identifier, and stores the damaged file with an object identifier under the corresponding folder. The object storage system control method described.   A control program for an object storage system for storing data constituting an object as a file, wherein an object management means sets an object identifier indicating a correspondence with the object in the file, and a file system stores the file in the storage device. The file system stores the update history of the file in a journal in the storage device, and the mount-time update file verification means is updated with reference to the journal when the file system is mounted. And verifying the file, and storing the damaged file with the object identifier discovered by the mount-time update file verifying means in the damaged file storing means with the object identifier. Control program of the object storage system.   10. The control program for an object storage system according to claim 9, wherein the damaged file storage unit with an object identifier has a step of storing the damaged file with an object identifier with a file name that can be identified by the object identifier. .

Images