WO2016120884A1

WO2016120884A1 - Failure atomic update of a single application data file

Info

Publication number: WO2016120884A1
Application number: PCT/IN2015/000061
Authority: WO
Inventors: Anton Ajay MENDEZ; Rajat VERMA; Sandya Srivilliputtur Mannarswamy; Terence P. Kelly; James Hyungsun PARK
Original assignee: Hewlett-Packard Development Company, L.P.
Priority date: 2015-01-30
Filing date: 2015-01-30
Publication date: 2016-08-04

Abstract

In one example, a system is described in which a storage device, communicatively coupled to a processor, includes a failure atomic update module. The failure atomic update module may create a file clone upon opening a file including data blocks and an atomic flag by the application. Further, the failure atomic update module may remap any modified data blocks of the file upon a subsequent modification and/or addition to the file by the application. Furthermore, the application may sync using a sync operation. In addition, the failure atomic update module may flush any modified data blocks in the file into a stable storage media (disk), and delete the created file clone by the file system. Moreover, the failure atomic update module may create a new file clone including any modified and unmodified data blocks.

Description

FAILURE ATOMIC UPDATE OF A SINGLE APPLICATION DATA FILE

Background

[0001] Many applications modify data on durable media, such as storage devices, and any untimely failures during updates/modifications, for example, application process crashes, operating system (OS) kernel panics, power outages and the like, may jeopardize the integrity of the application data.

Brief Description of the Drawings

[0002] Examples of the disclosure will now be described in detail with reference to the accompanying drawings, in which:

[0003] FIG. 1 illustrates a block diagram of an example system for a mechanism for failure atomic update of application data in a single application data file in a file system;

[0004] FIG. 2 illustrates a block diagram of another example system for mechanism for failure atomic update of application data in a single application data file in a file system;

[0005] FIG. 3 illustrates a block diagram illustrating an example implementation of a mechanism for failure atomic updates of application data in a single application data file in a file system, such as those shown in FIGS. 1 and 2;

[0006] FIG. 4 illustrates a flow chart of an example method for failure atomic update of application data in a single application data file in a file system; [0007] FIG. 5 illustrates a block diagram of an example computing device for a mechanism for applications for failure atomic update of application data in a single application data file in a file system.

Detailed Description

[0008] In the following detailed description of the examples of the present subject matter, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific examples in which the present subject matter may be practiced. These examples are described in sufficient detail to practice the present subject matter, and it is to be understood that other examples may be utilized and that changes may be made without departing from the scope of the present subject matter. The following detailed description is, therefore, not to be taken in a limiting sense.

[0009] Examples described herein provide enhanced methods, techniques, and systems for a mechanism for applications to perform failure atomic update of application data in single application data file in a file system. Generally, failure atomic updates, (consistent modification of application data, i.e., the problem of evolving durable application data without fear that failure will preclude recovery to a consistent state) protect integrity of application data from system failures, such as process, crashes, OS kernel panics and/or power outages.

[0010] Typically, file systems strive to protect internal metadata from corruption; however, file systems may not offer corresponding protection for application data, providing neither transactions on application data nor other unified solution to the consistent modification of application data problem. Instead, file systems may offer primitives for controlling the order in which application data attains durability; applications may shoulder the burden of restoring consistency to their data following failures. Consider for example, the task of failure-atomically updating a set of configuration files scattered throughout a directory tree atop a post operating system for Unix (POSIX)-like file systems. In such a scenario, the vast majority of file systems may not provide a straightforward operation that the failure atomic updates demand: the ability to modify application data in (sets of) files, failure atomically and efficiently.

[0011] Some existing mechanisms may provide imperfect support for solving failure atomic updates problem. Further, existing file systems may offer limited support for failure atomic updates, may be due to problems associated with OS interfaces. For example, POSIX may permit write to succeed partially, making it difficult to define atomic semantics for this call. Further for example, synchronization calls, such as fsync and msync may constrain the order in which application data reaches durable media. However, applications generally remain responsible for reconstructing a consistent state of their data following a crash. Sometimes, applications may circumvent the need for recovery by using the one failure-atomic mechanism provided in conventional file systems, i.e., the file rename. For example, desktop applications can open a temporary file, write the entire modified contents of a file to it, then use the rename to implement an atomic file update - a reasonable expedient for small files but may be untenable for large files. Further some existing mechanisms, may require special hardware and may apply only to single-file updates, and may not address modifications to memory-mapped files. Furthermore in some existing mechanisms, transaction size, i.e., size of atomically modified data in the file may be limited by the size of the journal, may employ software, which may carry substantial overheads. In addition, a journal based implementations of failure-atomic sync operation may suffer at least two shortcomings, one being a need to run a modified kernel that may impede adoption, and the other being use of the file system journal that can limit transaction sizes.

[0012] To help address these issues, the present disclosure describes various example mechanisms for applications for failure atomic update of application data in a single application data file in a file system. In one example, a simple interface to file system may offer applications a guarantee that the application data in a file always reflects the most recent successful sync invocation, such as fsync or msync operation, on the file. Further, the interface to the file system offers a sync mechanism that failure- atomically commits changes to files. Furthermore, failure-injection test verifies that the file system protects the integrity of application data from crashes. In addition, the interface to the file system runs on conventional hardware and operating system and the mechanism is implementable in any file system that supports per-file snapshots.

[0013] In addition, the example implementations describe a simple interface to the file system that generalizes failure-atomic variants of write and sync operations. If a file is opened with an atomic flag, the state of its application data will always reflect the most recent successful sync operation, such as msync, fsync, and/or fdatasync. Further, the size of atomic updates to the file may only be limited by the free space in the file system and not by the file system journal. Furthermore, opening a file with an atomic flag ensures that the file's application data reflects the most recent synchronization operation regardless of whether the file was modified with interfaces, such as write and/or mmap families of interfaces. Atomic flag may be implemented in a file system that supports per-file snapshots. Also, the sync operation described in the present disclosure ensures that the updates to a file are atomic in nature. The file system may not rely solely on the file system journal to implement atomic updates, and the size of atomic updates may be limited only by the amount of free space in the file system. Adding such an interface to the file system may be relatively easy as it can run on any conventional OS kernels and requires no special hardware. Further, file clone implementation in the file system enables a simple but effective failure atomic update via atomic flag.

[0014] The terms "storage media", "durable media", "storage device", and "disk drive" are used interchangeably throughout the document. Also, the terms "file" and "application data file" are used interchangeably throughout the document. Further the term "sync operation" refers to "synchronization operation". Furthermore, the term "application" refers to "application software". In addition, the term "file clone" refers to "file's clone". Moreover, the terms "system failure", "untimely failure", and "untimely system failure", as used herein, may refer to process crashes, OS kernel panics, power outages and the like. [0015] FIG. 1 illustrates a block diagram of an example system 100 for a mechanism for applications for failure atomic update of application data in a single application data file in a file system 106. The system 100 may represent any type of computing device capable of reading machine-executable instructions. Examples of computing device may include, without limitation, a server, a desktop computer, a notebook computer, a tablet computer, a thin client, a mobile device, a personal digital assistant (PDA), a tablet, and the like.

[0016] In the example of FIG. 1, the system 100 may include a processor 102 and storage device 104 coupled to the processor 102. In an example, the storage device 104 may be a machine readable storage medium (e.g., a disk drive). The machine-readable storage medium may also be an external medium that may be accessible to the system 100. Further, the storage device 104 may include the file system 106. Furthermore, the file system 106 may include failure atomic update module 108.

[0017] For example, the failure atomic update module 108 may refer to software components (machine executable instructions), a hardware component or a combination thereof. The failure atomic update module 108 may include, by way of example, components, such as software components, processes, tasks, co-routines, functions, attributes, procedures, drivers, firmware, data, databases, data structures and Application Specific Integrated Circuits (ASIC). The failure atomic update module 108 may reside on a volatile or non-volatile storage medium and configured to interact with a processor 102 of the system 100.

[0018] In one example, the file system 106 may include data blocks, snapshots of files, directory and/or file clones implemented by atomic updates as shown in FIG. 3. For example, file clones may include shared data blocks of a file (i.e., primary file) in the file system that are implemented by atomic updates. The file system may decouple logical file hierarchy from the physical storage. The logical file hierarchy layer may implement the naming scheme and portable operating system interface (POSIX) complaint functions, such as, creating, opening, reading, and writing files. The physical storage layer implements write-ahead logging, caching, file storage allocation, file migration, and/or physical disk input/output (I/O) functions. This is explained in more detail with reference to FIG. 3.

[0019] In operation, a file including an atomic flag may be opened upon invoking an open operation by an application. For example, the file may include data blocks: Block 0, Block 1 , and Block 2 as shown at 302 in FIG.3. The atomic flag may indicate the application's desire that changes to the application data in a file may be atomic.

[0020] A file clone including shared data blocks of the file may then be created by the application upon opening the file including the atomic flag. File clone may be a writable snapshot of the file at the time it is opened with using the atomic flag. The file clone may not change with any modification to the data blocks in the file. Further, the file clone may not be visible to the user visible namespace and may exist in a non- visible (hidden) namespace that may be accessible to the operating system (OS). For example, file clone CLONE 0 iNODE may be implemented utilizing a variant of copy-on-write (COW) operation as shown at 304 in FIG. 3. Further for example, when a file is cloned, a copy of the file's iNODE may be made as shown in FIG. 3. The iNODE may includes the file's block map, a data structure that maps logical file offsets to block numbers on the underlying block device as shown in FIG. 3. For example, it can be seen in FIG. 3, that the original file FILE iNODE and its file clone CLONE 0 iNODE have identical copies of the block map, they may initially share the same storage.

[0021] Any modified data blocks in the file are remapped by the file system upon a subsequent modification and/or addition to the file by the application. For example, modified data blocks may be remapped using COW operation and leaving the file clone's view of the file unchanged. For example, addition of Block 3 and remapping of added Block 3 via COW is shown at 306 in FIG. 3. It can be seen that the file clone CLONE 0 iNODE still points to the blocks: Block 0, Block 1 and Block 2 of the file at the time it was opened.

[0022] A sync operation may then be initiated by the application. Any modified data

blocks in the file are then flushed into a stable storage media, such as a disk drive and the created file clone may then be deleted and new file clone including any modified and unmodified data blocks may be created. The state of the file may reflect a logical state of the file at the time the application synched using the sync operation. Example sync operations are fsync operation, msync operation and fdatasync operation. For example, sync operation replacing created file clone CLONE 0 iNODE with new file clone CLONE 1 iNODE is shown at 308 in FIG. 3. In one example, the last close of a file opened with atomic flag and all cached blocks of the file are flushed and any existing file clones are deleted. In another example, the above mechanism repeats itself until the file is closed by the application.

[0023] In one example, the failure atomic update module 108 determines if there was an untimely system failure. Based on the outcome of the determination, if the untimely system failure occurs before deleting the file clone, the failure atomic update module 108 replaces the file with file clone next time the file is opened by the application. Based on the outcome of the determination, if there was no untimely system failure and the file clone is deleted, the failure atomic update module 108 creates the new file clone including any modified and unmodified data blocks. In another example, an intermediary approach may include a background daemon to search the file system for recoverable files after mount but before files are opened.

[0024] In one example, if the system fails, recovery of a file may be delayed until the file is accessed again. The file system's path name lookup function may check if the file's clone exists in the hidden namespace. The file clone is then renamed to the user visible file and a handle to it is returned if the file clone exists in the hidden namespace. The per-file recovery offers several attractions, for example, consider an OS kernel panic that occurs while many processes are updating many files. Upon reboot, the file system may recover quickly because the in-progress updates, interrupted by the crash trigger no recovery actions when the file system is mounted. In such a scenario, the applications that may not need recovery from interrupted atomic updates (e.g., applications that are merely reading files) may not share the recovery-time penalty incurred by the crash; only those applications that benefit from application-consistent recovery may pay the penalty. The above described atomic failure update mechanism is built on top of the file clone feature of file system, it can be envisioned that alternative implementations, such as using delayed journal writeback may be possible.

[0025] FIG. 4 illustrates a flow chart of an example method 400 for failure atomic update of application data in a single application data file in a file system. The method 400, which is described below, may be executed on a system such as a system 100 of FIG. 1 or system 200 of FIG. 2. However, other systems may be used as well. At block 402, a file including data blocks and an atomic flag is opened upon invoking an open operation by an application. The atomic flag may indicate the application's desire that any changes to the file be atomic.

[0026] At block 404, a file clone is created upon opening the file including the atomic flag by the application. The file clone may be a writable snapshot of the file at the time it is opened using the atomic flag. In one example, a file clone including shared blocks of the primary file is created upon opening the file including the atomic flag by the application. The primary file and the file clone may share same blocks until one or more blocks in the primary file is modified.

[0027] At block 406, any modified data blocks of the file are remapped upon a subsequent modification and/or addition to the file by the application. In one example, any modified data blocks of the file are remapped via copy of write (COW) operation and leaving the file clone's view of the file unchanged by the file system upon a subsequent modification and/or addition to the file the application.

[0028] At block 408, a sync operation may. be initiated by the application. Example sync operation is fsync operation, mysnc operation and/or fdatasync operation.

[0029] At block 410, any modified data blocks in the file are flushed to a stable storage media and the created file clone is deleted by the file system. Example stable storage media is a disk drive. In one example, any modified data blocks in the file is flushed into a stable storage media such that the state of the file reflects a logical state of the file at the time the application syncs using the sync operation, and then the created file clone is deleted by the file system. [0030] At block 412, a new file clone is created including any modified and unmodified data blocks. At block 414, a determination is made as to whether the application has closed the file. Based on the outcome of the determination at block 414, the process 400 goes to block 406 and repeats the steps outlined in blocks 406 to 414 if the file is still open and not closed by the application. Further, based on the outcome of the determination at block 414, the process 400 goes to block 416 and stops if the file is closed by the application.

[0031] In one example, the failure atomic update module 108 determines whether there was an untimely system failure. If the untimely system failure occurs before the deleting the file cone, the file is then replaced with the file clone the next time the file is opened by the application. Based on the outcome of the determination, if there was no untimely system failure and the file clone is deleted, a new file clone is created including any modified and unmodified data blocks.

[0032] FIG. 5 illustrates a block diagram of an example computing device 500 for a mechanism for failure atomic update of application data in single application data file in a file system. The computing device 500 includes a processor 502 and a machine- readable storage medium 504 communicatively coupled through a system bus. The processor 502 may be any type of central processing unit (CPU), microprocessor, or processing logic that interprets and executes machine-readable instructions stored in the machine-readable storage medium 504. The machine-readable storage medium 504 may be a random access memory (RAM) or another type of dynamic storage device that may store information and machine-readable instructions that may be executed by the processor 502. For example, the machine-readable storage medium 504 may be synchronous DRAM (SDRAM), double data rate (DDR), rambus DRAM (RDRAM), rambus RAM, etc., or storage memory media such as a floppy disk, a hard disk, a CD-ROM, a DVD, a pen drive, and the like. In an example, the machine- readable storage medium 504 may be a non-transitory machine-readable medium. In an example, the machine-readable storage medium 504 may be remote but accessible to the computing device 500. [0033] The machine-readable storage medium 504 may store instructions 402, 404, 406, 408, 410, 412, 414 and 416. In an example, instructions 402, 404, 406, 408, 410, 412, 414 and 416 may be executed by processor 502 to provide a mechanism for failure atomic update of application data in single application data file in a file system. Instructions 402, 404, 406, 408, 410, 412, 414 and 416 may be executed by processor 502 to implement failure atomic updates of application data. Instructions 402, 404, 406, 408, 410, 412, 414 and 416 may be executed by processor 502 to protect integrity of application data from failures, such as process crashes, OS kernel panics, and/or power outages.

[0034] It may be noted that the above-described examples of the present solution is for the purpose of illustration only. Although the solution has been described in conjunction with a specific embodiment thereof, numerous modifications may be possible without materially departing from the teachings and advantages of the subject matter described herein. Other substitutions, modifications and changes may be made without departing from the spirit of the present solution. All of the features disclosed in this specification (including any accompanying claims, abstract and drawings), and/or all of the steps of any method or process so disclosed, may be combined in any combination, except combinations where at least some of such features and/or steps are mutually exclusive.

Claims

Claims:

1. A system, comprising:

a processor; and

a storage device communicatively coupled to the processor, wherein the storage device comprises a failure atomic update module to:

create a file clone upon opening a file including an atomic flag by an application, wherein the file includes data blocks;

remap any modified data blocks of the file upon a subsequent modification and/or addition to the file by the application;

sync by using a sync operation by the application;

flush any modified data blocks in the file into a stable storage media, and delete the created file clone by the file system; and

create a new file clone including any modified and unmodified data blocks.

2. The system of claim 1, wherein the failure atomic update module is further configured to:

go to the step of remapping; and

repeat the steps of remapping any modified data blocks in the file, initiating a sync operation, flushing any modified data blocks and deleting the created file clone, and creating a new clone upon each modification to the set of data blocks in the file by the file system until the file is closed by the application.

3. The system of claim 2, wherein the failure atomic update module determines whether there was an untimely system failure, based on the outcome of the determination, if the untimely system failure occurs before deleting the file clone, replaces the file with the file clone the next time the file is opened by the application, and further based on the outcome of the determination, if there was no untimely system failure and the file clone is deleted, creates the new file clone including any modified and unmodified data blocks.

4. The system of claim 1, wherein the atomic flag indicates the application's desire that changes to the file be atomic.

5. The system of clam 1 , wherein the file clone comprises a writable snapshot of the file at the time it is opened with the atomic flag.

6. The system of claim 1 , wherein the sync operation comprises fsync operation, msync operation, and/or fdatasync operation.

7. A method for failure atomic update of a single application data file, comprising:

creating a file clone upon opening a file including an atomic flag by the application, wherein the file includes data blocks;

remapping any modified data blocks of the file upon a subsequent modification and/or addition to the file by the application;

syncing by using a sync operation by the application;

flushing any modified data blocks in the file into a stable storage media and deleting the created file clone by the file system; and

creating a new file clone including any modified and unmodified data blocks.

8. The method of claim 7, further comprising:

going to the step of remapping; and

repeating the steps of remapping, syncing, flushing and deleting, and creating upon each modification to the set of data blocks in the file by the file system until the file is closed by the application.

9. The method of claim 8, wherein creating the new file clone including any modified and unmodified data blocks , comprises:

determining whether there was an untimely system failure;

based on the outcome of the determination, if the untimely system failure occurs before deleting the file clone, replacing the file with the file clone the next time the file is opened by the application; and based on the outcome of the determination, if there was no untimely system failure and the file clone is deleted, creating the new file clone including any modified and unmodified data blocks.

10. The method of claim 7, wherein the atomic flag indicates the application's desire that changes to the file be atomic.

1 1. The method of clam 7, wherein the file clone comprises a writable snapshot of the file at the time it is opened with the atomic flag.

12. The method of claim 7, wherein the sync operation comprises fsync operation, msync operation and/or fdatasync operation.

13. A non-transitory machine-readable storage medium comprising instructions for a mechanism for applications for failure atomic update of application data in a single application data file, the instructions executable by a processor to:

create a file clone upon opening a file including an atomic flag by the application, wherein the file includes data blocks;

sync by using a sync operation by the application;

flush any modified data blocks in the file into a stable storage media and deleting the created file clone by the file system; and

create a new file clone including any modified and unmodified data blocks.

14. The article of claim 13, further comprised to:

go to the step of remapping; and

repeat the steps of remapping, initiating, flushing and deleting, and creating upon each modification to the set of data blocks in the file by the file system until the file is closed by the application.

15. The article of claim 14, wherein creating the new file clone including any modified and unmodified data blocks , comprises:

determining whether there was an untimely system failure;

based on the outcome of the determination, if the untimely system failure occurs before deleting the file clone, replacing the file with the file clone the next time the file is opened by the application; and

based on the outcome of the determination, if there was no untimely system failure and the file clone is deleted, creating the new file clone including any modified and unmodified data blocks following the deletion of the file clone.