KR20230115931A - Method and data structure to perform a garbage collection and transactions concurrently by using shadow garbage collection - Google Patents

Abstract

The operating system starts garbage collection before the first transaction is committed and stops the first transaction, the garbage collection execution step of migrating the data of the victim block selected by the garbage collection to the target block, and the garbage collection A garbage collection method comprising resuming the first transaction when it is terminated is disclosed. If the data of the victim block is the old data of the first data pinned to the page cache by the interrupted first transaction and stored in the storage, the migration is executed using the shadow page cache. The shadow page cache is different from the page cache.

Description

Garbage collection method and apparatus for concurrent processing of garbage collection and transactions in a log-based file system

The present invention relates to computing technology, and more particularly to a technology for performing garbage collection using a shadow page cache, which is a cache different from a page cache for transactions.

<transaction>

An application program (hereinafter referred to simply as 'application') running on the host may be a subject that stores data in a storage (storage device) (eg, SSD). However, the application cannot directly control the storage. Thus, applications can write data to the storage through the operating system, which can directly control the storage. The part of the operating system that handles storage can be referred to as a file system.

Errors can occur if data is not stored in storage in the way the application requires. Transactions can be used to ensure that data is stored in storage in the manner required by the application.

A transaction is a set of write data, and properties such as atomicity, consistency, isolation between transactions, and durability maintained in the event of a power outage must be satisfied. A set of write data that satisfies these characteristics can be called a transaction.

In some prior art, applications require transactions, but operating systems do not require transactions. In this case, the application may be configured to implement the transaction in some way, which may be inefficient. Inside the storage is a fast-acting cache. When an application sends a write call for write data to the operating system, and the operating system sends write data to the storage in response, the write data is usually stored in the cache of the storage. If the application controls the transaction, the cache of the storage may be rarely used. Because write data is cached in the storage's cache, the transaction order desired by the application can be broken. Therefore, in some prior art applications, to maintain the order of the first transaction and the second transaction, the application writes the first transaction to the cache of the storage, flushes it to the non-volatile area of the storage, and then writes the second transaction to the storage's cache. After writing to the cache, it flushes it to the non-volatile area of storage. Here, flushing data X may mean storing data X in a non-volatile memory of storage.

A transition contains special data called a commit block. Existence of the commit block in the storage guarantees that all transaction data associated with the commit block in the storage are completely written. In order to guarantee the condition that the commit block included in one transaction is written after all other blocks, all other blocks are first written to and flushed from the storage cache, and then the commit block is written to and flushed from the storage cache. The method is used by conventional applications.

Apart from transactions, applications use a unit called a file when writing data to storage.

In some prior art, transactions are not written to the original file, but instead a new journal file is created. The journal file may be stored in a non-volatile memory of storage. Then, the first transaction is written to the journal file. Only after the first transaction is completely written in the journal file, the first transaction is reflected in the original file. In the event of a power outage, only transactions completely written to the journal file can be recovered.

However, according to the prior art described above, there is a problem that the following inefficiencies exist. That is, when a journal file is created, computing resource usage increases because the source file must be written once after writing the journal file once to process write data, and there are also disadvantages of calling a lot of flushes.

There are prior arts that solve the above problems. The above problem lies in that only applications operate transactions, and operating systems do not support transactions. Therefore, in the prior art to solve the above problem, the operating system supports the transaction function. However, these conventional techniques also have disadvantages. That is, the write data that the application wants to write is pinned (fixed) to the memory (page cache) by the operating system. After that, when the application sends a commit call to the operating system, the operating system writes the pinned memory to storage at once. Even when writing to storage, the operating system uses the transaction form directly. At this time, since the write data requested by the application must be pinned to the memory, the size of the transaction is limited by the size of the memory.

<log structure file system>

Storage may include a block, which is a basic unit for recording data. Also, the storage may include a segment, which is a set of a plurality of blocks. That is, one segment may include a plurality of blocks.

In the log structure file system, when a certain data A recorded in the first block of storage is to be modified to A', A' is not written to the first block, but A' is written to a second block different from the first block. In addition, A written in the first block is invalidated at an appropriate time after A' is written in the second block. The invalidation may be executed by the operating system sending an invalidation command to the storage.

In this specification, a log-structured file system may also be referred to as a log-based file system.

In the file system, there is metadata that stores whether data in use or data not in use is stored in a block possessed by the storage, and this may be referred to as a block bitmap. The metadata is stored in a non-volatile memory in the storage and can be cached and used in the host DRAM. Invalidation refers to indicating that a block is not in use for a specific block by modifying corresponding metadata. Block bitmap contents cached in host DRAM can be flushed to storage's non-volatile memory at checkpoint.

As such, in the case of a log structure file system, even if old data A already recorded in the storage is modified with new data A', the old data A remains in the storage for a predetermined time.

<Garbage collection (garbage collection)>

File mapping information indicating a mapping relationship between a certain data A recorded in the storage and a location of a block in which the data A is stored is stored in a file called an inode. The inode file is also stored in storage and can be temporarily stored in the page cache managed by the operating system.

A block considered valid in one segment may be referred to as a live block. If it is declared in predetermined file mapping information that certain valid data is stored in a certain block, the certain block can be considered valid, and in this case, the certain block can be referred to as a live block.

A process of moving data stored in all live blocks among blocks included in a certain first segment to another second segment and then making all blocks included in the first segment into empty blocks may be referred to as garbage collection. . The purpose of garbage collection is to free up free segments. Here, the first segment may be referred to as a victim segment and the second segment may be referred to as a current segment.

The garbage collection process includes caching a first page of a first block stored in a victim segment in a page cache, and storing the first page cached in the page cache in a second page of a current segment. can do.

Execution of the transaction and garbage collection described above are independent processes, and both can be executed by the operating system.

If garbage collection is started during a time period after starting a certain first transaction and before the end of the first transaction, the first transaction is suspended until the garbage collection is finished. The suspended first transaction is restarted from the suspended phase after the garbage collection is terminated.

1 illustrates an embodiment of performing garbage collection according to the prior art.

First, it may be assumed that a first transaction for updating block A from an old block A to a new block A' has started. At this time, the step of pinning the new page A' to the page cache of the memory 13 managed by the operating system executing the first transaction may have already been executed. At this time, the old block A may be stored in the third block 313 of the first segment 310 of the storage.

In this case, before the first transaction is terminated (committed), a first garbage collection setting the first segment as a victim segment may be started. When the first garbage collection starts, the first transaction may be stopped while the new page A′ is pinned to the page cache. In this case, the first garbage collection cannot cache the data A of the old block A stored in the third block 313 of the first segment 310 in the page cache. Therefore, the data stored in the fifth block 325 of the second segment 320, which is the current segment, by the first garbage collection is not the old block A, but the data that was pinned to the page cache by the interrupted first transaction. This is the new page A'.

Therefore, in the above situation, there is a problem that the first garbage collection to move the old block A stored in the victim segment to the current segment fails.

<Page cache swapping function>

When copying a disk block to another location, the original disk block is read into memory and copied to the destination. If the original disk block exists in the page cache, the contents of the already existing page cache are copied to the destination.

That is, when copying the first block (A) of storage to another destination block in storage, first move the data (A) of the first block to the memory used by the operating system, and then copy the data (A) stored in the memory. The data (A) of the first block is stored in the destination block. However, before moving the data A of the first block to the memory, the data of the first block (for example, new data for updating the data of the first block) (A') is already stored in the page cache of the memory. exists, the data A' of the first block already existing in the page cache is stored in the destination block.

<Problem in supporting transaction in log structure file system>

If the swapped transaction page is garbage collected, there is a problem that the updated page (A'), not the original page (A), is restored when a system crash occurs.

In the operating system according to the prior art, the page cache is designed based on the assumption that the location of a disk block related to each page cache entry is statically determined. However, log-structured filesystems can dynamically change the disk blocks associated with the page cache through garbage collection. In this case, when a disk block is dynamically moved, if a block related thereto exists in the page cache, the contents of the page cache are stored in the disk. With the checkpointing operation of garbage collection, the contents of the transaction currently in progress can be saved to disk in a recoverable form before the transaction is terminated. As a result, the atomicity of the transaction is violated.

Also, according to the prior art, the page cache can be pinned to memory, but the pinned page cache is not connected to a disk block. As a result, there is no possibility of swapping, and large-capacity transactions cannot be supported.

An object of the present invention is to provide a technique for solving an error that occurs when a transaction and garbage collection collide in a file system of an operating system that supports transactions.

An object of the present invention is to provide a technique for solving the problem that the size of a transaction in a file system of an operating system supporting transactions is limited by the size of a memory managed by the operating system.

According to the present invention, it is not necessary to pin the page cache to the memory, and thus the contents of the page cache can be written to the swap area. When a disk block with swapped content is dynamically moved using garbage collection, a new page cache entry (shadow page cache entry) is allocated without using the originally connected page cache, so that the contents of the disk block are replaced with a new one. move to position As a result, the atomicity of the transaction is guaranteed.

The present invention introduces and uses a shadow page cache to solve an error occurring in a situation where a transaction conflicts with garbage collection. The shadow page cache is distinguished from a page cache managed by an operating system for transactions.

In the present invention, a shadow page cache is used as a concept different from the page cache used in the prior art. A page cache that is not a shadow page cache may also be referred to as a regular page cache.

In addition, the present invention proposes an intermediate storage device and technique for executing large-capacity transactions. In the present invention, this may be referred to as stealing. The large-capacity transaction may refer to a case in which the amount of write data included in one transaction is greater than the amount of the page cache.

Applications can send various operation calls to the operating system. The Tx start call is a system call indicating that the application is starting a transaction. The write call is a system call indicating that a file is to be written to storage. The Tx end call is a system call indicating that a transaction is to be terminated.

When an application executes a write call after executing a transmit start call, write data (pages) related to the write call are pinned to a page cache, which is a memory used by an operating system. If all of the write data cannot be pinned to the page cache, that is, when the page cache is small, the prior art has a problem of canceling the transaction. Here, the prior art may refer to a technology in which an operating system supporting a transaction executes a transaction.

In the present invention, during a transaction, the operating system first intermediately stores some data (some pages) among write data stored in a page cache directly managed by the operating system in storage, and performs an evict or evict step. refers to Blocks of the page cache in which the previously intermediately stored write data are recorded are now in a usable state. Then, the operating system pins all remaining write data requested by the application through the transaction in the page cache. After that, when a send end call is called by the application, the operating system executes a command to write all the data stored in the page cache to the storage.

In the present invention, evict may mean selecting a specific page, writing the contents stored in the specific page to storage, and making the selected specific page a blank page so that it can be used for other purposes.

After the interim save is executed, if the commit of the transaction is completed, the interim save may be regarded as a final save of the part of the write data. That is, the intermediate storage may be regarded as final storage of the partial data (partial pages) for the storage.

The above-described intermediate storage is possible because the operating system of the present invention uses a log structure file system. In the log structure file system, when a certain data A recorded in storage is to be modified to A', A' is not written to the first block of the storage where A is recorded, but A' to another second block. has the characteristic of writing In addition, A written in the first block is invalidated at an appropriate time. The invalidation may be executed by an operating system using an invalidation command. Here, a block may mean a unit storage in which data is stored. A plurality of blocks may be collectively referred to as one segment. As such, in the case of a log structure file system, even if old data A already recorded in storage is modified with new data A', the old data A remains in the storage for a predetermined time. This point is important in the present invention.

File mapping information indicating a mapping relationship between a certain data A recorded in the storage and a location of a block in which the data A is stored is stored in a file called an inode. The inode file is also stored in storage and can be temporarily stored in the page cache managed by the operating system.

A block considered valid in one segment may be referred to as a live block. A block declared as a block in which certain valid data is stored by predetermined mapping information may be referred to as a live block. A process of moving data stored in all live blocks among blocks included in a first segment to another second segment and then making all blocks included in the first segment into usable blocks may be referred to as garbage collection. . The purpose of garbage collection is to free up free segments.

A file system of a conventional operating system that supports transactions is not a log structure file system. That is, according to the prior art, when some old data (= old page) recorded in the storage is updated with new data (= new page), the old data immediately disappears from the storage.

In contrast, in the present invention using a log structured file system, when updating certain old data recorded in the storage with new data, the fact that the old data can be maintained in the storage under a predetermined condition is used. .

The log-structured filesystem supports an operation called checkpointing. The checkpoint operation flushes the file mapping information (inodes) stored in the page cache to the storage.

In this regard, the file mapping information stored in the non-volatile memory of the storage may be cached in the cache of the storage, and the file mapping information stored in the cache of the storage may be stored in the memory managed by the operating system of the host, that is, the page cache. can be cached. In order for the operating system to modify the file mapping information stored in the storage, the file mapping information stored in the storage may be first cached in the page cache.

According to one aspect of the present invention, if the page cache is saturated while a transaction is being executed, some of the pages stored in the page cache, for example, page A, may be interimly stored in storage. That is, the page A can be Evicted. In this case, the old page of page A may be written to a first block of storage, and the new page of page A may be written to a second block of storage. In this case, when the intermediate storage is executed, new file mapping information indicating that the new page is written in the second block may be recorded in the page cache and the cache of the storage. In this case, the old file mapping information indicating that the old page is written in the first block disappears. If a checkpoint operation is called in this state, the new file mapping information is flushed to the storage, and as a result, the old file mapping information existing in the storage disappears. Then, if a system crash (eg, power failure) occurs before the transaction is committed, the new file mapping information recorded in the storage can be used in the process of recovery after the system crash, but the information that has already disappeared Old file mapping information cannot be used. Therefore, there is a problem in that even if recovery is executed, the previous state of the unfinished transaction is not restored. In order to solve this problem, the present invention introduces the following additional configuration.

According to one aspect of the present invention, when the page A is evicted, the new file mapping information for the page A may be pinned to the page cache until the transaction is committed. In this way, even if a checkpoint operation is called, the new file mapping information is not flushed, and thus, the old file mapping information in the storage can be maintained as it is until the transaction is committed.

Calls to checkpoint operations can be performed periodically by the operating system or just before and after garbage collection is called.

As described above, invalidation of an old block stored in storage is not performed before committing a transaction for a page on which an Evict is made.

<Transaction support in log structure file system-exF2FS>

In the present invention, exF2FS, a file system with a transaction log structure, is proposed. The proposed file system is Membership-Oriented Transaction. It can consist of three main components called Stealing-Enable Transaction and Shadow Garbage Collection.

The membership-oriented transaction allows transactions to span multiple files where applications can explicitly specify the files involved in the transaction.

The stealing-enabled transaction allows application programs to execute transactions with small amounts of memory and encapsulate many updates (eg, hundreds of files with a total size of tens of GB) into a single transaction.

If the shadow garbage collection is used, the log structure file system can perform garbage collection without affecting error atomicity of ongoing transactions.

The transactional support of exF2FS can be carefully tuned to meet the critical needs of application programs while minimizing code complexity and avoiding performance side effects.

Using exF2FS increases SQLite multifile transaction throughput by a factor of 24 compared to stock SQLite multifile transactions. Implementing compression as a filesystem transaction increases RocksDB throughput by 87%.

Modern applications strive to protect data in a crashconsistent way, often split into multiple file abstractions. Without proper transactional support from the underlying filesystem, application programs use complex protocols to guarantee transactional updates across multiple files, resulting in long write sequences and fsync(). Text editors like vim and emacs use atomic rename() to save updated files atomically.

For transactions that update multiple database files, SQLite, a library-based embedded DBMS, maintains separate journal files for each database file, resulting in excessive fdatasync() calls and massive write amplification. Compaction operations in modern LSM-based key-value stores, such as RocksDB, are kept merge-sorted in a separate journal file known as the manifest file. To ensure that compression operations are fail-atomic, the key-value storage engine flushes output files individually and flushes the global state of compression to a manifest file.

The filesystem's transaction support allows application programs to replace multiple fsync()'s for each output file and manifest file with a single filesystem transaction, rendering higher performance by eliminating redundant IO.

Despite the clear benefits of transactional support, challenges remain for operating systems and filesystems.

Successfully deploying a transactional support system requires finding the right balance between four requirements: ease of use, code complexity, degree of ACID support, and performance. Unfortunately, achieving one of these often comes at a cost to another. System-level support for transactions can be largely classified into four types: native operating system support, kernel-level file system, user-level file system, and transaction block device. As a first-class citizen of the operating system, it is ideal to support transactions; However, it requires major changes to the operating system. User-level filesystem transaction support uses a user-level DBMS to provide full ACID transactions. ACID support sacrifices performance. Transaction support in kernel-level filesystems can be further classified according to the degree of ACID support: full ACID semantics, ACD without isolation support, or AC without isolation and durability support. F2FS transactions only support atomicity and do not support isolation and durability.

Transactions in F2FS cannot span multiple files. Ironically, despite having minimal support for transactions, F2FS is the only filesystem to successfully roll out transactional support to the masses. F2FS's transactional support has a specific target application called SQLite. F2FS's atomic writes allow SQLite to implement transactions without a rollback journal file and eliminate excessive flushing overhead.

The present invention revisits the problem of providing filesystem level transaction support. In particular, we focus our attention on log-structured filesystems. Most of the prior art for transactional filesystems use a journaling filesystem as a basic filesystem. This prior art uses the journaling layer of the filesystem to provide transactional functionality. F2FS, a log-structured filesystem designed for flash storage, has recently become widespread on smartphone platforms and is starting to expand to cloud platforms. Few studies have dealt with transactional support in log structured filesystems. Although some prior studies deal with transaction support in log structured file systems, the prior studies are limited in terms of transaction support. In addition, the prior art does not support multi-file transactions, transaction stealing, and conflict processing between transactions and garbage collection.

In the present invention, the transaction support of the log structured file system with three design goals is presented. (i) Transactions must be able to span multiple files, including directories, (ii) Transactions must be able to handle large amounts of updates, and (iii) Transactions must not be subject to garbage collection runs. Each of these requirements seems trivial and essential from the point of view of an application program. Unfortunately, developing a transaction log structured filesystem that meets these simple and banal requirements is a trivial task requiring substantial changes to the underlying filesystem in terms of design and implementation; Development of a new transaction model, redesign of the file system's page reclamation procedure, and redesign of the garbage collection procedure. Few modern transactional filesystems address these essential requirements in transaction management with sufficient maturity.

In the present invention, we present a membership-oriented transaction model that allows transactions to span multiple files. In addition, we present stealing for file system transactions so that transactions can handle large transactions that may consist of hundreds of files with tens of GB of data. Also, shadow garbage collection is suggested so that garbage collection does not interfere with ongoing transactions.

The main effects of the present invention are as follows.

First, according to the Membership-Oriented Transaction of the present invention, a file system maintains a kernel object, which is a transaction file group that designates a set of files related to a transaction, including directories. Membership-oriented transactions allow applications to explicitly specify a file for a transaction.

Second, using the stealing of the present invention, atomicity of the transaction is guaranteed while allowing dirty pages of uncommitted transactions to be evict. In the present invention , a delayed invalidation and relocation record are proposed to realize stealing in a file system transaction. The deferred invalidation prevents old disk locations of exploited pages from being garbage collected until the transaction commits. The relocation record maintains undo and redo information for aborting and committing an evicted page, respectively.

Third, by using the shadow garbage collection of the present invention, it is possible to prevent the garbage collection module from making dirty pages of uncommitted transactions to be durable and recoverable at an early stage. Shadow garbage collection allows filesystems to perform garbage collection transparently to in-flight transactions.

In the present invention, this function is implemented in F2FS. In the present invention, the newly developed file system is referred to as an extended F2FS (exF2FS). exF2FS improves SQLite performance by a factor of 24 over stock SQLite and reduces write volume by 1/6 compared to SQLite's PERSIST journal mode. Improves RocksDB performance by 87% on YCSB Workload-A. Special care needs to be taken not to change the disk structure of the existing F2FS to allow exF2FS to mount the existing F2FS partitions.

<multi-file transaction>

Multi-file transactions are an essential part of modern software. The following are some examples of multi-file transaction methods currently in use.

[Keep Browsing History in Web Browser] Chrome browser keeps user browsing activities such as visited URLs, list of downloaded files, access history for each URL and list of most frequently visited URLs. Chrome keeps each of these as separate files and updates these files in a failure-atomic fashion. For failure-atomicity, Chrome uses SQLite to update these files rendering excessive IO. SQLite transactions are inefficient.

[Compression of LSM-based key-value storage] Compression is the process of merge-sorting multiple SSTables with overlapping intervals into a sequence of output files with non-overlapping intervals. The error atomicity of the compact operation calls fsync() on each output file and its parent directory and flushes the global state of the transaction to a special file called a manifest file. On YCSB's "load" workload, a single zip in RocksDB can produce up to 198 output files (over 200 fsync()'s) for a total of 13.3GB.

[Software installation] Updating or installing new software requires downloading and modifying hundreds of files and updating the relevant directories in a failure-atomic fashion. Partially completed installations or updates often result in system instability.

[Mail client] The MAILDIR IMAP format maintains mailboxes and messages as directories, respectively, and as files in directories. Email clients update message files and associated directories in a transactional fashion. Without transactional support in the underlying filesystem, mail clients manage mailboxes and messages in a transactional fashion using expensive atomic renames.

<Multi-File Transactions and SQLite>

SQLite is a serverless embedded DBMS widely used in a variety of applications, including mobile applications such as Android Mail and Facebook App, desktop applications such as Gmail and Apple iWork, and distributed file systems such as Luster and Ceph. These applications use SQLite to continuously manage updates to multiple files in an error-atomic fashion. To understand how SQLite can benefit from the underlying filesystem's transactional support, we will instrument the IO behavior of SQLite's multi-file transactions.

Using SQLite to manage data continuously can make application programs simpler, but SQLite's file-assisted journaling and page segmentation physical logging result in significant write amplification and excessive flushing. A single insert() in SQLite results in 5 fdatasync()s with a write() of 40KB. There have been several studies to improve the extreme IO inefficiency of SQLite transactions. All these efforts are limited to improving the IO overhead of transactions with a single database file.

SQLite organizes a multifile transaction into a single file transaction and a collection of several flushes to record the global state of the multifile transaction in the master journal file. SQLite implements multi-file transactions in the four steps listed below. Steps 1 and 3 are for updating the master journal file. Steps 2 and 4 are for executing a series of single file processing.

Figure 2 shows how each of these steps relates to IO operation through physical experiments. Here, the transaction consists of three inserts into three different database files.

Step 1: Initialize the master journal file. SQLite records the name of the journal file in the master journal file. Then, the master journal file (S1 in Fig. 2) and the updated directory are flushed to disk (S2 in Fig. 2).

Phase 2: Logging and database updates. SQLite writes an undo record to the journal file and updates the database file. Each file is updated in the same way as in a single database transaction (S3 in Fig. 2). In Figure 2, there are three S3s, each corresponding to a single insert().

Step 3: Delete the master journal file. As an indication of a successful commit, SQLite deletes the master journal file and makes the associated directory durable (S4 in Figure 2).

Step 4: Reset the log. SQLite resets and flushes the journal file (S5 in Fig. 2).

In FIG. 2, the X axis and the Y axis represent time and LBA, respectively. Here, in the present invention, three areas of F2FS, that is, a metadata area, a data area of the main area, and a node area of the main area are explicitly designated. When SQLite flushes dirty file blocks via fdatasync(), the underlying F2FS flushes not only the data blocks, but also the associated node blocks into the data area and node area, respectively. Flushing the master journal file (fd(mj)) at S1 in Figure 2 renders two separate 4KB IOs to disk. One for flushing data blocks and one for flushing node blocks. Data blocks and related node blocks must be durable to ensure the integrity of the filesystem. Each insert() has three fdatasync()(S3). The first and second fdatasync() are for flushing the rollback journal file. The third is to flush the database files. On S4, SQLite deletes the master journal file and retains the parent directory. If the disconnection of the master journal file persists, the transaction is committed. At S5, SQLite resets the transaction's rollback journal file.

As shown in Figure 2, the IO overhead of SQLite multi-file transactions is rather fatal. Inserting three 100-byte records writes 15 fdatasync()s and 180KB to disk.

<log structured filesystem, F2FS and atomic writing>

In the present invention, F2FS is used as a basic log structure file system. F2FS has several key design features that differentiate it from the original log-structured filesystem design. Among them, two functions focused on in the present invention are a block allocation bitmap and a dual log partition layout. To make stealing and shadow garbage collection a reality, you need to examine how F2FS manipulates and updates the block allocation bitmap and the two logs.

The first is the block allocation bitmap. The original log-structured filesystem design had no explicit data structures specifying whether a given block of a filesystem partition was allocated. The filesystem determines that a block on a filesystem partition is allocated if it can be reached via file mapping. F2FS maintains a block allocation bitmap to indicate whether a given block in the filesystem is valid.

The second is a dual log partition layout. The legacy log structure filesystem treats a filesystem partition as a single log. Cluster the data blocks and their associated filemap1 together and flush them as a single unit. F2FS organizes the filesystem partition into two separate logs: a data area and a node area. F2FS places data blocks and node blocks respectively in related areas. Unlike the legacy log structure file system, F2FS writes data blocks and node blocks separately. To maintain filesystem integrity against system crashes, F2FS checks if data blocks are durable before connected node blocks. Due to this ordering mechanism of F2FS, block traces for data block writes appear before block traces for node block writes in each pair of writes for data blocks and node blocks, as shown in FIG.

F2FS provides atomic write functionality. An application can write multiple blocks to a single file in an error-atomic fashion. This feature is primarily intended to address the excessive IO overhead of SQLite's single file transactions.

start_atomic_write(fd) ;

write(fd, block1) ;

write(fd, block2) ;

commit_atomic_write(fd) ;

For atomic writes, F2FS maintains a list of dirty pages in the inode. When a transaction updates a file block, it inserts the dirty page into the per-inode list of dirty pages and pins the dirty page into memory. When the transaction commits, the filesystem unpins the dirty pages from the per-inode list of dirty pages and flushes the dirty pages and associated node blocks that hold the updated file mapping to disk. Atomic writes pin dirty pages into memory until they are committed, so F2FS by design cannot support stealing in atomic write transactions. When a transaction is committed, F2FS sets the FSYNC_BIT flag on the node block. If more than one node block is flushed, the atomic write places the FSYNC_BIT flag on the last node block. F2FS sets the FSYNC_BIT flag on a node block to indicate that it is eligible for rollforward recovery.

Log structure Filesystems periodically checkpoint status such as updated file mappings, updated bitmaps (F2FS only), and the disk location of the last block of each log. When a filesystem crashes, the recovery module restores the state of the filesystem with respect to the most recent checkpoint information. After rollback recovery, the recovery module scans the log at the last location and finds node blocks with FSYNC_BIT, i.e. transactions successfully completed since the most recent checkpoint, and recovers the related files.

In the present invention, three constraints that a transaction log structured file system must satisfy are defined: (i) multi-file transactions, (ii) stealing, and (iii) transaction-aware garbage collection. The present invention proposes exF2FS, a file system with a transaction log structure that satisfies these constraints. The key technical components of exF2FS are membership-oriented transactions, stealing-enabled transactions, and shadow garbage collection. Each component is summarized below.

[Membership-oriented transaction] Transactions in F2FS cannot span multiple files because dirty pages of transactions are maintained in inode units. In the present invention, a new transaction model called membership-oriented transaction is developed. In membership-oriented transactions, the filesystem defines transactional filegroups, which are sets of files whose updates must be transacted, and maintains a transactional dirty page for each transactional filegroup. In membership-oriented transactions, transactions can span multiple files, and application programs can explicitly specify the files to which a transaction applies.

[Stealing-Enabled Transaction] In case of stealing, when the file system reclaims dirty pages of an uncommitted transaction, page reclamation is performed so that the page reclamation result can be undone. ) check the procedure. With stealing-enabled transactions, the proposed file system can support large-scale transactions, such as hundreds of files containing tens of GB of data, with a small amount of memory.

[Shadow Garbage Collection] Garbage collection can make dirty pages associated with uncommitted transactions durable, and it can early checkpoint updated file mappings before transactions are committed. Develop shadow garbage collection to decouple garbage collection from uncommitted transactions.

<Membership-Oriented Transaction Model>

The present invention proposes a new transaction model called membership-oriented transaction. This model defines a new kernel entity, the transactional filegroup. A transactional file group is a set of files for which updates must be processed as one transaction, and is composed of a transaction membership (inode set), a dirty page list, a relocation list, and a master commit block as shown in FIG. 3 . It uses a hash table for the namespace of transactional filegroup objects, which is widely used to construct the namespace of kernel objects such as semaphores and pipes.

Transactional filegroups allow applications to specify which files should be included in a transaction. A dirty page list is a set of dirty pages for transactional member files. The dirty page list has two separate dirty page lists: a dirty data page list and a dirty node page list. A relocation list is a set of relocation records. The relocation record contains information about the page that was evicted: file ID, file offset, old disk location and new disk location. The master commit block holds the disk location of the last node block for each file within the transactional membership. Using transactional filegroups with master commit blocks allows transactions to span multiple files. Relocation lists are used for stealing and shadow garbage collection.

<Transaction API>

An application creates a transactional filegroup with an explicit call. When an application creates a transactional filegroup, the ID of the transactional filegroup is returned to the application. Applications can add or remove files from a transactional filegroup. Prohibit applications from adding or removing files from transactional filegroups of ongoing transactions to prevent conflicts between in-flight transactions. When a transaction creates a file, the newly created file inherits its membership from the parent directory, called membership inheritance. Membership inheritance saves transaction-created files from transaction conflicts because newly created files are added to the transaction filegroup before they are visible to the outside world. When a directory is removed from a transactional filegroup, any child files that inherit membership are also removed from the transactional filegroup.

When an application starts a transaction, it specifies the ID of the transactional filegroup. When a transaction begins, the filesystem sets a flag in the transaction member file's inode indicating that the file is associated with an ongoing transaction. When an application calls transaction commit, it specifies the ID of the transactional filegroup. exF2FS provides APIs for transaction abort and transaction deletion. When an application requests to delete a transactional filegroup and there are no transactions in progress for the transactional filegroup, the transactional filegroup and related objects are deallocated. When an application requests to delete a transactional filegroup, if there is a transaction in progress, exF2FS first aborts the transaction and then deletes the transactional filegroup.

In exF2FS, transactions can include directory updates such as rename(), unlink() and create(). F2FS transactions do not support directory updates in transactions.

<commit and abort>

When a transaction updates a file in the transactional filegroup, it inserts the updated page cache entry into the list of dirty data pages in the transactional filegroup.

When committing a transaction, the filesystem prepares dirty data pages, dirty node pages, and master commit block for committing the transaction. First, the file system inserts dirty data pages from the dirty page list into the active data segment and obtains the disk location for each dirty data page. Second, the filesystem updates the associated node page with the new disk location of each data page, inserts the updated node page into the list of dirty node pages, and determines the disk location of each dirty node page. Third, the file system allocates a master commit block and stores the disk location of each node page in the list of dirty node pages in the master commit block. The filesystem then sets the FSYNC_BIT flag in the master commit block.

When these steps are complete, exF2FS flushes dirty data pages, dirty node pages, and master commit blocks. Master commit blocks become durable only after data blocks and node blocks become durable. The master commit block is a key component for manipulating dirty pages from multiple files into a single multi-file transaction. If the master commit block persists, the filesystem scans the relocation list and invalidates the old disk location of the relocation record.

When a transaction is aborted, all entries in the dirty page list are deleted and the dirty page list becomes empty. If the aborted transaction has evicted pages, the file mapping information is reverted to its original location based on the relocation record.

If the system crashes, the recovery module performs a rollback recovery and sets the filesystem state to the most recent checkpoint. exF2FS then performs a roll-forward recovery. The log is scanned starting at the last logging offset recorded in the checkpoint. When a master commit block is encountered, the recovery module examines it and identifies the node block disk location of the file in the transaction. The recovery module of exF2FS then uses stock F2FS' roll-forward recovery routines to recover the files associated with each node block. If the system crashes before the master commit block persists, the temporary state of the transaction in memory is completely lost. Through this recovery mechanism, exF2FS guarantees the atomicity and durability of transactions.

<Concurrency Control and Isolation>

Since it is not a full-fledged DBMS, coarse file unit concurrency control is used in the present invention. A file can belong to only one transactional filegroup at a time. When adding a file to a transactional filegroup, the application checks whether the file is already in another transactional filegroup. If the file is already in another transactional filegroup, add_tx_file_group returns an error.

The present invention leaves isolation support up to the application, as other transactional filesystems do. As a general-purpose filesystem, it is difficult to meet the requirements of all different levels of isolation in various application programs at the same time. Carefully consider that the filesystem's limited support for isolation is redundant at best, unless the isolation level supported by the filesystem matches well with the isolation level required by the application program. Isolation is not required for text editors, application installers, compression of git and LSM-based key-value storage. SQLite and MySQL implement different levels of isolation on their own. In these applications, the filesystem's limited support for isolation won't help much. TxFS supports isolation of “repeatable reads”. Too strong for text editors, too loose for some applications like SQLite's "Serializable Read". SQLite must implement "Serializable Read" isolation in its own database layer using shared locks, even when using TxFS as the underlying filesystem. Filesystem support for isolation has a cost. Experiments have shown that TxFS's isolation support renders a 10% performance overhead due to the overhead of creating shadow copies of pages updated in a transaction. However, one limitation introduced by the lack of isolation support is that different processes cannot concurrently add, delete, or rename files in a directory that is part of another process's transaction. Concurrent directory modification support is reserved for future work.

Hereinafter, a method of stealing in a file system transaction, which is provided according to an aspect of the present invention, will be described.

Stealing proposed in the present invention represents a buffer management policy that allows dirty page eviction of uncommitted transactions. The DBMS's steal policy and the operating system's (or filesystem's) page reclamation are other manifestations of the same essential behavior, such as evict dirty pages to disk and free up physical memory. am. The two share essential behavior, but are at different ends of the extreme. In the case of database transaction stealing, the DBMS prohibits the Evicted dirty page from being displayed to the outside (isolation) and undoes the steal in the case of a transaction abort (atomic). When the operating system reclaims a file-backed dirty page, the result of the page eviction is displayed externally and cannot be undone. In a journaling file system, the escaped page overwrites the old file block, and in the log structure file system, the old file block of the escaped page is accessed due to the file mapping update. will not be able to

<Stealing and file system>

Still support for existing transactional filesystems leaves room for significant improvement. TxFS, F2FS, Isotope and Libnvmmio do not support transaction stealing.

TxFS cannot support transaction stealing due to a fundamental design limitation. TxFS's support for transactions is built on top of EXT4 journaling. EXT4 journaling pins log blocks in memory until journal transactions are committed. EXT4 limits the size of journal transactions (256 MB by default). When the size of a journal transaction reaches the limit, the EXT4 journaling module commits the journal transaction. In EXT4, dirty pages associated with a single system call can be split into two or more journal commits. TxFS can damage the atomicity of a transaction by prematurely persisting the transient state of the transaction, so this should not happen. To ensure atomicity, TxFS aborts transactions when the transaction size exceeds the limit.

F2FS pins the transaction's dirty pages into memory until they are committed. F2FS aborts all outstanding transactions when the dirty pages of uncommitted transactions exceed a certain threshold (15% of the total physical page frame by default). An example related to this will be described with reference to FIG. 4 .

Referring to FIG. 4 , log blocks '1', '2', '3', '4', '5', and '6' may be pinned to the memory 13 . F2FS pins the transaction's dirty pages to the page cache 133 in memory until committed. F2FS suspends all pending transactions when the number of dirty pages of uncommitted transactions exceeds a specific threshold (ex: 6 blocks shown in FIG. 4).

A rectangle 133 shown in FIG. 4 may mean a page cache of a memory.

CFS supports stealing. However, CFS relies on a non-existent transactional block device for stealing support.

AdvFS supports stealing with commodity hardware. AdvFS uses writable file copies for transactional updates. When the transaction commits, the file map is updated to reference the updated file block that was written in the wrong way. Due to these characteristics, AdvFS can freely support stealing. However, transactions in AdvFS can fragment files as the filesystem deletes old file blocks each time a transaction commits. The file defragmentation overhead of AdvFS is not yet known. Analysis of AdvFS is limited because it is a proprietary filesystem and the transaction module's source code is not publicly available.

In FIG. 4 , the page cache 133 may exist in a memory managed by an operating system of a host, for example, DRAM. The storage 20 shown in FIG. 4 may be a separate device distinct from the host. Data stored in non-volatile memory in storage 20 may be cached in volatile memory in storage 20 . Again, the content cached in the volatile memory may be cached in the host's DRAM. In this case, the unit of caching may be referred to as a block in the storage 20 . The block is not only a unit of caching but also a unit of writing/reading of the storage 20 . Storage can also be thought of as an arrangement of blocks. A number is assigned to each block, and this number can be regarded as a disk location. The host may use a page cache, which is part of DRAM, when caching the content of the block. The caching unit of the page cache is the page. Thus, the contents of blocks of storage can be cached in pages of the page cache.

<Delayed invalidation and node page pinning>

In the present invention, stealing is activated in a file system transaction.

Hereinafter, a method of operating a log structured file system provided according to an embodiment will be described with reference to FIGS. 5 and 6 .

5 and 6, the rectangle shown within the memory 13 represents the page cache 133.

5 illustrates a page eviction method of a log structured file system provided according to an embodiment.

In this specification, a space in a memory in which data “X” is stored may be referred to as “page”, and a page in which data “X” is stored may be referred to as “page X”. The “page X” may refer to data “X” stored in page X depending on the context. The memory may refer to a memory (ex: DRAM) in the host managed by an operating system of the host. In this specification, a space in storage in which data “X” is stored may be referred to as “block”, and a block in which data “X” is stored may be referred to as “block X”. The “block X” may refer to data “X” stored in block X depending on the context. In this specification, depending on the context, “data X”, “page X”, and “block X” may all mean “data X”. In this specification, depending on the context, “page X” may mean a page in which data X is stored, and “block X” may mean a block in which data X is stored.

The log structure file system evicts dirty pages as follows: the evicted page is written to the new disk location 232, the old disk location 231 of the evicted page is invalidated, and the file mapping ( The node page of F2FS) is updated to reference the new location 231 of the associated file block.

The page evict method (page evict routine) cannot be used with the stealing provided in accordance with one embodiment of the present invention for two important reasons.

The first is the invalidation of the old disk location 231. If the old disk location 231 is invalidated, the old file block 231 can be garbage collected and recycled before the transaction is committed. If the previous file block 231 is recycled before the transaction is committed, the transaction cannot be undone when the transaction is aborted.

The file blocks and node pages shown in storage 20 in FIG. 5 may be in volatile memory or non-volatile memory in storage 20 . A block obtained during a transaction may be written to the non-volatile memory of the storage 20 even before the transaction is committed.

5 shows a problem that occurs when the above-described EVICT process is applied to the prior art. When a transaction is executed, the transaction may modify data A to data A'. At this time, the contents of data A' may be cached in a volatile memory in the storage 20, and this caching is to hide the low speed of the storage 20. Contents cached in the volatile memory in the storage 20 may be cached once more in the volatile memory managed by the host. This caching is to reduce the effort of the host to access the storage 20. The host can modify the content cached in the host memory (= memory managed by the host, for example, DRAM) to A'. A' can be pinned to the host memory by the above transaction. When a situation in which the host memory is insufficient occurs, data A' may be transmitted to the storage 20 by ejecting page A' as shown in step S12 of FIG. 5 . The transmitted content may be stored in a volatile memory of the storage 20 . Through the evict process, an empty space is secured in the host memory. Data A′ transmitted to the storage 20 may be written to the non-volatile memory in the storage 20 when the host calls a flush command or when the volatile memory in the storage 20 runs out of free space. Data A' is written to block 232 instead of block 231 by the above Eviction. And the host can treat the data written to block 231 as invalidated data, so that block 231 can then be cleaned by garbage collection. Step S13 of FIG. 5 may be executed in a state in which commit is not called during the transaction. That is, when the host tries to write data B to the storage 20, a situation may occur where the non-volatile memory in the storage 20 runs out of space. At this time, the host can know that the data recorded in block 231 is invalid. Thus, data B can be written to block 231. As a result, data A recorded in block 231 is changed to data B, and as a result, data A is lost. Now, if a transaction is cancelled, a problem arises in that the data recovered due to cancellation is B, not A.

6 illustrates a checkpoint method of a log structured file system provided according to an embodiment.

The second is a premature checkpoint of updated node pages. If the dirty page is evicted, the updated node page 135 including the updated file mapping information may be checkpointed when the file system executes a periodic checkpoint operation before committing the transaction. Then, the updated node block 235 checkpointed to disk 20 refers to the new disk location (A:2) of the evicted page of the uncommitted transaction. If the file system crashes before the transaction is committed, the recovery module recovers the escaped page (A') of the uncommitted transaction in relation to the most recent file mapping information found on the disk 20 (S241) can do. As a result, it is possible to recover the filesystem to an invalid state.

There are two main issues that need to be addressed to support the stealable transactions of the present invention in a log structured filesystem: The first issue is to prevent the old disk location 231 from being garbage collected until the transaction is committed. The second issue relates to preventing the recovered block 232 of an uncommitted transaction from being recovered after a system crash. System crashes can occur across both hosts and storage.

In the present invention, a delayed invalidation method is proposed to solve the first issue. The delayed invalidation method is described with reference to FIG. 7 .

The rectangle shown in the memory 13 in FIG. 7 represents the page cache 133 .

In the delayed invalidation method, after ejecting dirty pages in an uncommitted transaction, the file system does not invalidate the old disk location 231 until the transaction is committed.

In the present invention, to solve the second issue , a node page pinning method is proposed. The node page pinning method will be described with reference to FIG. 8 .

The rectangle shown in the memory 13 in FIG. 8 represents the page cache 133 .

In the node page pinning method, the file system pins the updated node page 135 into memory until the transaction is committed to prevent the updated node page 135 from being prematurely checkpointed.

Hereinafter, in the drawings of this specification, an icon representing pinning is displayed in the pinned page cache. For example, the pinning icon is displayed on the upper right of the rectangle indicated by reference numeral 135 in FIG. 8 and reference numeral 133 in FIG. 12 .

For the delayed invalidation method and the node page pinning method, a new in-memory object, a relocation record, is introduced. Relocation records hold information related to page evict. The relocation record contains the file block ID (inode number and file offset), the old disk location 231 and the new disk location 232 of the file block of the evicted page. When the relocation record is used, the file system asynchronously invalidates the old disk location 231 when the transaction is committed, not when the dirty page is ejected (S33).

The relocation record may be written to DRAM of the host. There is no problem even if the relocation record disappears due to a system crash. The reason is that when rebooting after a system crash, the node page and block bitmap are not changed, so they are restored to their pre-transaction state.

Each transactional filegroup maintains a set of relocation records called a relocation list . The filesystem creates a relocation record and adds it to the relocation list when a transaction evicts a dirty page.

9 shows an example of executing stealing in exF2FS.

The rectangle shown in the memory 13 in FIG. 9 represents the page cache 133 .

The dirty page of file block A is initially mapped to LBA 1 (S51). File block A is evict to LBA 8 (S52). The node page 250 of the memory 13 is updated to map file block A to LBA 8 (S53). The block bitmap 260 of LBA 8 is set (S54). The block bitmap 260 for LBA 1 is not invalidated upon eviction due to delayed invalidation. The file system creates a relocation record 251 and inserts the newly created record 251 into the relocation list 250 (S55). The newly created relocation record 251 includes the file block ID (file block A), old location (LBA 1) and new location (LBA 8) of the evicted block. As LBA 1 is evict from disk 20, it is removed from the dirty page list of the associated transactional filegroup. When the transaction is committed (S56), the LBA is invalidated (S57) and the updated node page is continued (S58).

<Commits and aborts in stealing>

When the transaction is committed (S56), the file system makes the previous location (LBA 1) of the evict page unreachable any more. Before starting the dirty page flush, the file system searches the relocation list 270 in chronological order and invalidates the old disk location (LBA 1) of the evict block (delayed invalidation) (S57). When this operation is completed, the dirty data page of the transaction is flushed (S59). When the dirty page becomes durable, the file system unpins the node page 250 updated by eviction and inserts it into the dirty node page list. The filesystem then flushes the dirty node pages. A transaction commits successfully only if the master commit block becomes durable.

Alternatively, if the file system aborts the transaction, the file system searches the relocation list 270 in reverse chronological order. For each relocation record, the file system invalidates the new disk location (ex: LBA 8) and returns the node page 250 of memory 13 to map the file block to the old disk location (LBA 1). After returning the node page 250, pinning is unpinned.

10 shows an example of this.

At the time of transaction abort (Tx Abort), three pages 'A', 'E', and 'F' have already been evicted. The old and new positions of page 'A' correspond to '1' and '8', respectively. Upon abort, the file system returns the node pages 250 for 'A', 'E' and 'F' to reference pages '1', '10' and '11', respectively, according to the relocation list 270. It also invalidates the bitmaps 260 for the new disk locations LBA 8, LBA 29 and LBA 32.

Lazy invalidation in the event of a system crash can leave filesystem blocks that are allocated but unreachable. Delayed invalidation temporarily leaves both the old disk location 231 and the new disk location 232 valid from the time the page is evict until the transaction is committed. If the system crashes during this period, the file system can be restored in a state where both the old disk location 231 and the new disk location 232 are valid, but only the old disk location 231 is mapped to a file. In this case, the new disk location 232 must be collected via fsck (offline) or online transformation.

<Transaction Aware Garbage Collection>

When the garbage collection module selects a disk block (ex: reference numeral 313 in FIG. 1) associated with a dirty page (ex: A' of the page cache 133 in FIG. 1) of an uncommitted transaction as a migration target, garbage collection It says it conflicts with the transaction.

In the present invention, a transaction-aware garbage collection technology called shadow garbage collection is proposed. Shadow garbage collection transparently migrates victim blocks associated with uncommitted transactions without any side effects on the transaction. F2FS performs garbage collection in a transaction-aware manner, but there is considerable room for improvement. F2FS aborts all outstanding transactions when garbage collection collides with uncommitted transactions in the system.

<Garbage collection and transactions>

The log-structured filesystem performs garbage collection either in the foreground or in the background. Background garbage collection runs only when the filesystem is idle, so it cannot conflict with transactions. In the present invention, garbage collection implicitly indicates foreground garbage collection unless otherwise specified. The log structure file system performs garbage collection as follows. (i) First, the file system checkpoints the file system state (pre-GC checkpoint). (ii) The garbage collection module then selects a victim segment. (iii) Next, the garbage collection module migrates the valid blocks of the victim segment to the target segment (=current segment). This updates the associated file mapping information to refer to the victim block's new disk location. (iv) Finally, the filesystem checkpoints the updated state of the filesystem (post-GC checkpoint). The garbage collection module repeats steps (ii) and (iii) until it has reclaimed enough free segments. The spare segment may refer to a victim segment for which garbage collection has been completed. Pre-GC and post-GC checkpoints are essential for maintaining consistency in all log structured filesystems in preparation for unexpected filesystem failures.

In the pre-GC checkpoint step, the file system state in which the checkpoint is executed may be file mapping information. In the pre-GC checkpoint step, mapping information cached in a memory (eg, DRAM) of the host may be written to non-volatile memory in storage.

On F2FS and other log structured filesystems, the garbage collection module uses the page cache to migrate victim disk blocks to new locations. When migrating a victim block, the garbage collection module first checks whether the victim block is in the page cache. There can be only one page cache entry for a single disk block. If the associated disk block is already in the page cache, the old data block cannot be fetched as a page cache entry. If a page cache entry for the victim block exists, the garbage collection module does not fetch the victim block from disk and blindly writes the existing page cache entry to the target block. During this process, the garbage collection module can write dirty page cache entries of uncommitted transactions to the target. After the garbage collection module migrates the victim disk blocks to the target segment, the mapping of the associated file in memory is updated to reference the new disk location. When the migration is complete, garbage collection performs a checkpoint to make the filesystem's state sustainable. As a result, the updated file mapping referencing the new disk location of the victim block (the dirty page of the uncommitted transaction) persists before the transaction commits. If the system crashes after garbage collection is complete but before the transaction is committed, the recovery module recovers the dirty pages of uncommitted transactions. Then the atomicity of the transaction is broken.

<Shadow Garbage Collection>

In the shadow garbage collection method provided according to an aspect of the present invention, a set of page cache items is reserved for garbage collection. This area is referred to as a shadow page cache. When a victim block is associated with an uncommitted transaction, the garbage collection module uses the shadow page cache instead of the normal page cache to migrate the victim block and related node blocks as targets. By using the shadow page cache when migrating victim blocks to target segments, the filesystem prevents garbage collection from prematurely retaining dirty pages of uncommitted transactions.

11 shows an example of shadow garbage collection. The disk blocks A ₁ , A ₂ , and B ₁ are updated to A' ₁ , A' ₂ , and B' ₁ respectively in the page cache of the memory 13 . At this time, A ₁ and A ₂ are being modified by a transaction, and B' ₁ is independent of the transaction. A' ₁ and A' ₂ are pinned to the memory 13 because they are processed by the transaction, but B' ₁ may not be pinned to the memory 13 because it is not related to the transaction.

In FIG. 11 , the page cache 133 and the shadow page cache 130 are separately presented in the memory 13 .

At this time, the garbage collection module selects disk blocks A ₁ , A ₂ , and B ₁ as victim blocks. To migrate A ₁ and A ₂ in shadow garbage collection, the garbage collection module brings A ₁ and A ₂ (original versions before update) to the shadow page cache 130 and flushes them as target blocks. For the B ₁ migration, the garbage collection module is not associated with a transaction, so it uses a normal page cache. You can then write B' ₁ (an updated version of B ₁ ) to the target segment.

Reference number 1320 in FIG. 11 indicates a set of data to be written to the target segment (= active segment). Since the data B ₁ is not related to the transaction, there is no problem even if the garbage collection does not migrate the data B ₁ to the target segment and writes a non-B' ₁ . In the shadow garbage collection step of FIG. 11, the reason for migrating the old data A ₁ rather than the new data A' ₁ is that the old data of transactions already in progress must be maintained in the storage. On the other hand, if the data is in a file that is not related to the transaction, there is no problem even if the old data (B ₁ ) is lost.

Garbage collection can collide with uncommitted transactions in two ways. (i) A victim block may be associated with a page that has been evicted by stealing (Type E, Evicted) and (ii) a victim block may be associated with a cached page (Type C, Cached). )). If the victim block is associated with an evicted page, the original file block before update (type EO, Evicted and Old) or the updated file block (type EN, Evicted and New) may correspond to If the victim block is associated with a cached page, the victim block corresponds to the original file block before the update (type CO, cached and old). A victim block of type CN (Cached, New) cannot exist.

For each type of victim block, shadow garbage collection elaborates different mechanisms when migrating the victim block and associated node blocks.

[Type CO] If the victim block (LBA 1) corresponds to the previous (O) version of the uncommitted transaction's cached block (C) (A'), migrate the victim block (LBA 1) and update the node block ( A:8) to the new disk location 137 using the shadow page caches 131 and 132. When updating and saving the associated node block, shadow garbage collection updates the node block (A:1) 136 read from disk, not the node block that was already in the page cache. Node blocks in the page cache may have been updated since being read from disk and may contain temporary file mappings that should not be persisted. After the garbage collection module finishes migrating both the victim block (LBA 1) and the updated node block (A:8), it updates the node page in the page cache with the new file mapping. If a transaction is interrupted or the system crashes, the victim block in the migrated location can be recovered using the updated node blocks stored on disk.

An example of this can be seen in FIG. 12 . File block A was in LBA 1 and was updated to A' in the page cache 133 of the memory 13 (S511). Disk block LBA 1 is selected as the victim. File block A is migrated to LBA 8 using the shadow page cache 131 (S512). The node block (A:1) related to the file block A is read into the shadow page cache 132 and updated to be mapped to LBA 8 (S513). Then, the updated node page (A:8) 132 is flushed to the disk 137 (S514). After both the victim block (A) and the node block (A:8) are flushed, the node page 139 of the page cache 133 is updated and mapped to LBA 8 (S515). The node page 139 is related to file block A.

[Type EO] If the victim block (LBA 1) corresponds to the previous (O) version of the evicted page (E) (A'), migrate the victim block (LBA 1) and update the node page (A:8) The shadow page caches 131 and 132 are used to store to the new disk location 137. According to the characteristic of the evict process proposed in the present invention, a page cache entry (data page) associated with the evicted page (A') does not exist in the page cache. The node page (A':4) associated with the evicted page (A') is pinned to the memory 13 until the transaction is committed due to node page pinning. When migrating a victim block of type EO, the filesystem uses the shadow page cache to migrate the victim block. The nodeblock update uses the on-disk version of the nodeblock (A:1) 136 as in the case of CO victim block migration. After the shadow garbage collection module completes the migration of node blocks, it updates the pinned node blocks in memory with the updated file mapping.

Using the node block of the on-disk version means using a node block existing in the storage 20 .

An example of this is shown in FIG. 13 . The dirty page (A') of the uncommitted transaction is Evicted to LBA 4 (S521). File block A is migrated from LBA 1 to LBA 8 (S522). Shadow page caches 131 and 132 are used to migrate victim blocks and related node blocks. The node block for file block A is updated from "A:1" to "A:8" (S523), and then flushed to disk (S524). Mapping information (A':4) about the location (LBA 4) of the dirty file block of the evicted page (A') is pinned to the node page 135 of the page cache 133 (S525).

[Type EN] When migrating the victim block (LBA 4) to a new disk location (LBA 8) if the victim block (LBA 4) corresponds to the new (N) version of the evicted page (E)(A'). Use the normal page cache. Since the victim block (LBA 4) holds the latest copy (A') of the file block, the normal page cache, rather than the shadow page cache, can be used when migrating the victim block (LBA 4). The garbage collection module updates the node page in the page cache with the new file mapping (A':8) after migrating the victim block (LBA 4) to the new location (LBA 8).

An example is shown in FIG. 14 . During the transaction, page A' is evicted to block LBA 4 (S531), and at this time, the node page of the page cache 133 for page A' is updated to "A':4". The updated file block of the evicted page A' is migrated from LBA 4 to LBA 8 (S532). Here, the normal page cache 133 (not the shadow page cache) is used to migrate the victim block (LBA 4). After the migration is completed, the garbage collection module updates the page of the relevant node in the page cache 133 from "A':4" to "A':8" (S533).

When garbage collection migrates the disk blocks associated with the evicted pages, garbage collection updates the relocation record after the migration is complete. If the victim block is associated with the new disk location and the old disk location of the evicted page, it updates the new disk location field and the old disk location field of the relocation record, respectively.

When implementing shadow garbage collection, you can use Linux's existing META_MAPPING object as the shadow page cache. META_MAPPING is a special-purpose address_space object, which is dedicated to caching filesystem metadata. By utilizing the existing META_MAPPING object as the shadow page cache, shadow garbage collection does not require a new data structure for the shadow page cache in the kernel. Garbage collection in exF2FS (and F2FS) reclaims usable blocks at segment-granularity. The memory overhead of shadow garbage collection corresponds to the size of a single segment, 2 MB.

According to one aspect of the present invention, the operating system of the host, starting garbage collection and stopping the first transaction before the first transaction is committed; a garbage collection execution step of, by the operating system, migrating data of a victim block selected by the garbage collection to a target block; and resuming, by the operating system, the first transaction when the garbage collection is terminated. If the data of the victim block is old data (ex: A) of the first data (ex: A') pinned to the page cache by the aborted first transaction and stored in storage, the migration uses the shadow page cache. and the shadow page cache is another cache distinct from the page cache.

In this case, the executing garbage collection may include migrating, by the operating system, data of the victim block to the target block by using the shadow page cache.

In this case, the garbage collection execution step may include, after the migration step, caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache; updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage.

In this case, if the data of the victim block is the old data of the first data pinned to the page cache by the interrupted first transaction, the garbage collection execution step may cause the operating system to convert the updated file mapping information to a page A step of pinning to the cache may be further included.

At this time, if the data of the victim block selected by the garbage collection is old data of the first data stored in the storage by the interrupted first transaction, the garbage collection method, before the first transaction is interrupted, the pinning, by an operating system, the first data designated by an application to a first page of the page cache; The operating system determines whether a space for pinning the second data designated by the application exists in the page cache, and if it is determined that there is no space for pinning the second data in the page cache, the first data is stored in the storage. and pinning the second data to the first page.

At this time, the file system of the operating system may be a log structure file system.

In this case, the garbage collection is performed on a separate storage distinct from the host, and the victim block and the target block may be blocks existing in the storage selected by the garbage collection.

According to one aspect of the present invention, a memory; A host including a; and a processing unit executing an operating system may be provided. starting, by the operating system, garbage collection before the first transaction is committed and stopping the first transaction; a garbage collection execution step of migrating data of a victim block selected by the garbage collection to a target block; and resuming the first transaction when the garbage collection is terminated. If the data of the victim block is old data (ex: A) of the first data (ex: A') pinned to the page cache by the aborted first transaction and stored in storage, the migration uses the shadow page cache. and the shadow page cache is another cache distinct from the page cache.

In this case, the executing the garbage collection may include migrating, by the operating system, data of the victim block to the target block by using the shadow page cache.

In this case, the execution of garbage collection may include caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache; updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage.

At this time, if the data of the victim block is the old data of the first data pinned to the page cache by the interrupted first transaction, the garbage collection execution step is performed by the operating system, the updated file information is stored in the page cache A step of pinning may be further included.

At this time, if the data of the victim block selected by the garbage collection is the old data of the first data stored in the storage by the interrupted first transaction, before the first transaction is interrupted, the operating system, by the application pinning the designated first data to a first page of the page cache; and determining whether a space for pinning the second data designated by the application exists in the page cache, and if it is determined that there is no space for pinning the second data in the page cache, storing the first data in the storage. , pinning the second data to the first page; may be further executed.

In this case, the shadow page cache and the page cache are included in the memory, and the memory may be a volatile memory managed by the operating system.

According to one aspect of the present invention, starting a garbage collection and stopping the first transaction before the first transaction is committed by an operating system of the host; a garbage collection execution step of migrating data of a victim block selected by the garbage collection to a target block; and restarting the first transaction when the garbage collection is terminated; a computer-readable non-volatile storage medium having a software program recorded thereon, including instructions for executing the first transaction.

In this case, the execution of garbage collection may include caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache; updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage.

According to another aspect of the present invention, when the size of the page cache is smaller than the size of a plurality of write pages belonging to the first transaction, the operating system of the host writes some of the write pages to the storage. (evict) step; and a commit step of committing the first transaction by the operating system.

In this case, the data fixed to the page cache immediately before the commit is executed may be other write pages excluding some of the write pages among the plurality of write pages.

In this case, the commit step may be executed when the operating system receives a commit call of the first transaction from the application program.

In this case, the transaction execution method may include, before the evict step, receiving, by the operating system, a set of write operation calls for the plurality of write pages from an application program; and receiving, by the operating system, a commit call of the first transaction from the application program between the evict step and the commit step.

At this time, the file system of the operating system may be a log structure file system.

At this time, the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage, and the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage. By the vict step, the first set of new blocks are stored in the storage, and the operating system does not invalidate the first group of old blocks before the commit is executed. may be prevented.

In this case, the operating system may be configured to invalidate the old blocks of the first group simultaneously with execution of the commit or after execution of the commit.

At this time, the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage, and the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage. By the vict step, a first set of new blocks is stored in the storage, and before the vict step is executed, the old data and the first group of old blocks are stored in the storage. Old mapping information that maps them to each other may be recorded, and the operating system may be configured not to remove the old mapping information recorded in the storage before the commit is executed.

At this time, the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage, and the data of the partial write page is data that replaces old data stored in a first set of old blocks in the storage. A first set of new blocks in the storage is stored in the vict step, and in the vict step, the operating system determines that some of the write pages and the first group of write pages are stored in the page cache. It may include storing new mapping information indicating a mapping relationship between new blocks of .

In this case, the operating system may be configured not to perform a checkpoint operation on the new mapping information stored in the page cache before the commit is executed. Alternatively, a checkpoint operation for the new mapping information may be executed simultaneously with execution of the commit or after execution of the commit.

According to another aspect of the present invention, starting a first transaction including first data and second data by an operating system of a host; Before the operating system receives a commit call for the first transaction from an application, the first page is pinned at a first page of a page cache of a memory. an evict step of writing data to a first new block of storage; pinning, by the operating system, the second data at the first page; and a commit step of committing the first transaction by the operating system.

In this case, the transaction execution method may further include, before the evict step, receiving, by the operating system, a set of write operation calls for a plurality of data belonging to a first transaction from an application program. . The Evict step may be executed only when the size of the page cache is smaller than the total size of the plurality of pieces of data.

In this case, in the transaction execution method, between the evict step and the commit step, the operating system transmits first file mapping information indicating that the first data is mapped to the first new block to the first block of the memory. pinning a first file mapping information indicating that the first data is mapped to the first new block at a first node page of the memory (S53); Modifying, by the operating system, a block bitmap 260 so that the first new block is in an in-use state (modifying a block bitmap 260) ; The operating system generates a first relocation record 271 including an identifier of the first data, a location of a first old block that is an old block of the first data, and a location of the first new block of the first data. generating; and inserting, by the operating system, the generated first relocation record into the relocation list 270 .

In this case, the transaction execution method may include, between the inserting step and the committing step, checking the location of the first old block of the first data by searching the relocation list, by the operating system; invalidating, by the operating system, the identified first old block of the first data; flushing, by the operating system, dirty pages of the page cache; unpinning the first node page, by the operating system, if the dirty pages are determined to be durable; inserting, by the operating system, the first node page into a dirty node page list; and flushing the dirty node page list by the operating system.

In this case, the transaction execution method may include, if the first transaction is aborted, checking, by the operating system, the location of the first new block of the first data by searching the relocation list; invalidating, by the operating system, a first new block of the first data; recording, by the operating system, file mapping information indicating that the first data is mapped to the first old block in the first node page; and unpinning, by the operating system, the first node page.

In this case, the first transaction may be committed only when the commit block is determined to be sustainable.

Memory according to one aspect of the present invention; A host including a; and a processing unit executing an operating system may be provided. The operating system may be configured to execute each step included in the transaction execution method described above.

According to the present invention, it is possible to provide a technique for solving errors that occur when a transaction and garbage collection collide in a file system of an operating system that supports transactions.

According to the present invention, it is possible to provide a technique for solving the problem that the size of a transaction in a file system of an operating system supporting transactions is limited by the size of a memory managed by the operating system.

1 illustrates an embodiment of performing garbage collection according to the prior art.
Figure 2 shows the relationship between steps performed by SQLite to execute a multi-file transaction and IO operations.
3 shows the configuration of a transactional file group.
4 shows an example in which F2FS aborts an incomplete transaction.
5 illustrates a page eviction method of a log structured file system provided according to an embodiment.
6 illustrates a checkpoint method of a log structured file system provided according to an embodiment.
7 is for explaining a delayed invalidation method provided according to an embodiment of the present invention.
8 is for explaining a node page pinning method provided according to an embodiment of the present invention.
9 shows an example of executing stealing in exF2FS provided according to an embodiment of the present invention.
10 illustrates a method of handling a transaction suspension situation using a relocation list according to an embodiment of the present invention.
11 shows an example of a shadow garbage collection execution method provided according to an embodiment of the present invention.
12 illustrates a method of migrating a victim block when a victim block of garbage collection corresponds to a previous version of a cached block of an uncommitted transaction, according to an embodiment of the present invention.
13 illustrates a method of migrating a victim block for garbage collection when the victim block corresponds to a previous version of an evicted page, according to an embodiment of the present invention.
14 illustrates a method of migrating a victim block for garbage collection when the victim block corresponds to a new version of an evicted page.
15 illustrates a concept in which a host writes information to a storage according to an embodiment.
16 shows different transactions executed by the host.
17 is a flowchart illustrating a method of committing a multi-file transaction provided according to an embodiment of the present invention.
18 shows the structure of a master commit block provided according to one aspect of the present invention.
FIG. 19 illustrates the relationship between MCB and inode information stored in storage by the multi-file transaction commit method described in FIG. 17 .
20A and 20B are flowcharts illustrating a method of interimly storing transaction data for large-capacity transactions according to an embodiment of the present invention.
21 illustrates an example of a method of executing garbage collection according to an embodiment of the present invention.
22 illustrates another example of a method of executing garbage collection according to an embodiment of the present invention.
23A and 23B are flowcharts illustrating a method of executing garbage collection according to an embodiment of the present invention.
24A and 24B are flowcharts illustrating a method of executing garbage collection according to another embodiment of the present invention.
25A and 25B are flowcharts illustrating a method of executing garbage collection according to another embodiment of the present invention.
26 is a flowchart illustrating a method of executing garbage collection according to another embodiment of the present invention.
27 illustrates a configuration of a host provided according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein and may be implemented in various other forms. Terms used in this specification are intended to aid understanding of the embodiments, and are not intended to limit the scope of the present invention. Also, the singular forms used herein include the plural forms unless the phrases clearly dictate the contrary.

15 illustrates a concept in which a host writes information to a storage according to an embodiment.

The host 10 and the storage 20 may each be a computing device that operates by supplying power from a power supply. The host 10 and the storage 20 may exchange data and commands through one or more transmission channels 30 . The transmission channels 30 may be wireless channels or wired channels. The host 10 and the storage 20 may share power provided from one power supply or receive power from two different power supplies.

The host 10 may include a CPU, a memory, a power supply, and a communication device.

The storage 20 may include a controller 21 , a volatile memory 22 , and a non-volatile memory 23 .

The host 10 may transmit various commands and data to the storage 20 through the transmission channels 30 . The command may include a write command.

The controller 21 of the storage 20 may store data received from the transmission channels 30 in the volatile memory 22 based on commands received from the transmission channels 30 . Data stored in the volatile memory 22 may be stored in the non-volatile memory 23 according to a rule followed by the controller 21 . Data stored in the volatile memory 22 can be deleted when power supplied to the storage 20 is cut off, but data stored in the nonvolatile memory 23 is deleted even if power supplied to the storage 20 is cut off. It doesn't work.

The host 10 may execute an application 11 and an operating system 12 . The application 11 and the operating system 12 may be executed by executing predetermined command codes stored in a memory accessed by the host 10 by a CPU included in the host 10 .

In one embodiment, the application 11 may be a program that is executed or terminated when a user using the host 10 provides a user input through a user interface provided by the host 10 .

In one embodiment, the operating system 12 may be a program automatically executed by the host 10 when power is applied to the host 10 or a reset is performed.

The application 11 may send various system calls to the operating system 12 . The operating system 12 may execute a task corresponding to the system call.

The host 10 may execute transactions. It may take a certain amount of time from the start of a particular transaction to its end.

The application 11 can control the start and commit of a transaction. In addition, the application 11 may control one or more operations to be executed during the transaction. The application 11 may transmit system calls including a start call, a set of operation calls, and a commit call to the operating system 12 .

A specific transaction is started by the start call, commands to be transferred from the host 10 to the storage 20 are prepared by the set of operation calls, and the prepared commands are prepared by the commit call. It can be delivered to the storage 20 through the transmission channels 30 .

16 shows different transactions executed by the host.

A set of multiple operations may constitute one transaction.

The first transaction 41 may include four write operations (WO#1 to WO#4). Although FIG. 16 shows an example in which only write operations are included in each transaction, other types of operations may also be included.

17 is a flowchart illustrating a method of committing a multi-file transaction provided according to an embodiment of the present invention.

In step S110, the application 11 may transmit a start call to the operating system 12. Thus, the first transaction may be started. Upon receiving the start call, the operating system 12 may start a process for the first transaction.

In step S121, the application 11 may call a write operation call (WO#1) for the first page of the first file to the operating system 12.

In step S122, the application 11 may call the write operation call (WO#2) for the first inode of the first file to the operating system 12.

In step S131, the application 11 may call the write operation call (WO#3) for the second page of the second file to the operating system 12.

In step S132, the application 11 may call a write operation call (WO#4) for the second inode of the second file to the operating system 12.

In step S140, the application 11 may call a commit call to the operating system 12.

In step S150, the operating system 12 may process the first transaction in response to the commit call. That is, pages of all files included in the first transaction may be reflected in the storage 20 .

Step S150 may include steps S151 to S158.

In step S151, the operating system 12 may create one MCB (Master Commit Block) having a structure according to an embodiment of the present invention.

18 shows the structure of a master commit block provided according to one aspect of the present invention.

In one implementation, the total size of the master commit block 300 is N bytes (ex: N=4,096).

At this time, the number of inode block positions of the master commit block 300 is stored in the uppermost N1 byte 301 (ex: N1=4). In the remaining {N-N1} byte space of the master commit block 300, N/N1-1 block addresses having a size of N1 bytes may be stored.

In each part 302, 303, 304 of the master commit block 300, block positions (block numbers) of inodes written by the multi-file transaction according to the first transaction may be stored.

Here, the block location of each of the inodes may be an address indicating a location of a block in which the inode is stored in the storage 20 .

For example, the block position of the first inode of the first file (File#1) 401 can be stored in the first part 302, and the block position of the first inode of the first file (File#2) 402 can be stored in the second part 303. The block position of the second inode of ) may be stored. Here, the first file (File#1) 401 and the second file (File#2) 402 are files included in the first transaction.

Instead of attaching the FSYNC_BIT flag to the inode block, the FSYNC_BIT flag (307) can be attached to the master commit block.

The ordering between inodes and transaction contents may not be guaranteed either.

Instead, the order between the entire transaction including the inode and the master commit block 300 can be guaranteed.

When a crash occurs in the log structure file system, the master commit block 300 with the FSYNC_BIT flag 307 can be found in the storage.

The master commit block 300 is recorded in a state in which the writing order with the rest of the transaction is guaranteed. Therefore, finding the master commit block 300 means that all other transaction details have been recorded.

17 again, in step S152, the operating system 12 sends write commands (WC#1, WC#) corresponding to the write operation calls (WO#1, WO#2) to the storage 20. 2) can be transmitted. The write commands WC#1 and WC#2 may include information about the first inode Inode#1 of the first file File#1.

In step S153, the operating system 12 may store the block location (block number) of the first inode (Inode#1) in a part of the master commit block 300.

Step S153 of FIG. 17 is a step of reflecting the contents of the i-th file among the files included in the transaction to the storage. At this time, the file contents include the inode of the ith file. That is, since step S153 of FIG. 17 is an operation immediately after the inode is stored in the non-volatile memory, the operating system 12 can know the block location of the inode.

Steps S152 and S153 may be repeatedly executed for all other files included in the first transaction.

For example, in step S154, the operating system 12 may transmit write commands WC#3 and WC#4 corresponding to the write operation calls WO#3 and WO#4 to the storage 20. there is. Information on the third inode (Inode#3) of the third file (File#1) may be included in the write commands (WC#3, WC#4).

In step S155, the operating system 12 may store the block location (block number) of the second inode (Inode#2) in a part of the master commit block 300.

When all files included in the first transaction are written to the volatile memory 22 of the storage 20, the operating system 12 transmits a flush command FC to the storage 20 in step S157. can

Then, in step S158 , the operating system 12 may transmit an MCB write command to write the master commit block 300 to the storage 20 to the storage 20 . The MCB write command may include the contents of the master commit block 300 .

The storage 20 may perform the following steps in response to commands received from the operating system 12 .

In step S161, when the storage 20 receives the write commands WC#1 and WC#2, the first page (Page#1) and the first inode ( Save Inode#1).

In step S162, when the storage 20 receives the write commands WC#3 and WC#4, the second page (Page#2) and the second inode ( Save Inode#2).

In step S163, when the storage 20 receives the flush command FC, the storage 20 transfers information about the first transaction stored in the volatile memory 22 to the non-volatile memory. Save in (23).

In step S164 , the storage 20 may store the master commit block 300 in the non-volatile memory 23 .

If a problem such as power outage occurs in the storage 20 after the step S152 starts and before the step S164 is completed, the file related to the first transaction recorded in the non-volatile memory 23 Their contents are invalid.

FIG. 19 illustrates the relationship between MCB and inode information stored in storage by the multi-file transaction commit method described in FIG. 17 .

19, reference number 231 denotes the first block 231 in which the first inode (Inode#1) is stored, and reference number 232 denotes the second block 232 in which the second inode (Inode#2) is stored. and reference number 233 indicates a third block 233 in which the master commit block 300 is stored.

The first inode pointer 2331 included in the third block 233 has a value related to the address of the first block 231, and the second inode pointer 2332 corresponds to the second block ( 232) has a value related to the address.

20A and 20B are flowcharts illustrating a method of interimly storing transaction data for large-capacity transactions according to an embodiment of the present invention.

20a and 20b may be collectively referred to as FIG. 20 .

In step S210, the operating system 12 may receive a transaction start call from the application 11.

In step S220, the operating system 12 performs a first write operation call (WO#1) for the first to third pages (pages 1 to 3) of the first file (fd#1) from the application 11. ) can be received.

In step S310, the operating system 12 may pin the data of the first page to the third page in a page cache of a memory (eg, DRAM) managed by the operating system 12. .

The step (S310) indicates that a space for storing pages designated by the first write operation call (WO#1) (the first to third pages in the example of FIG. 20) is already prepared in the page cache. It can be executed based on the condition.

In step S230, the operating system 12 performs a second write operation call (WO#2) for the fourth to eighth pages (pages 4 to 8) of the first file (fd#1) from the application 11. ) can be received.

When it is determined that the page cache does not have a place to store all of the pages designated by the second write operation call (WO#2) (the fourth to eighth pages in the example of FIG. 20), the operating system (12) may execute step S320.

Step S320 may include the following steps S321, S322, and S323.

In step S321, the operating system 12 selects all pages that can be stored in the remaining space of the page cache among the pages designated by the second write operation call (WO#2) (in the example of FIG. 4th to 6th pages) can be pinned.

In step S322, the operating system 12 selects standby pages (7 in the example of FIG. In order to secure space to store pages to eighth pages), evict pages (first to second pages in the example of FIG. 20), which are some of the pages already pinned in the page cache, are transferred to the storage 20. ivic can do it To this end, the operating system 12 may transmit an Evict command for the Evict pages already pinned in the page caching to the storage 20 .

In this case, the size of the standby pages and the size of the evict pages may be equal to each other.

In step S410, the storage 20 may store the Evict pages in the non-volatile memory of the storage 20 when receiving an Evict command for the Evict pages from the operating system 12. there is.

In step S323, the operating system 12 pins new file mapping information, which is mapping information between the evict pages and blocks in the storage 20 in which the evict pages are stored, to the page cache. can do.

In step S330, the operating system 12 may pin the standby pages in the space occupied by the Evict pages in the page cache.

In this case, the write priority of the Evict pages may be higher than the write priority of pages pinned to the page cache after the execution of the Evict command.

In step S240, the operating system 12 may receive a commit call from the application 11.

In step S340, the operating system 12 may process transaction #1, and may specifically include steps S341 to S345.

In step S341, the operating system 12 may transmit a write command to the storage 20. In this case, the write command may be a write command for pages of the first set pinned to the page cache.

In step S420 , the storage 20 may store the first set of pages in a volatile memory in the storage 20 . In the example of FIG. 20 , the pages of the first set are third to eighth pages.

In step S342, the operating system 12 may transmit a flush command to the storage 20.

In step S430, the storage 20 may store the information stored in the volatile memory in the storage 20 to the non-volatile memory in the storage 20 (flush).

In step S343, the OS 12 may transmit a command to invalidate old blocks occupied by the Evict pages in the storage 20 before the execution of the Evict command, to the storage 20. .

In step S440, the storage 20 may invalidate the old blocks.

In step S344, the operating system 12 may cancel the pinning of the new file mapping information.

In step S345, the operating system 12 may transmit a checkpoint command for the new file mapping information to the storage 20.

In step S450, the storage 20 may store the new file mapping information in a non-volatile memory within the storage 20.

In the transaction processing method shown in FIG. 20, steps S345 and S450 may be omitted.

21 illustrates an example of a method of executing garbage collection according to an embodiment of the present invention.

First, it can be assumed that a first transaction for updating data A of a certain file from old data A to new data A' is being executed. At this time, the step of pinning the new data A' to the page cache 133 managed by the operating system executing the first transaction may have already been executed. At this time, the old data A may be stored in the third block 313 of the first segment 310 of the storage 20 .

In this case, before the first transaction is terminated (committed), a first garbage collection setting the first segment as a victim segment may be started. When the first garbage collection is triggered and started, the first transaction may be stopped while the new data A′ is pinned to the page cache 133 .

In this case, the first garbage collection caches the old data A stored in the third block 313, which is a victim block of the first segment 310, in the shadow page cache 131 that is distinct from the page cache 133. can

Both the page cache 133 and the shadow page cache 131 may be included in memory (eg, DRAM) managed by an operating system.

Next, the first garbage collection may store (migrate) the old data A stored in the shadow page cache 131 to the fifth block 325 of the second segment 320 .

Then, when the first garbage collection is terminated, the interrupted first transaction may be resumed.

22 illustrates another example of a method of executing garbage collection according to an embodiment of the present invention.

In step S141, a transaction for changing data A stored in the storage 20 to A' is executed, and data A' may be pinned to the page cache 133. At this time, the node block 235 of the storage 20 may store file mapping information (A:3) indicating that data A is stored in the third block LBA 3 of the storage 20 . Also, the file mapping information (A:3) of the node block 235 may be cached in the node page 135 of the page cache managed by the host.

In a state where the transaction is not committed, garbage collection may be triggered in the log structure file system of the operating system of the host and garbage collection may be performed. As a result, before step S142 is executed, the transaction is suspended, and the operating system may first checkpoint the file system state (pre-GC checkpoint).

Then, in step S142, the garbage collection module of the operating system selects the segment 310 as a victim segment. At this time, it may be checked whether the data stored in the victim block 313 of the selected victim segment 310 has been updated by the transaction stopped by the garbage collection. If it is confirmed that the data stored in the victim block 313 of the selected victim segment 310 has been updated by the transaction stopped by the garbage collection, the data A stored in the victim block 313 is converted to a shadow page. After caching in the cache 131 , data A stored in the shadow page cache 131 may be migrated to the target block 325 of the target segment 320 . Further, file mapping information of the node page 135 for data A may be modified by reflecting the result of garbage collection. That is, old file mapping information (A:3) recorded in the node page 135 may be modified to new file mapping information (A:5). That is, the associated file mapping 135 is updated to reference the new disk location 325 (LBA 5) of the victim block 313 (LBA 3).

In step S143, the operating system may checkpoint the updated state of the file system (post-GC checkpoint), and as a result, the new file mapping information (A: 5) stored in the node page 135 It may be reflected in the node block 235. And the garbage collection may be terminated.

When the garbage collection is terminated, the suspended transaction may be resumed and the transaction may be completed. However, a system crash may occur before the resumed transaction is completed, that is, before the transaction is committed. When the system crash occurs, the operating system restores the state of the file system to a state prior to the execution of the transaction.

Step S144 is an example to explain such a recovery process. The operating system may cache the new file mapping information (A: 5) stored in the node block 235 of the storage 20 in the page block 135 in step S43. Data A stored in the target block 325 (LBA 5) of the storage 20 may be cached in the page cache 133 based on the new file mapping information (A:5). Accordingly, it can be understood that, when a system crash occurs, the state of the file system can be restored to the state prior to the start of execution of the transaction.

23A and 23B are flowcharts illustrating a method of executing garbage collection according to an embodiment of the present invention.

Hereinafter, FIGS. 23A and 23B may be collectively referred to as FIG. 23 .

In step S510, the operating system 12 may receive a transaction start call regarding the first transaction from the application 11.

In step S520, the operating system 12 may receive a first write operation call (WO#1) for first data from the application 11.

In step S710, the operating system 12 may receive file mapping information for reading the first data from the storage 20. In this case, the file mapping information may be referred to as old file mapping information.

In step S610, the operating system 12 may pin the first data A' to the page cache. The first data A' may be referred to as new data.

In step S620, the operating system 12 may start garbage collection and stop the first transaction.

In step S621, the operating system 12 determines, if the data of the victim block selected by the garbage collection is old corresponding to the first data A' pinned to the page cache by the suspended first transaction. If it is data (A), the data of the victim block can be migrated to the target block using the shadow page cache. In this case, the shadow page cache may be another cache distinct from the page cache.

In this case, the file system of the operating system may manage the page cache using a first pointer variable, and manage the shadow page cache using a second pointer variable different from the first pointer variable. The shadow page is a cache used when migrating blocks related to transactions for which garbage collection has not been completed, and the page cache is used for various file operations, transactions, or general blocks for which garbage collection is not completed This is the cache used when migrating blocks that are not related to unrelated transactions).

The target block is a block selected by the garbage collection.

Step S621 may include step S720 and step S730.

In step S720, the operating system 12 may cache the old data A stored in the victim block in the shadow page cache.

In step S730, the operating system 12 may migrate the old data A stored in the shadow page cache to a target block of the storage 20.

In step S622, the first file mapping information (ex: A:1) of the victim block stored in the storage 20 may be cached in the shadow page cache.

In step S623, the first file mapping information (ex: A:1) stored in the shadow page cache may be updated with file mapping information (ex: A:8) having the location of the target block.

In step S624, the updated file mapping information (ex: A:8) stored in the shadow page cache may be flushed to the storage 20.

In step S625, the updated and cached file mapping information (ex: A:8) may be pinned to the page cache, garbage collection may be terminated, and the first transaction may be resumed.

In step S530, the operating system 12 may receive a commit operation call from the application 11.

In step S690, the operating system 12 may process the first transaction. The step (S690) may include a step (S750) of storing the first data (A'), which is new data, in the storage 20.

24A and 24B are flowcharts illustrating a method of executing garbage collection according to another embodiment of the present invention.

Hereinafter, FIGS. 24A and 24B may be collectively referred to as FIG. 24 .

In step S510, the operating system 12 may receive a transaction start call regarding the first transaction from the application 11.

In step S520, the operating system 12 may receive a first write operation call (WO#1) for first data from the application 11.

In step S610, the operating system 12 may pin the first data A' to the page cache. The first data A' may be referred to as new data.

In step S521, the operating system 12 may receive a second write operation call (WO#2) for second data from the application 11.

In step S611, the operating system 12 may determine whether the second data B', which is new data, can be pinned to the page cache. That is, it may be determined whether there is a page for pinning the second data (B′) in the page cache. If there is no page left in the page cache to pin the second data B', the first data A', which is new data, may be ejected to the storage 20. That is, the first data A' is written to the storage 20, and the space where the first data A' was written can be secured in the page cache. That is, for example, if the first data (A') has been pinned to the first page of the page cache, the first page can be retrieved as a space for pinning other data. Accordingly, the operating system 12 may pin the second data B' to the secured space of the page cache.

In step S640, the operating system 12 may start garbage collection and stop the first transaction.

In step S641, the operating system 12 determines whether the data of the victim block selected by the garbage collection is old data of the data (ex: first data A') that has been evicted by the interrupted first transaction. can determine whether If the data of the victim block is the old data (ex: A) of the data (ex: first data (A')) that has been evicted by the interrupted first transaction, the data of the victim block is stored in the shadow page cache. can be used to migrate to the target block.

Step S641 may include step S720 and step S730.

In step S720, the operating system 12 may cache the old data A stored in the victim block in the shadow page cache.

In step S730, the operating system 12 may migrate the old data A stored in the shadow page cache to a target block of the storage 20.

In step S642, the operating system 12 may cache first file mapping information (eg, A:1) of the victim block stored in the storage 20 in a shadow page cache.

In step S643, the operating system 12 converts the first file mapping information (ex: A:1) stored in the shadow page cache to file mapping information (ex: A:8) having the location of the target block. can be renewed

In step S644, the operating system 12 may flush the updated file mapping information (ex: A:8) stored in the shadow page cache to the storage 20.

In step S645, the operating system 12 may terminate garbage collection and resume the first transaction.

In step S530, the operating system 12 may receive a commit operation call from the application 11.

In step S690, the operating system 12 may process the first transaction.

The step (S690) may include a step of storing the second data (B'), which is new data, in step (20).

25A and 25B are flowcharts illustrating a method of executing garbage collection according to another embodiment of the present invention.

Hereinafter, FIGS. 25A and 25B may be collectively referred to as FIG. 25 .

In step S510, the operating system 12 may receive a transaction start call regarding the first transaction from the application 11.

In step S520, the operating system 12 may receive a first write operation call (WO#1) for first data A', which is new data, from the application 11. The first data (A′) may be used to replace the old data (A) already recorded in the storage 20 .

In step S610, the operating system 12 may pin the first data A' to the page cache. The first data A' may be referred to as new data.

In step S521, the operating system 12 may receive a second write operation call (WO#2) for second data B', which is new data, from the application 11.

In step S611, the operating system 12 may determine whether the second data B', which is new data, can be pinned to the page cache. That is, it may be determined whether a page for pinning the second data (B′) remains in the page cache. If there is no page left in the page cache to pin the second data (B'), the first data (A') pinned in the page cache may be ejected to the storage 20 . That is, the first data A' is written to the storage 20, and the space where the first data A' was written can be secured in the page cache. That is, for example, if the first data (A') has been pinned to the first page of the page cache, the first page can be retrieved as a space for pinning other data. Accordingly, the operating system 12 may pin the second data B' to the secured space of the page cache.

Step S611 may include a step S711 in which the operating system 12 evicts the first data A′ to the storage 20 .

In step S660, the operating system 12 may start garbage collection and stop the first transaction.

In step S661, the operating system 12 determines whether the data of the victim block selected by the garbage collection is data (ex: first data A') obtained by the interrupted first transaction. can do. If the data of the victim block is the data (ex: first data (A')) that has been evacuated by the interrupted first transaction, the target designated by the garbage collection using the page cache You can migrate to blocks.

Step S661 may include step S720 and step S730.

In step S720, the operating system 12 may cache the evicted data (ex: first data A') stored in the victim block in the page cache.

In step S730, the operating system 12 may migrate the evicted data (ex: first data (A')) stored in the shadow page cache to a target block of the storage 20. .

In step S662, the operating system 12 pins new file mapping information (ex: A': 8) having the location of the target block in the page cache, ends garbage collection, and executes the first transaction. can be resumed

In step S530, the operating system 12 may receive a commit operation call from the application 11.

In step S690, the operating system 12 may process the first transaction.

The step S690 may include storing the second data B', which is new data, in the storage 20 .

26 is a flowchart illustrating a method of executing garbage collection according to another embodiment of the present invention.

In step S510, the operating system 12 may receive a transaction start call regarding the first transaction from the application 11.

In step S520, the operating system 12 may receive a first write operation call (WO#1) for first data from the application 11.

In step S670, the operating system 12 may start garbage collection and stop the first transaction.

In step S671, the operating system 12 stores the data of the victim block selected by the garbage collection as old data (ex: A) or old data (ex: A) of the data (ex: A') stored in the storage 20 by the interrupted first transaction. If the data of the victim block is pinned to the page cache by the aborted first transaction or is old data (ex: A) of data (ex: A') stored in the storage 20, the operating system 12 Data (ex: A) of the victim block may be migrated to a target block selected by the garbage collection using the shadow page cache.

Step S671 may include step S720 and step S730.

In step S720, the operating system 12 may cache the old data A stored in the victim block in the shadow page cache.

In step S730, the operating system 12 may migrate the old data A stored in the shadow page cache to a target block of the storage 20.

In step S672, the operating system 12 may cache the first file mapping information (eg, A:1) of the victim block stored in the storage 20 in the shadow page cache.

In step S673, the operating system 12 converts the first file mapping information (ex: A:1) stored in the shadow page cache to file mapping information (ex: A:8) having the location of the target block. can be renewed

In step S674, the operating system 12 may flush the updated file mapping information (ex: A:8) stored in the shadow page cache to the storage 20.

In step S675, the operating system 12 may terminate garbage collection and resume the first transaction.

In step S530, the operating system 12 may receive a commit operation call from the application 11.

In step S690, the operating system 12 may process the first transaction.

The embodiment shown in FIG. 23 and the embodiment shown in FIG. 24 may be a special embodiment of the embodiment shown in FIG. 26 .

27 illustrates a configuration of a host provided according to an embodiment of the present invention.

Host 10 may be a computing device. The host 10 may include a memory 13 , a power supply unit 16 , a storage unit 17 , a processing unit 18 , and a communication unit 19 . The power supply unit 16 supplies operating power to the host 10 . The storage unit 17 may store a program including instructions for causing the processing unit 18 to execute the application 11 and/or the operating system 12 . The storage unit 17 may be a non-volatile memory such as SSD or HDD. The processing unit 18 may be configured to execute the application 11 and/or the operating system 12 by executing the program. Processing unit 18 may be, for example, an AP or a CPU. The memory 13 may be directly controlled by the operating system 12, such as DRAM. The memory 13 may include the aforementioned page cache and shadow page cache. The communication unit 19 may be configured to drive a communication medium that delivers commands of the operating system 12 to the storage 20 . The communication medium may transmit a baseband communication signal or a modulated communication signal.

In this specification, pinning may refer to an operation of preventing content written in the page cache from being written to storage.

Using the above-described embodiments of the present invention, those belonging to the technical field of the present invention will be able to easily implement various changes and modifications without departing from the essential characteristics of the present invention. The contents of each claim of the claims may be combined with other claims without reference relationship within the scope understandable through this specification.

Claims (15)

starting garbage collection and stopping the first transaction before the first transaction is committed, by an operating system of the host;
a garbage collection execution step of, by the operating system, migrating data of a victim block selected by the garbage collection to a target block; and
resuming, by the operating system, the first transaction when the garbage collection is terminated;
Including,
If the data of the victim block is the old data of the first data stored in the storage after being pinned to the page cache by the interrupted first transaction, the migration is executed using the shadow page cache;
The shadow page cache is another cache distinct from the page cache.
Garbage collection method. According to claim 1,
The garbage collection execution step,
migrating, by the operating system, data of the victim block to the target block using the shadow page cache;
including,
Garbage collection method. According to claim 1,
The garbage collection execution step,
After the migration step,
caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache;
updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and
flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage;
including,
Garbage collection method. According to claim 3,
If the data of the victim block is the old data of the first data pinned to the page cache by the aborted first transaction,
The garbage collection execution step further comprises pinning, by the operating system, the updated file mapping information to a page cache.
Garbage collection method. According to claim 1,
If the data of the victim block selected by the garbage collection is old data of the first data stored in the storage by the interrupted first transaction,
Before the first transaction is suspended,
pinning, by the operating system, the first data designated by an application to a first page of the page cache;
The operating system determines whether a space for pinning the second data designated by the application exists in the page cache, and if it is determined that there is no space for pinning the second data in the page cache, the first data is stored in the storage. and pinning the second data to the first page;
Including more,
Garbage collection method. The garbage collection method according to claim 1, wherein the file system of the operating system is a log structured file system. According to claim 1,
The garbage collection is executed for a separate storage distinct from the host,
The victim block and the target block are blocks existing in the storage selected by the garbage collection,
Garbage collection method. Memory; And a processing unit that runs an operating system; as a host including,
the operating system,
starting garbage collection and stopping the first transaction before the first transaction is committed;
a garbage collection execution step of migrating data of a victim block selected by the garbage collection to a target block; and
resuming the first transaction when the garbage collection is terminated;
is supposed to run
If the data of the victim block is old data (ex: A) of the first data (ex: A') pinned to the page cache by the aborted first transaction and stored in storage, the migration uses the shadow page cache. is executed,
The shadow page cache is another cache distinct from the page cache.
host. According to claim 8,
The garbage collection execution step,
migrating, by the operating system, data of the victim block to the target block using the shadow page cache;
including,
host. According to claim 8,
The garbage collection execution step,
caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache;
updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and
flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage;
including,
host. According to claim 10,
If the data of the victim block is the old data of the first data pinned to the page cache by the aborted first transaction,
The execution of the garbage collection further comprises pinning, by the operating system, the updated file information to a page cache.
host. According to claim 8,
If the data of the victim block selected by the garbage collection is old data of the first data stored in the storage by the interrupted first transaction,
Before the first transaction is suspended,
the operating system,
pinning the first data designated by an application to a first page of the page cache; and
determining whether a space for pinning the second data designated by the application exists in the page cache, and if it is determined that there is no space for pinning the second data in the page cache, storing the first data in the storage; pinning the second data to the first page;
which is supposed to run more
host. The host of claim 7 , wherein the shadow page cache and the page cache are included in the memory and are volatile memories managed by the operating system. the host's operating system,
starting garbage collection and stopping the first transaction before the first transaction is committed;
a garbage collection execution step of migrating data of a victim block selected by the garbage collection to a target block; and
resuming the first transaction when the garbage collection is terminated;
A software program containing instructions to execute is recorded,
A non-volatile storage medium that can be read by a computer. According to claim 14,
The garbage collection execution step,
caching, by the operating system, file mapping information of the victim block stored in the storage in the shadow page cache;
updating, by the operating system, the file mapping information cached in the shadow page cache to file mapping information having a location of the target block; and
flushing, by the operating system, the updated file mapping information stored in the shadow page cache to the storage;
including,
A non-volatile storage medium that can be read by a computer.

Images