KR20170014276A - Variable block processing file system and file variable block processing method based on fixed block - Google Patents
Variable block processing file system and file variable block processing method based on fixed block Download PDFInfo
- Publication number
- KR20170014276A KR20170014276A KR1020150107330A KR20150107330A KR20170014276A KR 20170014276 A KR20170014276 A KR 20170014276A KR 1020150107330 A KR1020150107330 A KR 1020150107330A KR 20150107330 A KR20150107330 A KR 20150107330A KR 20170014276 A KR20170014276 A KR 20170014276A
- Authority
- KR
- South Korea
- Prior art keywords
- block
- file
- data
- change
- variable
- Prior art date
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0643—Management of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The present invention relates to a variable block processing file system and a file variable block processing method based on a fixed block, and a variable block processing file system based on a fixed block according to the present invention includes a storage medium for storing file data, A file manager for storing the file data in a plurality of fixed blocks if the file is first stored in the storage medium by the application program; A file change detecting unit for storing changed block data in the memory when the file change data is detected by the application program; and, when the file change data is requested to be stored by the application program, Only the index information is changed for the missing block, And a file change processing unit for processing the block having the data change by processing it as a variable block, thereby minimizing the writing process to the storage medium, thereby increasing the usable period of the storage medium.
Description
The present invention relates to a variable block processing file system and a file variable block processing method based on a fixed block, and more particularly, to a variable block processing method and a variable block processing method, which are based on a fixed block for minimizing a writing process to a storage medium, A file system, and a file variable block processing method.
The file system is responsible for managing various types of files. That is, the file system is responsible for storing the operating system, various executable files, and user data in the computer system. In the user area, the file is regarded as a byte stream, and the file data is read or written through the file handle.
The read / write request for the file in the user area is transferred to the file system of the kernel area, and finally converted to a read / write request for the block of data stored in the block device. Therefore, the byte stream processed in the user area is converted into a block unit at the device level, which causes various problems in the operation on the file.
For example, a case in which a portion of a file is read into a memory buffer of a user area, and then a byte of data is added or deleted in the middle of the buffer will be described. In this case, it is possible to easily change the data while moving the file offset to the area to be changed in the user area and inserting or deleting the new byte. However, if such a change is stored in the file system, . The byte stream present on the memory buffer must ultimately be stored in a block on the disk, and the data change of one byte will require the modification of all subsequent data blocks in succession.
This problem occurs in the same way when insertion or deletion of data occurs in units of blocks as well as insertion or deletion in units of bytes. For example, an application such as Winzip or WinRAR, which is widely used in desktop computers, is a structure in which a plurality of sub-files constitute a single file, and each sub-file has a block-aligned structure I have. Therefore, the process of deleting or inserting a subfile corresponds to a process of inserting or deleting consecutive blocks. However, most file systems handle file modification operations like ordinary files, without regard to operations related to insertion or deletion of such block-level data.
In order to solve the above-described problems, the present inventors have filed a patent application No. 2014-0107590 as a variable block processing file system and method based on a fixed block that enables storage of variable-size file data on a fixed-size block. The application of the variable block in the block unit file system solves the overload problem due to the write operation in the existing file system by changing the index to the data block instead of performing the write operation in the whole block, The elimination of operations has also resulted in a significant improvement in overall system performance. However, a write operation to a permanent storage medium such as a hard disk or a flash memory may not be stored last after completion of a document operation, for example, when a document operation is performed, but may involve a lot of write operations during a document operation.
In order to solve the above problems, the present invention provides a variable block processing file system and a file variable block processing method based on a fixed block for minimizing the writing process to a storage medium in a fixed block unit file system capable of variable block processing And to provide the above objects.
According to an aspect of the present invention, there is provided a variable block processing file system based on a fixed block, comprising: a storage medium for storing file data; a memory for storing temporarily changed block data; A file management unit for storing file data in a plurality of fixed blocks if the file is first stored in the storage medium by the application program; and a change management unit for storing the change block data in the memory when the change of the file data by the application program is detected A file change detecting unit for determining whether or not the file change data is requested to be stored by the application program, and for storing the file change data in the memory by using the change block data stored in the memory, To the block with the data change. And a file change processor for processing and storing the variable block.
Wherein the file management unit includes a file management area and a data storage area, and the file management unit stores the fixed block information in the file management area when the file is first stored in the storage medium, Wherein the block change processing unit changes only the index information of the file management area for a block that is not changed among a plurality of fixed blocks stored in the storage medium, The variable block information may be further stored in the management area and the corresponding variable block data may be stored in the data storage area.
The fixed block information includes position information of the data storage area for each block, and the variable block information may include the number of change blocks and the length of file data stored in the change block.
Wherein the file change detection unit classifies one file data to be stored in the storage medium into a predetermined unit size smaller than the block size and extracts a plurality of hash values when the file storage in the storage medium is first performed by the application program, It is possible to detect whether there is a modification of the file data by the application program through the similarity comparison using a plurality of hash values.
Wherein the file change detector stores change block and variable length block information in the memory in response to the change block data, and the file change processor, when the file change data is requested to be stored by the application program, Comparing the changed block stored in the memory and storing at least one block of the file change data as a variable block using information stored in the variable length block if the block matches the change block stored in the memory.
According to another embodiment of the present invention, a variable block processing based on a fixed block including a storage medium for storing file data and a memory for storing temporarily changed block data, The method includes: a file management step of storing file data in a plurality of fixed blocks if the file storage in the storage medium by the application program is the first; and, when a change of file data by the application program is detected, And a file change detection step of, when the file change data is requested to be stored by the application program, using the change block data stored in the memory, Only the index information is changed, and the data And a file change processing step of processing the block having the change by processing it as a variable block and storing it.
According to the above-described configuration, the present invention minimizes the writing process to the storage medium in the file system of the fixed block unit capable of variable block processing, thereby increasing the usable period of the storage medium.
In addition, the present invention makes it possible to store file data of a variable size in a block of a fixed size, so that file storage according to a change of file data can be performed quickly.
In addition, the present invention can extend the lifetime of a flash memory or the like, which is a semiconductor type nonvolatile memory, by minimizing the recording of data in the data area of the storage medium.
1 is a block diagram illustrating an entire block diagram of a fixed block processing file system based on a fixed block according to an embodiment of the present invention.
Figs. 2A to 2C are diagrams for explaining variable processing based on a fixed block according to the present invention. Fig.
3 is a diagram showing an embodiment of metadata according to the present invention.
4A and 4B are diagrams for explaining file change data processing in a variable block processing file system based on a fixed block according to the present invention.
FIG. 5 is a diagram showing the file change detection unit shown in FIG. 1 in more detail.
6 is a diagram illustrating the operation of the overall system according to another embodiment of the present invention.
7 is a flowchart illustrating a file variable block processing method based on a fixed block according to another embodiment of the present invention.
Hereinafter, preferred embodiments of a variable block processing file system and a file variable block processing method based on a fixed block according to the present invention will be described with reference to the accompanying drawings. In the following description of the present invention, it is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the technical scope of the present invention. Will be.
1 is a block diagram illustrating an entire block diagram of a fixed block processing file system based on a fixed block according to an embodiment of the present invention.
1, a variable block processing file system based on a fixed block includes a
The
In the
The
The
The file
The hash
The
The
The file
The change
If the block
The
Hereinafter, embodiments of the present invention will be described in more detail.
Figs. 2A to 2C are diagrams for explaining variable processing based on a fixed block according to the present invention. Fig.
FIG. 2A is a conceptual diagram of a conventional file system. When the file is stored in the
2B is a conceptual diagram of a conventional file system for modifying stored file data. As shown in FIG. 2B, when the "K" is deleted, the file is changed from the third block to the internal file data in the chunking step. In this case, the existing block may be invalidated according to the file system, the new block may be allocated and the file data may be stored, or the existing block may be overwritten.
FIG. 2C is a conceptual diagram for modifying file data using a variable based on a fixed block according to the present invention. As shown in FIG. 2C, when the "K" is deleted, the file data size smaller than 512 bytes is stored in the third block in the chunking step. And INODE modifies the block index on the changed contents. In the case of storing file data according to the present invention, only the third block in the data block is changed, and no change occurs in the remaining blocks.
3 is a diagram showing an embodiment of metadata according to the present invention.
INODE stores basic information for file I / O. In the present invention, an existing INODE is extended to be based on a fixed block to support variable processing of the block. As shown in FIG. 3, in the metadata information according to the present invention, two pieces of information are added in addition to the information (B0, B1, B2, and B3) about the existing fixed block as well as the variable processing of the fixed block. A variable block (VB) records how many block variables it has. If VB is 1, it indicates that one block has been changed by the variable processing of the block in the file data. A variable block index (VBI) indicates an index of a block in which variable processing for a block has occurred. In FIG. 3, V1, V2, V3 and V4 are shown.
As shown in FIG. 3, it is assumed that 16 bytes have been deleted due to the change of the file data in the third block. In this case, only 496 bytes are stored in the block located at 53, and the remaining 16 bytes are meaningless data. In order to refer to this in the file system, the corresponding information must be described in the INODE. 1 is stored in VB because one variable process is generated for the fixed block. In V1, it is informed that
In FIG. 3, variable processing for the fixed block is limited to a maximum of four, but the maximum number can be increased as needed. However, when variable processing for a maximum number of fixed blocks is required, garbage collection is forcibly performed to remove all the variable processing for the fixed block.
4A and 4B are diagrams for explaining file change data processing in a variable block processing file system based on a fixed block according to the present invention.
4A shows a file change data processing method in a conventional file system. As shown in FIG. 4A, when a new file is created, the conventional file system can not utilize the block even though there are four blocks without data change. Because there is a block without data change, it is difficult to share block by block due to byte-shift phenomenon.
4B shows a method of processing file change data using variable processing based on a fixed block according to the present invention. As shown in FIG. 4B, when the variable block processing file system based on the fixed block implemented by the present invention is used, chunking is performed based on the unchanged block to find no change for the four blocks And four blocks with no change can be stored in the nonvolatile memory instead of being newly stored in the nonvolatile memory by modifying the index into blocks of the existing file.
FIG. 5 is a diagram showing the file change detection unit shown in FIG. 1 in more detail.
5, the file
When the
The
The similarity digest hashing (SDHASH) scheme or the SSDEEP scheme can be used to process the file similarity.
Instead of choosing arbitrary chunk sizes using a rabin fingerprint, SDHASH uses stochastic features using the newness entropy, so that the feature point selection does not shift to a specific part of the data, Reliability is improved. The selected feature points measure the statistical based file similarity using Bloom filter. The 64-byte minutiae are used to generate the minutiae by selecting the minutiae that contain the popularity score above a given threshold value. In addition, among the selected minutiae, minutiae which have a value of more than 100 and less than 990 are excluded, thereby lowering the positive error rate of the Bloom filter.
In SDHASH, the Bloom filter is generated with a fixed size of 256 bytes. 128 feature point values are assigned to each generated bloom filter. If the number of minutiae exceeds 128, a new Bloom filter is added and minutiae values are assigned. Each feature point is computed as a 160-bit hash value using the SHA-1 hash function. The SDHASH comparison process compares all generated Bloom filters one by one and measures the file similarity by averaging all Bloom filter comparison values.
SSDEEP performs file similarity measurement using two hash methods. The rolling hash function moves the contents of the file one byte at a time and generates a hash value. Calculate the trigger position, which is the boundary point inside the file, using the generated rolling hash value. If a trigger location is found, it is treated as a single chunk from the beginning of the file to the trigger location, and the junk generates a junk hash using the MD5 hash function. The generated chunk hash value is converted into a bit string using Base64 encoding, extracts the last 6 bits of the converted bit string, and concatenates the bit values of each bit string to generate a file digest.
The file
6 is a diagram illustrating an operation of a fixed block processing file system based on a fixed block according to another embodiment of the present invention.
The user application program may be a user application program, such as a general text editor, for example a Hangul word processor (HWP) program.
The VLFS library is an example of a virtual file system that provides basic functions for opening, closing, and locating files in an existing file system, such as open (), close (), and lseek () Block change check function vlfs_check (), a read function vlfs_read () for a variable length block, and a write function vlfs_write () for a variable length block.
The above open (), close () and lseek () functions are related to the
The block change check function vlfs_check () is used to check whether a modification operation occurs by using a user application program. It checks whether there is a change block before calling the vlfs_write () function. If the change block is empty, It terminates without calling vlfs_write (), but if the change block is not empty, it determines that a modification has occurred and calls vlfs_write () to save it to disk. In addition, the block change check function vlfs_check () is used for comparing and judging whether or not the corresponding block data is modified data when it is stored in the disk.
In the
When a user application program needs to make modifications such as adding new data to a file using the VLFS library, it is necessary to know the location of the data added through the application and the size of the added data. The additional data information for this is basically stored in the change block. In addition, a "change check function" is used to check whether or not data is added.
For example, the file " A " stored in the disk which is the
Hereinafter, the process of detecting the file change of the modified data will be described in more detail.
The file
If the size of the modified data is larger than the size of the given data block, the added data is divided into several parts using the size of the given block. If the size of the given block is 4KB and the size of the added data is 10KB, the data is divided into 3 blocks having sizes of 4KB, 4KB and 2KB, and all three blocks are stored in the change block. In this case, since only 2 KB smaller than the size of a given block is used in the last third block, the remaining area is treated as a variable area and the corresponding information is additionally stored in the variable-length block.
Hereinafter, the process of storing the added new data in the storage medium will be described in more detail.
When the vlfs_wirte function is called (S902), the user application program passes the entire block contents of data from the beginning to the vlfs_check function.
The change
If the same block is not found, the change
When a change block is detected by the change
7 is a flowchart illustrating a file variable block processing method based on a fixed block according to another embodiment of the present invention.
The
If the file is first stored in the
The file
When the modification of the file data is detected, the file
Thereafter, when the user application program requests storage of changed file data, the change
If there is no change block, the
The scope of protection of the present invention should be interpreted according to the claims. It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit and scope of the invention as defined by the appended claims. It should be interpreted that it is included in the scope of right.
110: Storage medium 112: File management area
114: Data storage area 120: File management part
122: file generation module 130: file change detection unit
132: hash value extraction module 134: similarity comparison module
140: memory 142: change block storage area
150: File change processing unit 152: Change block detection module
164: block variable processing module 156: index modification module
510: HDD 520: file creation memory
530: Hash value storage memory
Claims (8)
A memory for storing temporarily changed block data;
A file management unit for storing file data in a plurality of fixed blocks if the file storage in the storage medium by the application program is the first,
A file change detecting unit for storing change block data in the memory when a change of file data by the application program is detected;
When the storage of the file change data is requested by the application program, only the index information is changed for the block among the plurality of fixed blocks stored in the storage medium using the change block data stored in the memory, And a file change processing unit for processing the block having the variable block as a variable block and storing the processed variable block as a variable block.
Wherein the storage medium includes a file management area and a data storage area,
Wherein the file management unit stores the fixed block information in the file management area and stores the file data in the data storage area allocated to the fixed block information when the file storage in the storage medium is the first,
Wherein the block change processing unit changes only the index information of the file management area for a block that has not been changed among the plurality of fixed blocks stored in the storage medium and outputs the variable block information to the file management area And stores the variable block data in the data storage area. The variable block processing file system according to claim 1,
Wherein the fixed block information includes position information of the data storage area for each block,
Wherein the variable block information includes a number of change blocks and a length of file data stored in the change block.
Wherein the file change detection unit classifies one file data to be stored in the storage medium into a predetermined unit size smaller than the block size and extracts a plurality of hash values when the file storage in the storage medium is first performed by the application program, And detects whether there is a modification of file data by the application program through a similarity comparison using a plurality of hash values.
Wherein the file change detection unit stores change block and variable length block information for the change block data in the memory,
Wherein the file change processing unit compares the entire block of the file change data with the change block stored in the memory when the application program requests the storage of the file change data and at least one block of the file change data is stored in the memory Block, and stores the information in the variable block using the information stored in the variable-length block.
A file management step of storing file data in a plurality of fixed blocks if the file storage in the storage medium by the application program is the first,
A file change detection step of storing change block data in the memory when a change of file data by the application program is detected;
When the storage of the file change data is requested by the application program, only the index information is changed for the block among the plurality of fixed blocks stored in the storage medium using the change block data stored in the memory, And a file change processing step of storing the processed block in a variable block and storing the processed variable block in a variable block.
Wherein the storage medium includes a file management area and a data storage area,
Wherein the file management step stores the fixed block information in the file management area and stores the file data in the data storage area allocated to the fixed block information,
Wherein the block change processing step changes only the index information of the file management area for a block having no change among the plurality of fixed blocks stored in the storage medium, And stores the variable block data in the data storage area. The method of claim 1,
The file change detection step may include a step of dividing one file data to be stored in the storage medium into a predetermined unit size smaller than a block size and extracting a plurality of hash values, And detecting whether there is a modification of the file data by the application program through comparison of similarity using the plurality of hash values.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020150107330A KR101729624B1 (en) | 2015-07-29 | 2015-07-29 | Variable block processing file system and file variable block processing method based on fixed block |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020150107330A KR101729624B1 (en) | 2015-07-29 | 2015-07-29 | Variable block processing file system and file variable block processing method based on fixed block |
Publications (2)
Publication Number | Publication Date |
---|---|
KR20170014276A true KR20170014276A (en) | 2017-02-08 |
KR101729624B1 KR101729624B1 (en) | 2017-05-02 |
Family
ID=58155657
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
KR1020150107330A KR101729624B1 (en) | 2015-07-29 | 2015-07-29 | Variable block processing file system and file variable block processing method based on fixed block |
Country Status (1)
Country | Link |
---|---|
KR (1) | KR101729624B1 (en) |
Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20140107590A (en) | 2011-12-22 | 2014-09-04 | 스카니아 씨브이 악티에볼라그 | Method and module for determining of at least one reference value |
-
2015
- 2015-07-29 KR KR1020150107330A patent/KR101729624B1/en active IP Right Grant
Patent Citations (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR20140107590A (en) | 2011-12-22 | 2014-09-04 | 스카니아 씨브이 악티에볼라그 | Method and module for determining of at least one reference value |
Also Published As
Publication number | Publication date |
---|---|
KR101729624B1 (en) | 2017-05-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8959089B2 (en) | Data processing apparatus and method of processing data | |
CN107787489B (en) | File storage system including a hierarchy | |
US20190324954A1 (en) | Two-stage front end for extent map database | |
US9141633B1 (en) | Special markers to optimize access control list (ACL) data for deduplication | |
US10860232B2 (en) | Dynamic adjustment of fingerprints added to a fingerprint index | |
US8914338B1 (en) | Out-of-core similarity matching | |
US9158465B2 (en) | Performing authorization control in a cloud storage system | |
US9424185B1 (en) | Method and system for garbage collection of data storage systems | |
EP2363815B1 (en) | System for permanent file deletion | |
US9639275B2 (en) | Managing data within a storage device based on file system metadata | |
EP3864519A1 (en) | Key-value store tree data block spill with compaction | |
US9183218B1 (en) | Method and system to improve deduplication of structured datasets using hybrid chunking and block header removal | |
US11609849B2 (en) | Deduplication system threshold based on a type of storage device | |
CN106980680B (en) | Data storage method and storage device | |
US11169968B2 (en) | Region-integrated data deduplication implementing a multi-lifetime duplicate finder | |
KR101729624B1 (en) | Variable block processing file system and file variable block processing method based on fixed block | |
US11507273B2 (en) | Data reduction in block-based storage systems using content-based block alignment | |
US20190108238A1 (en) | Block storage device with optional deduplication | |
KR101553028B1 (en) | Variable processing file system and file variable processing method based on fixed block | |
CN107506156B (en) | Io optimization method of block device | |
US20240037034A1 (en) | Data intake buffers for deduplication storage system | |
US11119681B2 (en) | Opportunistic compression | |
WO2023241771A1 (en) | Deduplication mechanism on sequential storage media | |
CN118159936A (en) | Parallel deduplication mechanism on sequential storage media | |
WO2024032898A1 (en) | Choosing a set of sequential storage media in deduplication storage systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
A201 | Request for examination | ||
E902 | Notification of reason for refusal | ||
E701 | Decision to grant or registration of patent right | ||
GRNT | Written decision to grant |