CN105787093B - A kind of construction method of the log file system based on LSM-Tree structure - Google Patents

A kind of construction method of the log file system based on LSM-Tree structure Download PDF

Info

Publication number
CN105787093B
CN105787093B CN201610152908.3A CN201610152908A CN105787093B CN 105787093 B CN105787093 B CN 105787093B CN 201610152908 A CN201610152908 A CN 201610152908A CN 105787093 B CN105787093 B CN 105787093B
Authority
CN
China
Prior art keywords
function
file system
file
lsm
tree structure
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201610152908.3A
Other languages
Chinese (zh)
Other versions
CN105787093A (en
Inventor
陈康
武永卫
郑纬民
王振钊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Tsinghua University
Original Assignee
Tsinghua University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tsinghua University filed Critical Tsinghua University
Priority to CN201610152908.3A priority Critical patent/CN105787093B/en
Publication of CN105787093A publication Critical patent/CN105787093A/en
Application granted granted Critical
Publication of CN105787093B publication Critical patent/CN105787093B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/13—File access structures, e.g. distributed indices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/14—Details of searching files based on file metadata
    • G06F16/148—File search processing
    • G06F16/152—File search processing using file content signatures, e.g. hash values
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/10—File systems; File servers
    • G06F16/18—File system types
    • G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
    • G06F16/1815—Journaling file systems

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Databases & Information Systems (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Library & Information Science (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention proposes a kind of construction methods of log file system based on LSM-Tree structure, include the following steps to construct the log file system fuse framework interface based on LSM-Tree structure, includes the following steps: the multiple directory operation functions for constructing the log file system based on LSM-Tree structure and file manipulation function；Record data are added and inquired into the log file system based on LSM-Tree structure of building using Hash mapping function.The present invention can effectively improve the readwrite performance of catalogue, small documents under the premise of keeping big file read-write performance constant.

Description

A kind of construction method of the log file system based on LSM-Tree structure
Technical field
The present invention relates to file system technology field, in particular to a kind of log file system based on LSM-Tree structure Construction method.
Background technique
File system is the mechanism of operating system storage and management data on the disk of computer.1964 by Bell's reality The Multics time sharing operating system for testing room, the Massachusetts Institute of Technology and the joint development development of North America General Electric Co. Limited, is put forward for the first time Directory tree structure thought, indicates the origin of modern file system.
This tree-like thought is used in the File system design of oneself by UNIX operating system, form comprising bootstrap block, The file system architecture of four superblock, index node and data block modules.Hereafter, various file system have all continued to use this One organizational structure.
In order to which I/O performance is continuously improved, there is within 1984 Flase File System (Fast File System, abbreviation FFS).It introduces the concept of cylinder cells (Cylinder Group), is as far as possible stored in several files under same catalogue In same group, several data blocks of same file are stored in same group, total tracking time can be substantially reduced in this way, Promote readwrite performance.After FFS, there is within 1989 log-structured file system (Log-structured File System), the thought that it uses log additional, carries out the write-in of file in the form of record log, and does on log-structured Index is used for the reading of file, greatly improves write performance, quick after can be realized file system crash using this structure Restore.
1994, with the birth of Linux1.0 kernel, extend file system (The Extended File System, Abbreviation ext) series initially enters people's sight, and ext1 is first Linux Virtual File System (Linux Virtual File System, abbreviation Linux VFS), manageable maximum disk space is 2GB, and ext1 various aspects are also shown slightly simple and crude at this time.ext2 Appearance, gradually come into vogue, it have can manage maximum disk space 16TB, maximum file size 2TB, longest filename Many advantages, such as 255 byte, but it has apparent defect in terms of log management, be not suitable for high to security requirement be System.2001, ext3 came into being, it joined sound journal function on the basis of ext2, solved the fatal weak of ext2 Point, the reliability of file system data can be greatly improved using ext3, even if improper delay machine occurs, only need after powering It can restore data within 10 seconds.In order to support bigger file, the bigger disk space of management, make several optimizations, produce Ext4 file system, it improves performance using delayed allocation, muti-piece distribution, without logging mode etc., and supports an infinite number of Subdirectory, online defragmentation, the persistently functions such as predistribution, are advanced optimized to ext3.
Ext3 is presently most popular Linux file system, selects representative of the ext3 as traditional file systems herein, Using ext3 as research object, the common feature of this kind of file system architectures is analyzed.
The traditional file systems such as ext3 for user provide high availability, high access speed, more logging modes support etc. it is all While more advantages, such traditional file systems framework also has its insufficient place, for small documents storage, tool The defect in terms of following three is embodied in for body: data distribution randomization, space waste and index node resource are limited.
Summary of the invention
The purpose of the present invention aims to solve at least one of described technological deficiency.
For this purpose, it is an object of the invention to propose a kind of building side of log file system based on LSM-Tree structure Method.
To achieve the goals above, the embodiment of the present invention provides a kind of log file system based on LSM-Tree structure Construction method, include the following steps:
Step S1 constructs the log file system fuse framework interface based on LSM-Tree structure, includes the following steps:
Step S11 calls fuse_main () function that fuse is file system mounted on mount point, and creation UNIX is local Socket creates and runs subprocess fusermount, and then calling fuse_new () function is that fuse file system distributes number According to memory space, carry is completed；
Step S12, after completing carry, fuse_main () function call fuse_loop () opens conversation modes, to user Conversational services are provided；
Step S13 is unloaded fuse file system using fusermount-uPATH order, then interrupts the session clothes Business, recycles corresponding memory space；
Step S2 constructs multiple directory operation functions and the file operation of the log file system based on LSM-Tree structure Function；
Step S3, added using Hash mapping function into the log file system based on LSM-Tree structure of building and Inquiry record data.
The construction method of log file system according to an embodiment of the present invention based on LSM-Tree structure, construction LevelFS file system can effectively improve catalogue, small documents under the premise of keeping big file read-write performance constant Readwrite performance.
Further, the multiple directory operation function include: directory creating function fs_mkdir, catalogue storage list function Fs_readdi, directory delete function fs_rmdir；
The multiple file manipulation function include: file renaming function fs_rename, File Open function fs_open, Function fs_truncate, file permission is arranged in file function reading fs_read, file write-in function fs_write, file size Modification growth function fs_chmod, file account information Modification growth function fs_chown, filesystem information function reading fs_statvfs, Document time stamp renewal function fs_utimens, be directed toward target Symbolic Links document creation function fs_symlink, The path inumber obtains function get_disk_path and disk file opens function open_disk_file.
Further, in the step S3, it is described using Hash mapping function to building based on LSM-Tree structure Addition record data, include the following steps: in log file system
If key is mapped to the k number between [0, m-1] respectively using k Hash mapping function, when needing to be written one When record, corresponding k number is found by mapping, the number in byte arrays in this k corresponding position is all then added 1, Show that there are such one records in system.
Further, in the step S3, it is described using Hash mapping function to building based on LSM-Tree structure Inquiry record data, include the following steps: in log file system
The corresponding position that record data are found by Hash mapping function, judges whether the value on each position is both greater than 0, If it is, reading the record data.
Further, further include following steps after the step S3: deletion record data then remember this in array Number on record corresponding position all subtracts 1.
The additional aspect of the present invention and advantage will be set forth in part in the description, and will partially become from the following description Obviously, or practice through the invention is recognized.
Detailed description of the invention
Above-mentioned and/or additional aspect of the invention and advantage will become from the description of the embodiment in conjunction with the following figures Obviously and it is readily appreciated that, in which:
Fig. 1 is the process according to the construction method of the log file system based on LSM-Tree structure of the embodiment of the present invention Figure；
Fig. 2 is the building according to the embodiment of the present invention based on the log file system fuse framework interface of LSM-Tree structure Flow chart；
Fig. 3 is the fuse framework interface of the log file system based on LSM-Tree structure according to the embodiment of the present invention Schematic diagram；
Fig. 4 is the mkdir order of the log file system based on LSM-Tree structure according to the embodiment of the present invention Data flowchart；
Fig. 5 is addition record x, y data structure diagram according to the embodiment of the present invention；
Fig. 6 is that p, q operational flowchart are inquired according to the embodiment of the present invention；
Fig. 7 is the deletion record data structure diagram according to the embodiment of the present invention.
Specific embodiment
The embodiment of the present invention is described below in detail, examples of the embodiments are shown in the accompanying drawings, wherein from beginning to end Same or similar label indicates same or similar element or element with the same or similar functions.Below with reference to attached The embodiment of figure description is exemplary, it is intended to is used to explain the present invention, and is not considered as limiting the invention.
The present invention deficiency low for the access efficiency of small documents for traditional file system, proposes that one kind is based on The construction method of the log file system of LSM-Tree (Log-Structured-Merge Tree) structure, building are based on LSM- The log file system of Tree structure realizes based on the usage log of above-mentioned basic framework the file system constructed LevelFS improves the effect to a large amount of small documents accessing operation so as to design the technical solution of access optimization for small documents Rate.
The construction method of log file system proposed by the present invention based on LSM-Tree structure, comprising the following steps: logical It crosses and write buffer is set in memory, the disk random write of several small documents is become into disk sequential write and writes efficiency to improve, simultaneously The storage distance of associated data is reduced on disk to improve reading efficiency, to improve the readwrite performance of file system.
As shown in Figure 1, the construction method of the log file system based on LSM-Tree structure of the embodiment of the present invention, including Following steps:
Step S1 constructs the log file system fuse framework interface based on LSM-Tree structure, includes the following steps:
Shown in referring to figs. 2 and 3, fuse framework interface is divided into three parts: carry, reply and unloading.
Step S11 calls fuse_main () function that fuse is file system mounted on mount point, and creation UNIX is local Socket creates and runs subprocess fusermount, and then calling fuse_new () function is that fuse file system distributes number According to memory space, carry is completed.
Carry part: user runs the executable file after compiling link, and fuse_main () function can be called to parse carry The parameters such as point, multithreading support, indicate the beginning of FUSE file system life cycle.Fuse_main () function can call Fuse_mount () function is file system mounted on mount point by fuse, creates the local UNIX socket, creates and run son Process fusermount, then calling fuse_new () function is that fuse file system distributes data space fuse_ Datastructure, carry finish.
Step S12, after completing carry, fuse_main () function call fuse_loop () opens conversation modes, to user Conversational services are provided.
Session section: fuse_main () function called after the completion of carry fuse_loop () open conversation modes, for Family provides the continuous clothes for receiving reply (receive session), handling session (process session), returning to session Business.
Fuse file system is unloaded using fusermount-uPATH order, then interrupts call back service by step S13, is returned Receive corresponding memory space.
Unload part: user is unloaded fuse file system using fusermount-u PATH order, then conversation, Memory space recycling, indicates the end of FUSE file system life cycle.
Step S2 constructs multiple directory operation functions and the file operation of the log file system based on LSM-Tree structure Function.
Fig. 4 is the data of the mkdir order of the log file system based on LSM-Tree structure described in the embodiment of the present invention Flow chart, as shown in figure 4, create directory " hi " under "/", after parsing layer by layer by taking the mkdir order under linux as an example Pass to int fs_mkdir (const char*path, mode_t mode) function of LevelFS.Lower mask body introduces one Lower catalogue, the function and main points of file operation correlation function.
In one embodiment of the invention, multiple directory operation functions include: directory creating function fs_mkdir, catalogue Function fs_readdi, directory delete function fs_rmdir are listed in storage.
(1)fs_mkdir
Function: it creaties directory on specified path.
Main points: firstly, path is resolved to parent directory par_path and subdirectory file dir_name, according to par_path The meta information for the also father node for finding the number par_inumber of father node, while taking out, judges to use in conjunction with account information Whether family has the permission that file is created under par_path catalogue, if returning to mistake without if.If had permission, for the mesh It records file and distributes i_number=cur_inumber+1, creation fs_inode data structure is believed as the metadata of new_dir Breath fills account information, timestamp information, and setting hard link number is 1, and i_number is arranged and is ++ cur_inumber, setting text Part type is catalogue.LevelDB finally is written into { par_inumber }/{ dir_name } -> { i_number }: { fs_inode } In.
(2)fs_readdir
Function: the directory entry list being stored on specified path is listed.The ls that this order corresponds to linux terminal refers to It enables.
Main points: firstly, catalogue file (detailed process please refers to fs_open) is opened according to path, if opening successfully Take out the i_number of directory node；Then, using the Iterator traversal key in LevelDB in range [i_number+ "/", i_number+1) in key-value record；Finally the filename that each item records is packed into filler list, It returns.
(3)fs_rmdir
Function: by the directory delete on specified path.
Main points: firstly, parent directory par_path and subdirectory file dir_name are equally parsed from path, according to Par_path finds number par_inumber, meta of father node, judges to delete permission；If had permission, only need to Write-in record { par_inumber }/{ dir_name } -> Delete in LevelDB.
Multiple file manipulation functions include: file renaming function fs_rename, File Open function fs_open, file Function reading fs_read, file write-in function fs_write, file size setting function fs_truncate, file permission modification Function fs_chmod, file account information Modification growth function fs_chown, filesystem information function reading fs_statvfs, file Update of time stamp function fs_utimens, the road document creation function fs_symlink, inumber for being directed toward target Symbolic Links Diameter obtains function get_disk_path and disk file opens function open_disk_file.
(4)fs_rename
Function: the file under specified path is renamed.
Main points: firstly, being equally to read parent directory, judge whether there is the permission write (just to repeat no more here parent directory ), take out par_inumber；Then, the Value information for reading { par_path }/{ old_name }, is denoted as r_value； Finally write-in record { par_inumber }/{ old_name } -> Delete and { par_inumber }/{ old_ into LevelDB Name } -> { r_value }.
(5)fs_open
Function: by the File Open on specified path.
Main points: firstly, reading parent directory, obtaining par_inumber and par_meta, checks permission according to par_meta；So The entry fs_record that key is { par_inumber }/{ filename } is stored in cache afterwards, creates a fs_ Openfile data structure, fills address pointer, user's account information of fs_record etc., and setting file current location is 0, starting position of the file in r_value is set, and it is 1 that the r_count in fs_record, which is arranged, file is successfully opened.
(6)fs_read
Function: the content on the designated position of the file on specified path is read.
Main points: it before the content for reading a file, needs to open file using fs_open first, at this moment file content has been Through in memory, the data volume recorded due to LevelDB every is smaller, by the data of file also one while taking file meta And it takes out.If file belongs to big file, is opened by get_disk_path and open_disk_file and be located at local file system Big file in system.Either big file or small documents finally pass through the f_start in fs_openfile data structure The read work of data is carried out with f_pos.
(7)fs_write
Function: data are write toward the designated position of the file on specified path.
Main points: as fs_read, before written document content, file is opened using fs_open first；Then by fs_ F_pos in openfile is adjusted to specified position, starts the write-in of data；With the write-in of data, f_pos is also constantly past After move.It should be noted that needed before each write operation check write after file total size, if file total size Threshold value is had exceeded, then is moved to the file content being stored in LevelDB originally in local file system.
(8)fs_truncate
Function: the new size of the file on setting specified path.
Main points: after the metadata information for obtaining file, the timestamp of corresponding size information and file is modified.It needs If should be noted that, the preceding document size of truncate is less than threshold, and file size is greater than after truncate Threshold is then needed to migrate the small documents in LevelDB into local file system, be deposited in the form of big file Storage, not vice versa.
(9)fs_chmod
Function: the new authority mark position of the file on modification specified path.
Main points: the metadata information of file is obtained by parent directory, stat therein is read, then by stat.st_mode It is changed to new mode, finally writes back modified entry.
(10)fs_chown
Function: the account information of the file on modification specified path.
Main points: it is similar with fs_chmod, the metadata information of file is obtained by parent directory first, is read therein Then stat is arranged stat.st_uid=uid and stat.st_gid=gid, finally writes back modified record item i.e. It can.
(11)fs_statvfs
Function: filesystem information, such as use space, free space, free time block tree, deposit statvfs are read In, this is the statistical information of local file system, and statvfs (path, the stbuf) interface for calling directly local file system is It can.
(12)fs_utimens
Function: the timestamp information of the file on specified path is updated.
Main points: it is similar with fs_chmode, the metadata information of file is obtained by parent directory first, is read therein Then new st_atim, st_mtim is arranged in stat, and for st_ctim, since creation time is operating system management, So not allowing user to modify.
(13)fs_symlink
Function: the file of a direction target Symbolic Links is created on specified path
Main points: obtaining the fs_inode of file by parent directory first, and the target character string being linked to is stored in In value, is_large=false is set, the path record of generation is write back.
(14)get_disk_path
Function: path corresponding to inumber is obtained
Main points: non-external interface mainly calculates corresponding index path, for reflecting for big file according to inumber It penetrates and plays an important role.Specifically, for the mapping ruler of big file for example,
Firstly, obtaining 64 positive integer i_number that LevelFS is file distribution, need to observe its feature.Although I_number range is 64, but it is constantly up increased since 0, is to be randomly assigned unlike memory address 's.This also just determines that, when file system scale is smaller, the possibility of i_number only has lower several positions to have data, and A most of high position is all 0；When file system is gradually increased, i_number data are also increasing, the digit of valid data It is more and more.
Then, this 64 positive integers are grouped since low level with 13 for one group, share 5 groups of numbers, is named as P0, p1, p2, p3, p4, in addition to p4 is discontented with 13, the range of remaining each group number is all 0~2^13-1.
Finally, LevelDB is mapped using following rule:
(1) if i_number < 226(namely 81922), then mapping path is/{ p1 }/{ p0 }, such as i_number= 9000, then the store path corresponding to it is just/1/808；
(2) if i_number >=226, and i_number < 239, then mapping path is /a/ { p2 }/{ p1 }/{ p0 }；
(3) if i_number >=239, and i_number < 252, then mapping path be /b/ { p3 }/{ p2 }/{ p1 }/ {p0}；
(4) if i_number >=252, and i_number < 264, then mapping path be /c/ { p4 }/{ p3 }/{ p2 }/ {p1}/{p0}。
It should be noted why LevelFS uses above-mentioned mapping ruler, main following points reason:
As soon as firstly, when the directory entry quantity in catalogue is excessive, the speed that scanned for it by filename What can be become is very slow.Therefore, LevelFS is using the thought being classified, with 13bit for one group, by the directory entry in each directory node Number control is 213Within the order of magnitude.
Then, if directlying adopt the mode of multiple index, every 13 are established level-one index, then retrieving a file 5 grades are needed, even retrieving No. 123 files in file system data amount very little and also needing 5 grades of indexes, such expense compares Greatly, also It is not necessary to.Therefore, LevelFS use nonequilibrium mode, when file system scale is smaller, with regard to use/ Two layers of indexed mode of { p1 }/{ p0 } can be reduced total index number compared with direct multiple index.
Finally, being more than 10 after i_number is very big8, the scale of supporting paper system is very big, at this moment Scale decision further according to i_number is removal search/path a/ or the/path b/ or the/path c/.
By the way of this non-equilibrium multiple index, the characteristics of inode number sequence increases can be met very well, so that file System no matter small-scale or it is fairly large can have suitable index series, reduce averagely index number as much as possible； Meanwhile and sufficiently limit the directory entry quantity in each catalogue, reduce the index time in each catalogue.
(15)open_disk_file
Function: the disk file of fs_inode_header*iheader object is opened.
Main points: non-external interface calls get_disk_path to obtain file path, then according to iheader.st_ino Flags is cooperated to call the open function of local file system, record opens the file information.
Step S3, added using Hash mapping function into the log file system based on LSM-Tree structure of building and Inquiry record data.
Record data are added into the log file system based on LSM-Tree structure of building using Hash mapping function, Include the following steps:
If key is mapped to the k number between [0, m-1] respectively using k Hash mapping function, when needing to be written one When record, corresponding k number is found by mapping, the number in byte arrays in this k corresponding position is all then added 1, Show that there are such one records in system.
Fig. 5 is addition record x, y data structure diagram described in the embodiment of the present invention, as shown in Figure 5:
Assuming that using k Hash mapping function, the k number that can respectively be mapped to key between [0, m-1].When needs are write When entering a record, corresponding k number is found by mapping, then by the number in byte arrays in this k corresponding position All plus 1, show that there are such one records in system.As an example it is assumed that k=3, write-in record x and y, hashs (x)=[2, 4,7], hashs (y)=[4,8,11], array at this time become shown in Fig. 5.
Record data are inquired into the log file system based on LSM-Tree structure of building using Hash mapping function, Include the following steps:
The corresponding position that record data are found by Hash mapping function, judges whether the value on each position is both greater than 0, If it is, reading the record data.
Fig. 6 is inquiry p, q operational flowchart described in the embodiment of the present invention, as shown in fig. 6, when needing read record, first Corresponding position is found by hashs, judges whether the value in each position is both greater than 0.If hashs (p)=[Isosorbide-5-Nitrae, 11], looks into number Corresponding number is 0 on 1 position of group discovery, then records p centainly not in LevelDB, need not continue to look for down, returns and reads to lose It loses；If hashs (q)=[4,8,11], the number in each position is all larger than 0, then records q and be possible in LevelDB, the reading of q Taking cannot be filtered.
After step s 3, further include following steps: deletion record data then record this in array on corresponding position Number all subtract 1.
Fig. 7 is deletion record data structure diagram described in the embodiment of the present invention, as shown in fig. 7, when needing to delete a record When, then this in array is recorded into the number on corresponding position and all subtract 1, the array content after deletion record x is as shown in Figure 7.
The construction method of log file system according to an embodiment of the present invention based on LSM-Tree structure, construction LevelFS file system can effectively improve catalogue, small documents under the premise of keeping big file read-write performance constant Readwrite performance.
In the description of this specification, reference term " one embodiment ", " some embodiments ", " example ", " specifically show The description of example " or " some examples " etc. means specific features, structure, material or spy described in conjunction with this embodiment or example Point is included at least one embodiment or example of the invention.In the present specification, schematic expression of the above terms are not Centainly refer to identical embodiment or example.Moreover, particular features, structures, materials, or characteristics described can be any One or more embodiment or examples in can be combined in any suitable manner.
Although the embodiments of the present invention has been shown and described above, it is to be understood that above-described embodiment is example Property, it is not considered as limiting the invention, those skilled in the art are not departing from the principle of the present invention and objective In the case where can make changes, modifications, alterations, and variations to the above described embodiments within the scope of the invention.The scope of the present invention It is extremely equally limited by appended claims.

Claims (4)

1. a kind of construction method of the log file system based on LSM-Tree structure, which comprises the steps of:
Step S1 constructs the log file system fuse framework interface based on LSM-Tree structure, includes the following steps:
Step S11 calls fuse_main () function that fuse is file system mounted on mount point, and creation UNIX is locally socketed Word creates and runs subprocess fusermount, and then calling fuse_new () function is that fuse file system distribution data are deposited Space is stored up, carry is completed；
Step S12, after completing carry, fuse_main () function call fuse_loop () opens conversation modes, provides a user Conversational services；
Fuse file system is unloaded using fusermount-uPATH order, then interrupts the conversational services by step S13, is returned Receive corresponding memory space；
Step S2 constructs the multiple directory operation functions and file operation letter of the log file system based on LSM-Tree structure Number；
Step S3 is added and is inquired into the log file system based on LSM-Tree structure of building using Hash mapping function Record data, comprising: set using k Hash mapping function, key is mapped to the k number between [0, m-1] respectively, when needs are write When entering a record, corresponding k number is found by mapping, then by the number in byte arrays in this k corresponding position All plus 1, show that there are such one records in system.
2. the construction method of the log file system as described in claim 1 based on LSM-Tree structure, which is characterized in that institute State multiple directory operation functions include: directory creating function fs_mkdir, catalogue storage list function fs_readdi, catalogue is deleted Except function fs_rmdir；
Multiple file manipulation functions include: file renaming function fs_rename, File Open function fs_open, file reading Function fs_truncate, file permission Modification growth function is arranged in function fs_read, file write-in function fs_write, file size Fs_chmod, file account information Modification growth function fs_chown, filesystem information function reading fs_statvfs, document time Stamp renewal function fs_utimens, the path document creation function fs_symlink, inumber for being directed toward target Symbolic Links are obtained Function get_disk_path and disk file is taken to open function open_disk_file.
3. the construction method of the log file system as described in claim 1 based on LSM-Tree structure, which is characterized in that It is described to be inquired using Hash mapping function into the log file system based on LSM-Tree structure of building in the step S3 Data are recorded, are included the following steps:
The corresponding position that record data are found by k Hash mapping function, judges whether the value on k position is both greater than 0, such as Fruit is then to read the record data.
4. the construction method of the log file system as described in claim 1 based on LSM-Tree structure, which is characterized in that Further include following steps after the step S3: deletion record data, and this in byte arrays is recorded on corresponding position Number all subtracts 1.
CN201610152908.3A 2016-03-17 2016-03-17 A kind of construction method of the log file system based on LSM-Tree structure Active CN105787093B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201610152908.3A CN105787093B (en) 2016-03-17 2016-03-17 A kind of construction method of the log file system based on LSM-Tree structure

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201610152908.3A CN105787093B (en) 2016-03-17 2016-03-17 A kind of construction method of the log file system based on LSM-Tree structure

Publications (2)

Publication Number Publication Date
CN105787093A CN105787093A (en) 2016-07-20
CN105787093B true CN105787093B (en) 2019-07-02

Family

ID=56392824

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201610152908.3A Active CN105787093B (en) 2016-03-17 2016-03-17 A kind of construction method of the log file system based on LSM-Tree structure

Country Status (1)

Country Link
CN (1) CN105787093B (en)

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106777146A (en) * 2016-12-19 2017-05-31 北京奇虎科技有限公司 Data processing method and device based on LSM tree structures
CN109189487B (en) * 2018-08-14 2021-08-31 郑州云海信息技术有限公司 Restarting method, system and related components of Ceph distributed storage system
CN111694992B (en) * 2019-03-15 2023-05-26 阿里巴巴集团控股有限公司 Data processing method and device
CN109960630B (en) * 2019-03-18 2020-09-29 四川长虹电器股份有限公司 Method for rapidly extracting logs from large-batch compressed files
CN110515897B (en) * 2019-07-17 2021-11-23 华东计算技术研究所（中国电子科技集团公司第三十二研究所） Method and system for optimizing reading performance of LSM storage system
CN111104403B (en) * 2019-11-30 2022-06-07 北京浪潮数据技术有限公司 LSM tree data processing method, system, equipment and computer medium
CN112862425B (en) * 2021-01-12 2021-11-12 江苏中威科技软件系统有限公司 Device and technology for reading OFD format file life cycle information
CN113688099B (en) * 2021-08-09 2023-10-13 上海沄熹科技有限公司 SPDK-based database storage engine acceleration method and system
CN113760829B (en) * 2021-08-23 2023-01-24 华东计算技术研究所（中国电子科技集团公司第三十二研究所） Safe storage system based on fuse file interface and operating system
CN113961514B (en) * 2021-12-20 2022-03-08 支付宝(杭州)信息技术有限公司 Data query method and device
CN114580325B (en) * 2021-12-31 2023-07-25 上海盈方微电子有限公司 Tarmac log analysis method in chip RTL verification stage

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103034684A (en) * 2012-11-27 2013-04-10 北京航空航天大学 Optimizing method for storing virtual machine mirror images based on CAS (content addressable storage)
CN104408091A (en) * 2014-11-11 2015-03-11 清华大学 Data storage method and system for distributed file system
CN104639658A (en) * 2015-03-12 2015-05-20 浪潮集团有限公司 Realization method for accessing object storage by file system mounting

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8402051B2 (en) * 2009-01-06 2013-03-19 Emc Corporation Consumer share quota feature

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103034684A (en) * 2012-11-27 2013-04-10 北京航空航天大学 Optimizing method for storing virtual machine mirror images based on CAS (content addressable storage)
CN104408091A (en) * 2014-11-11 2015-03-11 清华大学 Data storage method and system for distributed file system
CN104639658A (en) * 2015-03-12 2015-05-20 浪潮集团有限公司 Realization method for accessing object storage by file system mounting

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Linux Filesystem in Userspace(FUSE);adaptiver;《https://blog.csdn.net/adaptiver/article/details/7014807》;20111126;1-7 *
TABLEFS: Enhancing Metadata Efficiency in the Local File System;Kai Ren等;《USENIX ATC"13 Proceedings of the 2013 USENIX conference on Annual Technical Conference》;20130628;145-156 *

Also Published As

Publication number Publication date
CN105787093A (en) 2016-07-20

Similar Documents

Publication Publication Date Title
CN105787093B (en) A kind of construction method of the log file system based on LSM-Tree structure
EP3103025B1 (en) Content based organization of file systems
US8849759B2 (en) Unified local storage supporting file and cloud object access
CN100483420C (en) Fine grit document and catalogs version management method based on snapshot
CN102158546B (en) Cluster file system and file service method thereof
CN101866305B (en) Continuous data protection method and system supporting data inquiry and quick recovery
CN103282899B (en) The storage method of data, access method and device in file system
CN102129435B (en) Data storage service control method and system
US8010505B2 (en) Efficient backup data retrieval
CN102915278A (en) Data deduplication method
CN103577123A (en) Small file optimization storage method based on HDFS
CN106682003B (en) The path segmentation mapping method and device of distributed storage NameSpace
CN104408111A (en) Method and device for deleting duplicate data
CN108984686A (en) A kind of distributed file system indexing means and device merged based on log
CN109522283A (en) A kind of data de-duplication method and system
CN108287869A (en) A kind of mass small documents solution based on speedy storage equipment
CN103942301B (en) Distributed file system oriented to access and application of multiple data types
CN103177112B (en) A kind of backup browsing file system
CN105824867A (en) Mass file management system based on multi-stage distributed metadata
CN102831240A (en) Storage method and storage structure of extensible metadata documents
CN111782150A (en) Multi-bucket storage system and method based on object storage
CN106126555A (en) A kind of file management method and file system
Prabavathy et al. Multi-index technique for metadata management in private cloud storage
CN117435559B (en) Metadata hierarchical management method and device, storage medium and electronic equipment
CN105912267A (en) Processing method and device for file allocation tables

Legal Events

Date Code Title Description
C06 Publication
PB01 Publication
C10 Entry into substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant