CN109063103A - A kind of non-volatile file system of distribution - Google Patents

A kind of non-volatile file system of distribution Download PDF

Info

Publication number
CN109063103A
CN109063103A CN201810846379.6A CN201810846379A CN109063103A CN 109063103 A CN109063103 A CN 109063103A CN 201810846379 A CN201810846379 A CN 201810846379A CN 109063103 A CN109063103 A CN 109063103A
Authority
CN
China
Prior art keywords
memory headroom
band
storage
meta data
file system
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810846379.6A
Other languages
Chinese (zh)
Inventor
苏楠
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Zhengzhou Yunhai Information Technology Co Ltd
Original Assignee
Zhengzhou Yunhai Information Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Zhengzhou Yunhai Information Technology Co Ltd filed Critical Zhengzhou Yunhai Information Technology Co Ltd
Priority to CN201810846379.6A priority Critical patent/CN109063103A/en
Publication of CN109063103A publication Critical patent/CN109063103A/en
Pending legal-status Critical Current

Links

Landscapes

  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a kind of non-volatile file system of distribution, including multiple meta data servers, multiple storage servers, one or more nonvolatile memories are provided in each storage server, all nonvolatile memories are mapped to the memory headroom of an entirety;Wherein, any one meta data server, for when receiving the storage request for file destination, file destination to be divided into multiple bands;According to the size of each band, by search space can allocation table, distribute corresponding memory headroom for each band;Initial address and offset of each band in memory headroom are recorded in the metadata of file destination.Using technical solution provided by the embodiment of the present invention, file storage is carried out using nonvolatile memory, quick metadata access may be implemented, reduce the lookup time of metadata, reduces data copy, reduces read and write access delay.

Description

A kind of non-volatile file system of distribution
Technical field
The present invention relates to computer application technologies, more particularly to a kind of non-volatile file system of distribution.
Background technique
With the fast development of computer technology, performance gap is increasing between processor and memory, especially locates The gap for managing performance between device and external memory is particularly evident, so that I/O bottleneck problem becomes increasingly conspicuous.
To alleviate I/O bottleneck problem, conventional method is to reduce access external memory by the way that disk file caching is arranged in memory Number.
But demand of the growth of data explosion formula to memory size is increasing, and conventional method is by increasing buffer memory capacity band The benefit come is reduced instead, and frequent exchange data can generate biggish expense between interior external memory, and read and write access delay is larger.
Summary of the invention
The object of the present invention is to provide a kind of non-volatile file system of distribution to be subtracted with realizing quick metadata access Few metadata lookup time, data copy is reduced, reduces read and write access delay.
In order to solve the above technical problems, the invention provides the following technical scheme:
A kind of non-volatile file system of distribution, including multiple meta data servers, multiple storage servers, each storage One or more nonvolatile memories are provided in server, all nonvolatile memories are mapped to the memory headroom of an entirety;Its In,
Any one meta data server, for receive for file destination storage request when, by the target File is divided into multiple bands;According to the size of each band, by search space can allocation table, it is corresponding for the distribution of each band Memory headroom;Initial address and offset of each band in memory headroom are recorded in the metadata of the file destination Amount.
In a kind of specific embodiment of the invention,
Any one meta data server, when being specifically used for carrying storage position information in storage request, according to The size of each band, by search space can allocation table, be preferentially every in the corresponding storage server in the storage position A band distributes corresponding memory headroom.
In a kind of specific embodiment of the invention,
Any one meta data server, specifically for according to the size of each band and the load journey of storage server Degree, by search space can allocation table, distribute corresponding memory headroom for each band.
In a kind of specific embodiment of the invention,
Any one meta data server, specifically for the size according to each band, by search space can allocation table, It is that each band distributes corresponding memory headroom in the memory headroom that loading level is less than the storage server of preset threshold.
In a kind of specific embodiment of the invention,
Any one meta data server is also used to when monitoring that memory headroom has space release, can in the space The memory headroom of release is recorded in allocation table.
In a kind of specific embodiment of the invention,
Any one meta data server is also used to when receiving the storage request for file destination, by the mesh The wave file of mark file is mapped to memory headroom storage.
In a kind of specific embodiment of the invention, the ground for the whole memory headroom that all nonvolatile memories are mapped to Location is globally shared.
In a kind of specific embodiment of the invention, the ground for the whole memory headroom that all nonvolatile memories are mapped to Location includes virtual address and physical address.
In a kind of specific embodiment of the invention,
Any one meta data server, specifically for including recording each band in the metadata of the file destination Deposit initial fictive address and the offset in space.
In a kind of specific embodiment of the invention,
Any one meta data server is also used to when receiving the access request for the file destination, according to Initial fictive address and offset of the file destination in memory headroom determine the mesh by audit memory mapping table The physical start address of file is marked, and reads the file destination.
Using technical solution provided by the embodiment of the present invention, each storage server in distributed non-volatile file system In be provided with one or more nonvolatile memories, all nonvolatile memories are mapped to the memory headroom of an entirety, metadata clothes File destination is divided into multiple bands, according to each band when receiving the storage request for file destination by business device Size, by search space can allocation table, distribute corresponding memory headroom for each band, remember in the metadata of file destination Record initial address and offset of each band in memory headroom.File storage is carried out using nonvolatile memory, may be implemented Quick metadata access reduces the lookup time of metadata, reduces data copy, reduces read and write access delay.
Detailed description of the invention
In order to more clearly explain the embodiment of the invention or the technical proposal in the existing technology, to embodiment or will show below There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this Some embodiments of invention for those of ordinary skill in the art without creative efforts, can be with It obtains other drawings based on these drawings.
Fig. 1 is a kind of structural schematic diagram of the non-volatile file system of distribution in the embodiment of the present invention;
Fig. 2 is a kind of implementation flow chart of file memory method in the embodiment of the present invention.
Specific embodiment
In order to enable those skilled in the art to better understand the solution of the present invention, with reference to the accompanying drawings and detailed description The present invention is described in further detail.Obviously, described embodiments are only a part of the embodiments of the present invention, rather than Whole embodiments.Based on the embodiments of the present invention, those of ordinary skill in the art are not making creative work premise Under every other embodiment obtained, shall fall within the protection scope of the present invention.
The embodiment of the invention provides a kind of non-volatile file system of distribution, the non-volatile file system of the distribution includes Multiple meta data servers, multiple storage servers are provided with one or more nonvolatile memories, institute in each storage server There is nonvolatile memory to be mapped to the memory headroom of an entirety.The storage that file is carried out by means of nonvolatile memory, can be more preferable Ground utilizes the random access performance of non-volatile memory apparatus.The address for the whole memory headroom that all nonvolatile memories are mapped to Globally shared, the nodes such as all meta data servers, storage server can be seen consistent in distributed non-volatile file system View.I.e. each nonvolatile memory forms the memory headroom in each storage server and is managed collectively, and is mapped to an entirety Memory headroom, same address is directed toward the same position, this unify address space can byte addressing, by persistence it is non-volatile in Deposit composition.
As shown in Figure 1, for the structural schematic diagram of distributed non-volatile file system, including meta data server 1, metadata Server 2, meta data server 3, storage server 1, storage server 2, storage server 3 are set in each storage server It sets there are two nonvolatile memory, all nonvolatile memories are mapped to the memory headroom of an entirety, the memory headroom tool of the entirety It is distributed the characteristics such as formula, highly reliable, persistence, unified address, byte addressing.Client can be by file by meta data server It stores to memory headroom, while file can be read in memory headroom, execute corresponding I/O operation.It supports multi-user, concurrently visit It asks.
It should be noted that meta data server, storage server shown in Fig. 1, nonvolatile memory number are merely illustrative, In practical application, there can be more.
As shown in Fig. 2, any one meta data server passes through following steps in the non-volatile file system of the distribution Carry out file storage:
S210: when receiving the storage request for file destination, the file destination is divided into multiple bands;
S220: according to the size of each band, by search space can allocation table, distribute corresponding memory for each band Space;
S230: initial address and offset of each band in memory headroom are recorded in the metadata of the file destination Amount.
In embodiments of the present invention, client can be issued to corresponding meta data server for text according to actual needs The storage of part is requested.For any one meta data server in distributed non-volatile file system, metadata clothes File destination can be divided into multiple bands when receiving the storage request for file destination by business device.
Specifically, division rule can be preset, for example, divided according to being sized, last after division A stripe size is sized less than or equal to this, alternatively, dividing according to files in different types setting different size.Metadata clothes Business device can carry out band division to file destination according to the setting rule, obtain multiple bands.
After file destination is divided into multiple bands, search space can be passed through further according to the size of each band Can allocation table, distribute corresponding memory headroom for each band.Space can have the free time for being currently available for distribution by record in allocation table Memory headroom.By search space can allocation table, can know the size of the memory headroom of each free time.Each band it is big It is small may it is identical or different, for each band distribute corresponding memory headroom when, as far as possible progress size matching, avoid fractionlet and The formation in cavity.
In a kind of specific embodiment of the invention, any one meta data server is specifically used in storage request When carrying storage position information, according to the size of each band, by search space can allocation table, it is preferentially corresponding in storage position Storage server in be that each band distributes corresponding memory headroom.
In practical applications, client can file determines according to actual conditions storage position, such as according to storage service Load state, communication cost of device etc. determine, can carry storage position information in storage request in this way.Meta data server When receiving the storage request for file destination, the storage position information carried in storage request can be obtained.By target After file is divided into multiple bands, can according to the size of each band, by search space can allocation table, preferential selection storage The memory headroom of the corresponding storage server in position is allocated.When in the corresponding storage server in storage position not free time When depositing space, the memory headroom of other storage servers can be distributed according to default rule.
In another specific embodiment of the invention, any one meta data server is specifically used for according to each The size of band and the loading level of storage server, by search space can allocation table, distribute corresponding memory for each band Space.
Specifically, can according to the size of each band, by search space can allocation table, be less than in loading level default It is that each band distributes corresponding memory headroom in the memory headroom of the storage server of threshold value.
In embodiments of the present invention, meta data server is when receiving the storage request for file destination, Ke Yixian Obtain storage server loading level, by search space can allocation table, determine loading level be less than preset threshold storage The memory headroom of free time in server is that each band is adaptive in these memory headrooms according to the size of each band Distribute corresponding memory headroom.
After distributing corresponding memory headroom for each band, each band is recorded in the metadata of file destination in memory Initial address and offset in space.
Specifically, initial fictive address of each band in memory headroom can be recorded in the metadata of file destination And offset.
In embodiments of the present invention, the address for the whole memory headroom that all nonvolatile memories are mapped to includes virtually Location and physical address, i.e., the whole memory headroom that all nonvolatile memories are mapped to can be divided into virtual address space and physics Address space.Physical address space refers to PM (Persistent Memory, lasting memory), storage include PM metadata information, Memory mapping table, file system space etc..PM metadata information include PM interior size, metadata partition initial position, reflect The initial position of firing table and size, the initial position of file system space and size.Memory mapping table saves file system sky Between page (page) arrive virtual address mapping.File system space saves the data of true file, metadata information.
In one embodiment of the invention, any one meta data server is also used to receiving for file destination Access request when, according to initial fictive address of the file destination in memory headroom and offset, mapped by audit memory Table, determines the physical start address of file destination, and reads the file destination.
File destination is mapped to after being stored in memory headroom, client can access to file destination.Member Data server is when receiving the access request for file destination, according to initial fictive of the file destination in memory headroom Address and offset can determine the physical start address of file destination by audit memory mapping table, be originated based on the physics Address and offset can read file destination, respond access request.
It divides documents into page and is mapped to virtual address, guarantee that virtual address is continuous.Text need to only be recorded to the access of file The initial fictive address of part and offset can find the storage position of file.When offset is added in the initial position to file The position of file need to be recalculated.Virtual address is loaded in memory headroom, and the access efficiency of file system can be improved.Virtually The space structure of address includes superblock super block, metadata table inode table and file area file region.
In one embodiment of the invention, any one meta data server is also used to monitoring that memory headroom is free Between when discharging, the memory headroom of release can be recorded in allocation table in space.
The file stored in memory headroom may be because of reasons releasing memory spaces such as deletions.Meta data server is being supervised When measuring memory headroom has space release, the memory headroom of release can be recorded in allocation table in space, by the memory of release Space sub-distribution again, realizes the recycling and reusing of memory headroom.Space can the size of memory headroom can be with byte in allocation table For unit, the waste of memory headroom is avoided.
In one embodiment of the invention, any one meta data server is also used to receiving for file destination Storage request when, by the wave file of file destination be mapped to memory headroom storage.It can for the storage mode of wave file With consistent with file destination, details are not described herein for the embodiment of the present invention.
Using technical solution provided by the embodiment of the present invention, each storage server in distributed non-volatile file system In be provided with one or more nonvolatile memories, all nonvolatile memories are mapped to the memory headroom of an entirety, metadata clothes File destination is divided into multiple bands, according to each band when receiving the storage request for file destination by business device Size, by search space can allocation table, distribute corresponding memory headroom for each band, remember in the metadata of file destination Record initial address and offset of each band in memory headroom.File storage is carried out using nonvolatile memory, may be implemented Quick metadata access reduces the lookup time of metadata, reduces data copy, reduces read and write access delay.
Each embodiment in this specification is described in a progressive manner, the highlights of each of the examples are with it is other The difference of embodiment, same or similar part may refer to each other between each embodiment.
Professional further appreciates that, unit described in conjunction with the examples disclosed in the embodiments of the present disclosure And algorithm steps, can be realized with electronic hardware, computer software, or a combination of the two, in order to clearly demonstrate hardware and The interchangeability of software generally describes each exemplary composition and step according to function in the above description.These Function is implemented in hardware or software actually, the specific application and design constraint depending on technical solution.Profession Technical staff can use different methods to achieve the described function each specific application, but this realization is not answered Think beyond the scope of this invention.
The step of method described in conjunction with the examples disclosed in this document or algorithm, can directly be held with hardware, processor The combination of capable software module or the two is implemented.Software module can be placed in random access memory (RAM), memory, read-only deposit Reservoir (ROM), electrically programmable ROM, electrically erasable ROM, register, hard disk, moveable magnetic disc, CD-ROM or technology In any other form of storage medium well known in field.
Used herein a specific example illustrates the principle and implementation of the invention, and above embodiments are said It is bright to be merely used to help understand technical solution of the present invention and its core concept.It should be pointed out that for the common of the art , without departing from the principle of the present invention, can be with several improvements and modifications are made to the present invention for technical staff, these Improvement and modification are also fallen within the protection scope of the claims of the present invention.

Claims (10)

1. a kind of non-volatile file system of distribution, which is characterized in that including multiple meta data servers, multiple storage services Device is provided with one or more nonvolatile memories in each storage server, and all nonvolatile memories are mapped to an entirety Memory headroom;Wherein,
Any one meta data server, for receive for file destination storage request when, by the file destination It is divided into multiple bands;According to the size of each band, by search space can allocation table, for the distribution of each band it is corresponding in Deposit space;Initial address and offset of each band in memory headroom are recorded in the metadata of the file destination.
2. the non-volatile file system of distribution according to claim 1, which is characterized in that
Any one meta data server, when being specifically used for carrying storage position information in storage request, according to each The size of band, by search space can allocation table, be preferentially each in the corresponding storage server in the storage position Band distributes corresponding memory headroom.
3. the non-volatile file system of distribution according to claim 1, which is characterized in that
Any one meta data server, specifically for leading to according to the size of each band and the loading level of storage server Cross search space can allocation table, distribute corresponding memory headroom for each band.
4. the non-volatile file system of distribution according to claim 3, which is characterized in that
Any one meta data server, specifically for the size according to each band, by search space can allocation table, negative It is that each band distributes corresponding memory headroom that load degree, which is less than in the memory headroom of the storage server of preset threshold,.
5. according to claim 1 to described in any item non-volatile file system of distribution among 4, which is characterized in that
Any one meta data server is also used to when monitoring that memory headroom has space release, can be distributed in the space The memory headroom of release is recorded in table.
6. the non-volatile file system of distribution according to claim 1, which is characterized in that
Any one meta data server is also used to when receiving the storage request for file destination, by the target text The wave file of part is mapped to memory headroom storage.
7. the non-volatile file system of distribution according to claim 1, which is characterized in that all nonvolatile memories are mapped to Whole memory headroom address it is globally shared.
8. the non-volatile file system of distribution according to claim 7, which is characterized in that all nonvolatile memories are mapped to The address of whole memory headroom include virtual address and physical address.
9. the non-volatile file system of distribution according to claim 8, which is characterized in that
Any one meta data server, specifically for recording each band in the metadata of the file destination in memory sky Between in initial fictive address and offset.
10. the non-volatile file system of distribution according to claim 9, which is characterized in that
Any one meta data server is also used to when receiving the access request for the file destination, according to described Initial fictive address and offset of the file destination in memory headroom determine the target text by audit memory mapping table The physical start address of part, and read the file destination.
CN201810846379.6A 2018-07-27 2018-07-27 A kind of non-volatile file system of distribution Pending CN109063103A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810846379.6A CN109063103A (en) 2018-07-27 2018-07-27 A kind of non-volatile file system of distribution

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810846379.6A CN109063103A (en) 2018-07-27 2018-07-27 A kind of non-volatile file system of distribution

Publications (1)

Publication Number Publication Date
CN109063103A true CN109063103A (en) 2018-12-21

Family

ID=64835788

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810846379.6A Pending CN109063103A (en) 2018-07-27 2018-07-27 A kind of non-volatile file system of distribution

Country Status (1)

Country Link
CN (1) CN109063103A (en)

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110221779A (en) * 2019-05-29 2019-09-10 清华大学 The construction method of distributed persistence memory storage system
CN110474992A (en) * 2019-09-25 2019-11-19 苏州浪潮智能科技有限公司 A kind of document transmission method, device, equipment and storage medium
CN110807125A (en) * 2019-08-03 2020-02-18 北京达佳互联信息技术有限公司 Recommendation system, data access method and device, server and storage medium
CN111061652A (en) * 2019-12-18 2020-04-24 中山大学 Nonvolatile memory management method and system based on MPI-IO middleware
CN111435342A (en) * 2019-01-14 2020-07-21 深圳市茁壮网络股份有限公司 Poster updating method, poster updating system and poster management system
CN112114738A (en) * 2019-06-20 2020-12-22 杭州海康威视数字技术股份有限公司 Data storage method and storage device
CN112667577A (en) * 2020-12-25 2021-04-16 浙江大华技术股份有限公司 Metadata management method, metadata management system and storage medium

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101997918A (en) * 2010-11-11 2011-03-30 清华大学 Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment
CN104144150A (en) * 2013-05-10 2014-11-12 中国电信股份有限公司 Method, device and system for having access to information through metadata
CN104348793A (en) * 2013-07-30 2015-02-11 阿里巴巴集团控股有限公司 Storage server system and storage method for data information
CN107967124A (en) * 2017-12-14 2018-04-27 南京云创大数据科技股份有限公司 A kind of distribution persistence memory storage system and method
CN108268208A (en) * 2016-12-30 2018-07-10 清华大学 A kind of distributed memory file system based on RDMA

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101997918A (en) * 2010-11-11 2011-03-30 清华大学 Method for allocating mass storage resources according to needs in heterogeneous SAN (Storage Area Network) environment
CN104144150A (en) * 2013-05-10 2014-11-12 中国电信股份有限公司 Method, device and system for having access to information through metadata
CN104348793A (en) * 2013-07-30 2015-02-11 阿里巴巴集团控股有限公司 Storage server system and storage method for data information
CN108268208A (en) * 2016-12-30 2018-07-10 清华大学 A kind of distributed memory file system based on RDMA
CN107967124A (en) * 2017-12-14 2018-04-27 南京云创大数据科技股份有限公司 A kind of distribution persistence memory storage system and method

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111435342A (en) * 2019-01-14 2020-07-21 深圳市茁壮网络股份有限公司 Poster updating method, poster updating system and poster management system
CN111435342B (en) * 2019-01-14 2023-12-05 深圳市茁壮网络股份有限公司 Poster updating method, poster updating system and poster management system
CN110221779A (en) * 2019-05-29 2019-09-10 清华大学 The construction method of distributed persistence memory storage system
CN110221779B (en) * 2019-05-29 2020-06-19 清华大学 Construction method of distributed persistent memory storage system
CN112114738A (en) * 2019-06-20 2020-12-22 杭州海康威视数字技术股份有限公司 Data storage method and storage device
CN112114738B (en) * 2019-06-20 2024-02-20 杭州海康威视数字技术股份有限公司 Method for storing data and storage device
CN110807125A (en) * 2019-08-03 2020-02-18 北京达佳互联信息技术有限公司 Recommendation system, data access method and device, server and storage medium
CN110474992A (en) * 2019-09-25 2019-11-19 苏州浪潮智能科技有限公司 A kind of document transmission method, device, equipment and storage medium
CN110474992B (en) * 2019-09-25 2022-07-22 苏州浪潮智能科技有限公司 File transmission method, device, equipment and storage medium
CN111061652A (en) * 2019-12-18 2020-04-24 中山大学 Nonvolatile memory management method and system based on MPI-IO middleware
CN111061652B (en) * 2019-12-18 2021-12-31 中山大学 Nonvolatile memory management method and system based on MPI-IO middleware
CN112667577A (en) * 2020-12-25 2021-04-16 浙江大华技术股份有限公司 Metadata management method, metadata management system and storage medium

Similar Documents

Publication Publication Date Title
CN109063103A (en) A kind of non-volatile file system of distribution
US9058112B2 (en) Retrieving data in a storage system using thin provisioning
KR101726824B1 (en) Efficient Use of Hybrid Media in Cache Architectures
CN104899156B (en) A kind of diagram data storage and querying method towards extensive social networks
KR101767710B1 (en) Card-based management of discardable files
CN101556557B (en) Object file organization method based on object storage device
CN105389264B (en) The method and system of garbage collection in storage system
US11314689B2 (en) Method, apparatus, and computer program product for indexing a file
CN108804350A (en) A kind of memory pool access method and computer system
CN103116552A (en) Method and device for distributing storage space in distributed type storage system
US11809330B2 (en) Information processing apparatus and method
CN107632791A (en) The distribution method and system of a kind of memory space
CN109117088B (en) Data processing method and system
US8904128B2 (en) Processing a request to restore deduplicated data
CN109471843A (en) A kind of metadata cache method, system and relevant apparatus
CN111190537B (en) Method and system for managing sequential storage disk in additional writing scene
CN104079600B (en) File memory method, device, access client and meta data server system
KR20120074707A (en) Flash memory based storage and method for address mapping and data allocation therefor
US20170255393A1 (en) Storage device and storage method
CN117312261B (en) File compression encoding method, device storage medium and electronic equipment
CN109446115A (en) A kind of mapping table management method, device and computer readable storage medium
CN117312256A (en) File system, operating system and electronic equipment
KR20110070656A (en) Method and apparatus for processing data of flash memory
US9170929B2 (en) Memory controller
US11853577B2 (en) Tree structure node compaction prioritization

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication
RJ01 Rejection of invention patent application after publication

Application publication date: 20181221