CN110609817A - File storage system capable of preventing file fragmentation - Google Patents

File storage system capable of preventing file fragmentation Download PDF

Info

Publication number
CN110609817A
CN110609817A CN201810519947.1A CN201810519947A CN110609817A CN 110609817 A CN110609817 A CN 110609817A CN 201810519947 A CN201810519947 A CN 201810519947A CN 110609817 A CN110609817 A CN 110609817A
Authority
CN
China
Prior art keywords
file
memory
storage
data
block
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201810519947.1A
Other languages
Chinese (zh)
Inventor
戴瑾
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shanghai Ciyu Information Technologies Co Ltd
Original Assignee
Shanghai Ciyu Information Technologies Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shanghai Ciyu Information Technologies Co Ltd filed Critical Shanghai Ciyu Information Technologies Co Ltd
Priority to CN201810519947.1A priority Critical patent/CN110609817A/en
Publication of CN110609817A publication Critical patent/CN110609817A/en
Pending legal-status Critical Current

Links

Landscapes

  • Techniques For Improving Reliability Of Storages (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)

Abstract

The invention discloses a file storage system for preventing file fragmentation, which comprises: a first memory, a second memory and a main control unit; the first memory is a nonvolatile memory which adopts monoblock reading and writing; when the write-in operation of the large file is executed, continuously storing the data of the large file into a second memory, and when the data volume of the large file stored in the second memory reaches the capacity of one storage block, transferring the first batch of data into a first storage block of a first memory; and continuously storing the rest data of the large file into a second memory, and when the data volume reaches the capacity of one memory block again, transferring the second batch of data into a second memory block of the first memory until all the data of the large file is stored into the first memory. The invention greatly reduces the possibility of file fragmentation; the erasing frequency of the NAND is reduced, and the service life of the product is prolonged; it also allows the NAND to continue to maintain high performance after long term use.

Description

File storage system capable of preventing file fragmentation
Technical Field
The invention relates to a file storage system, in particular to a file storage system for preventing fragmentation of files, and belongs to the technical field of file storage.
Background
In computers, smart phones and tablet computers, user data and files are stored in hard disks or NAND flash memory chips and solid state disks composed of NAND flash memories. Replacement of traditional mechanical hard disks by NAND and solid state disks is a big trend in industry development.
NAND is a read-write-in-block memory device, writing (write 0) is much slower than reading, and erasing (write 1) is much slower than writing. The smallest readable and writable unit is called a page (page), the smallest erasable unit is called a block, one block often consists of a plurality of pages (page), and the pages (page) inside after the block (block) is erased can be independently written into (program). One problem with NAND flash memory is that NAND has a limited lifetime. After each page (page) is erased and written for a certain number of times, the page (page) will be permanently disabled and can not be used continuously. The current industry trend is that NAND capacity and data density increase very rapidly, but at the expense of reduced lifetime. The number of times of rewriting is reduced from the first 10 ten thousand times to 3000 times at present.
At the software level, the mobile phone and the computer are structured as follows: the application software sends file opening, closing, reading and writing instructions to an operating system; the file system software part in the operating system converts the instruction of reading and writing files into the instruction of reading and writing the storage block; the NAND driving and management software receives the instruction of reading and writing the memory block area, optimizes the cache, the write balance and the like, and sends the instructions of pages (pages), write blocks (blocks) and the like to the chip; in computers and mobile phones, NAND management software is usually run on a control chip inside an SSD or a memory card; in a mobile phone, sometimes NAND read-write software is also used as a software module closely related to an operating system and runs on a main control chip. The software system is somewhat simpler if a Hard Disk (HD) is used as the storage medium. And no complex NAND management software exists, and the file system software directly controls the reading and writing of the hard disk.
Because of the above characteristics of the NAND flash memory, the NAND management software inside the SSD is complicated. In order not to damage some blocks in advance where write operations often occur, a write leveling process is required. The logical and physical addresses identified by the file system software are different and a table is needed to correspond the two. Since NAND erasure is too slow, a content is generally modified without updating the original block area, but instead the new content is written to a new block area, the old block area is marked as invalid, and the CPU waits for its idle time to erase it. Thus, the lookup table of logical address physical addresses is constantly dynamically updated. This table is proportional to the total SSD capacity, and is stored in DDR DRAMs, with corresponding tags in NAND. With the rapid increase in SSD capacity on the market, this table becomes the largest consumer of DRAM.
The file system software is an important part in the operating system software and is responsible for solving the storage problem of files, and the interface comprises files creation, file opening, file reading, file writing and the like. It divides the file into individual blocks, typically each corresponding to a page in the NAND flash memory. Each time a new file is created, the filesystem software assigns a block to it, and when a block is full, assigns a new block. As many files in a system grow, different blocks in a file are often not contiguous in address. This is called fragmentation of the file. And after a file is deleted, the released free blocks can be occupied by a new file. So, as the product is used, the fragmentation of the file becomes more and more serious.
An important background to the present invention is the development of MRAM technology. MRAM is a new memory and storage technology that can be read and written randomly as fast as RAM, and can also permanently retain data after power is off, as NAND flash memory. Unlike NAND, MRAM can be erased an unlimited number of times. However, MRAM is expected to have a lower capacity and higher cost than NAND for a considerable period of time in the future.
If a hard disk is used, fragmentation of the file is a big problem because the head of the hard disk needs a long time to find a new address block, and the fragmentation of the file causes a significant degradation of the performance of the system.
If NAND storage is used, the time to find a new address is not a substantial problem and fragmentation has a relatively small impact on performance. Fragmentation still has a large impact on NAND lifetime. The smallest erasable unit of a NAND is a block (block) of 512 pages, which is erased when a 512-page file is deleted if the file is stored in 512 different blocks. If 512 files are stored in a block, when any one of the files is deleted or modified, the data of other files is erased and rewritten.
This aspect results in significant degradation of system performance after a period of use of the memory card or solid state drive due to the inability to keep up with the cleaning garbage. On the other hand, the longer NAND erasing times can shorten the NAND life, leading to the solid state disk or the mobile phone being scrapped in advance.
File fragmentation is a big problem for computing systems, and many organizations provide various solutions. US20110119462a1 provides an defragmentation algorithm.
Since the inception of MRAM into the industry, there has also been research into introducing MRAM into file storage systems. US008626987B2 is still the idea of defragmentation, in which MRAM is used as a buffer.
Defragmentation is accomplished by a user initiating a special program when severe fragmentation has occurred over a period of time that affects system performance. The specific operation process is shown in fig. 3, and is selected from fig. 17 of US008626987B 2.
This process requires checking all blocks in the entire storage device, and moving the file is very time consuming. Often resulting in several hours of system unavailability. After defragmentation, all pages in a logical block are translated into the same physical block. But fragmentation is really eliminated, and a file system is also needed to be matched to ensure that different pages in the same file are stored in the same logic block as much as possible. In this regard, the above-described disclosed invention does not provide any countermeasure.
Fragmentation causes a problem for a NAND flash memory-based storage system in that the number of times of erasing is too large, resulting in not only performance degradation after long-term use but also a shortened lifetime. For this, defragmentation procedures have limited effectiveness.
How to prevent or reduce fragmentation of a file, rather than eliminate it after fragmentation has occurred, is currently being studied very rarely.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: whether a hard disk or a NAND memory file is used, the prevention of file fragmentation is of great significance. However, it is necessary to ensure that a large file is stored in a whole block, and data that can be stored in one block is not distributed among different blocks. That is, it is most important to prevent file fragmentation at the block (block) level.
In order to solve the technical problem, the invention modifies the file system software and adds a new interface; the NAND management system was also modified to make a new interface. The MRAM is matched with the hard disk or the solid state disk to solve the problem of file fragmentation. The specific technical scheme is as follows:
a file storage system that prevents fragmentation of files, comprising:
a first memory, a second memory and a main control unit; the first memory is a nonvolatile memory which adopts a whole block of read-write, and the second memory is a nonvolatile memory which supports random access;
the main control unit runs file system software, and when the creation operation of the file is executed, the file system software is identified as a large file for the file which is equal to or larger than the capacity of one storage block of the first storage; for files with a storage block capacity smaller than that of the first storage, the file system software is marked as small files;
when the write operation of the large file is executed, first batch of data of the large file is continuously stored in a second memory, and when the data volume of the large file stored in the second memory reaches the capacity of one storage block, the first batch of data is transferred to the first storage block of the first memory;
and continuously storing the rest second batch of data of the large file into the second memory, and when the data volume reaches the capacity of one memory block again, transferring the second batch of data into the second memory block of the first memory, and repeating the steps until all the data of the large file is stored into the first memory.
Further, the file system software comprises: creating a file interface, identifying a file interface, opening a file interface, reading a file interface and writing a file interface.
Further, the main control unit records the writing time of each large file in the second memory, and when more large files are written to cause the storage space of the second memory to be insufficient, the file data which has not been written for the longest time is transferred to the first memory.
Further, the second memory is an MRAM, the first memory is a hard disk, and data is written in the hard disk in sequence.
Further, the second memory is an MRAM and the first memory is a NAND. The master control unit or the flash memory control unit runs NAND management software.
Further, a part of the storage space of the NAND is divided into a large file area, and another part of the storage space is divided into a small file area. For large file areas, translating logical addresses and physical addresses on a block basis; for a small file area, logical and physical addresses are translated on a page basis.
Further, the NAND management software comprises a whole block writing interface, when the large file is written, the whole block writing interface is called, and the data of the large file is written into the large file area.
Further, for a memory block which is not full in the large file area, a free page in the memory block can still accept the storage of a small file.
The invention has the beneficial effects that: the invention greatly reduces the possibility of file fragmentation. Reducing fragmentation of the file system reduces the erase-write times of the NAND, prolongs the product life, and also enables the NAND to maintain higher performance after long-term use. Because the page-based logical physical address translation table occupies a large amount of memory, the modification of the NAND management system based on the translation table of the block to the memory of a few percent ensures that the fragmentation of a large file caused by the translation of the logical physical address is avoided, and meanwhile, a part of the translation of the block is stored and used, the consumption of the memory is greatly reduced, and better performance can be obtained with lower cost.
Compared with the prior invention, the invention uses the large file area to automatically ensure that different pages in the same logic block are translated into the same physical block. The potential for fragmentation can be effectively reduced rather than eliminating it after it has occurred. The invention can reduce fragmentation eliminating operation affecting use.
Drawings
A more complete understanding of the present invention, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:
FIG. 1 is a diagram of a prior art architecture in which files are stored at the software level;
FIG. 2 is a schematic structural diagram of a file storage system according to an embodiment of the present invention;
fig. 3 is a flow diagram of defragmentation in the prior art.
Detailed Description
In order to make the aforementioned objects, features and advantages of the present invention comprehensible, embodiments accompanied with figures are described in detail below. It is to be noted that the drawings are in simplified form and are not to precise scale, which is provided for the purpose of facilitating and distinctly claiming the embodiments of the present invention.
The invention relates to modification of file system software and NAND management software in an operating system, in particular to a method for cooperatively storing files by utilizing MRAM and NAND, which mainly aims to ensure that a whole block (block) is used for keeping large files.
As shown in fig. 2, a file storage system provided in an embodiment of the present invention includes: a main control unit 201, a first memory 202, a second memory 203. The first memory 202 is a nonvolatile memory which adopts whole block read-write, the second memory 203 is a nonvolatile memory which supports random access memory, and the second memory 203 has longer service life and equal or faster read-write speed than the first memory 202. The first memory 202, in conjunction with the second memory 203, serves the task of storing files and user data.
The main control unit 201 runs a file system software, which includes: creating a file interface, opening the file interface, reading the file interface and writing the file interface, and increasing options of identifying the size file when creating the file interface. When the creation operation of the file is executed, for the file which is equal to or larger than the capacity of one storage block of the first storage, the file system software identifies the file as a large file; for files that are smaller than the capacity of one memory block of the first memory, the file system software identifies them as small files.
In a common file system, the attributes of each file are stored in a data structure called an inode. The information in the inode includes a file name, a storage address, various attributes such as owner, authority, last write time, and the like. All inodes are stored in a single sector of the storage device. Our modification requires the addition of an attribute in the inode: large/small files. When the file is created, the application software uses the size identification function to post this attribute to the inode.
When the write operation of the large file is executed, first data of the large file are continuously saved in the second memory 203, and when the data amount of the large file saved in the second memory 203 reaches the capacity of one storage block of the first memory 202, the first data are transferred to the first storage block 202 of the first memory. The remaining second batch of data of the large file is continuously saved in the second memory 203, and when the data amount reaches the capacity of one memory block of the first memory 202 again, the second batch of data is transferred to the second memory block of the first memory 202, and the process is repeated until the data of the large file is completely saved in the first memory 202. The inode records file storage position information, and in the execution of the steps, the storage position record in the inode is changed along with each step, so that the correct address can be found all the time when the application software reads and writes the file.
The main control unit 201 records the time of writing each large file in the second memory 203, and when more large files are written to cause the storage space of the second memory 203 to be insufficient, the file data which has not been written for the longest time is transferred to the first memory 202. The last write time of the file can be found in the inode.
In the present embodiment, the second Memory 203 is a Magnetic Random Access Memory (MRAM), and the first Memory 202 is a NAND flash Memory or a solid-state hard disk. In other embodiments, all the memories satisfying the corresponding conditions of "the nonvolatile memory supporting random access", "the nonvolatile memory using whole block reading and writing", and "the second memory has a longer service life and an equal or faster reading and writing speed than the first memory" may be implemented, for example: with the development of memory technology, a nonvolatile memory which supports random access memory, has a long service life and a high read/write speed and is more suitable than the MRAM may appear as a second memory in the future. Of course, the first memory 202 may also be a mechanical hard disk, in which data is written sequentially.
MRAM is a new Memory and storage technology, which can be read and written randomly as a Random Access Memory (RAM) and can permanently retain data after power-off as a NAND flash Memory. Unlike NAND, MRAM can be erased an unlimited number of times, but it is expected that MRAM will have lower capacity and higher cost than NAND for a considerable period of time in the future, so the embodiments of the present invention employ control of file storage in conjunction with MRAM and NAND, which assists NAND in serving both file and user data storage tasks.
In this embodiment, the life of the first memory 202 and the second memory 203 is measured by the number of times of erasing. Because of the technical characteristics of MRAM, it can be erased and written infinitely many times, and after some erasing and writing, NAND will be permanently ineffective and unable to be used, so the service life of MRAM is far longer than that of NAND.
In practical implementation, the main control unit 201 can be implemented by a combination of software and hardware, wherein the software part is embodied in the modification of the file system software in the operating system. The standard interface of the file system software comprises creating, opening, reading and writing files and the like, a unique file identification interface is added in the file system software in the embodiment, and when the application software creates the files, if the size of the known or expected files is close to or exceeds one block (usually 4-8MB), the created files are marked as large files. Or conversely, pages of expected shorter length are marked as small files.
The hardware part executes the file system software through a memory controller, a Central Processing Unit (CPU) and the like, for example, in a communication terminal such as a mobile phone and the like, the improved file system software in an operating system can run on a main control chip, and in a computer, the improved file system software is usually run on a solid state hard disk controller. Of course, those skilled in the art will appreciate that the corresponding file system software may also be implemented as firmware to hardware.
The main control unit 201 or the flash memory control unit runs NAND management software. A part of the storage space of the NAND is divided into a large file area, and the other part of the storage space is divided into a small file area. For a large file area, setting a translation table from a logical block address to a physical block address, and translating the logical address and the physical address on a block basis; for the small file area, a translation table from a logical page address to a physical page address is set, and the logical address and the physical address are translated on a page basis. The NAND management software includes a one-block write interface corresponding to the file identification interface of the file system software. When the large file is written, the whole block writing interface is called, and the data of the large file is written into the large file area. For a memory block that is not full in the large file area, the start of a free page is marked if not full. The free pages in the memory block can still accept the storage of small files.
The invention utilizes the combination of random nonvolatile storage and block storage equipment to greatly reduce the possibility of file fragmentation. A large file in a file system, such as a copy, a network-incoming file, a photo, a video, and other multimedia files, is mostly created by application software to determine that it is a large file. And these files are mostly written continuously for a short time. These files will be stored in a complete block (block) after temporary storage by the MRAM in the system using the present invention. Reducing fragmentation of the file system can reduce the erasing frequency of NAND and prolong the service life of the product. It also results in higher performance being maintained over extended periods of use.
The modification of the NAND management system ensures that the large file fragmentation caused by the translation of the logical physical address is avoided, and simultaneously greatly reduces the consumption of the RAM. Since the vast majority of a file system is occupied by large files. The logical-to-physical address mapping table required for page-based translation requires a large amount of memory, 512 pages for a block, and is smaller than page-based 1/512. So that better performance can be achieved at lower cost.
The foregoing detailed description of the preferred embodiments of the invention has been presented. It should be understood that numerous modifications and variations could be devised by those skilled in the art in light of the present teachings without departing from the inventive concepts. Therefore, the technical solutions available to those skilled in the art through logic analysis, reasoning and limited experiments based on the prior art according to the concept of the present invention should be within the scope of protection defined by the claims.

Claims (9)

1. A file storage system, comprising:
a first memory, a second memory and a main control unit; the first memory is a nonvolatile memory which adopts a whole block of reading and writing, and the second memory is a nonvolatile memory which supports random access memory;
the main control unit runs file system software, and when the creation operation of a file is executed, the file system software is identified as a large file for the file which is equal to or larger than the capacity of one storage block of the first storage; or for files with a storage block capacity smaller than that of the first storage, the file system software is identified as small files;
when the write operation of the large file is executed, first batch of data of the large file is continuously saved in the second memory, and when the data volume of the large file saved in the second memory reaches the capacity of one storage block, the first batch of data is completely transferred to the first storage block of the first memory;
and continuously storing the rest second batch of data of the large file into the second memory, and when the data volume reaches the capacity of one storage block again, transferring the second batch of data into the second storage block of the first memory, and repeating the steps until all the data of the large file is stored into the first memory.
2. The file storage system according to claim 1, wherein said master control unit records the last time each large file in the second storage was written, and when more large files were written to cause the storage space of said second storage to be insufficient, the file data which has not been written for the longest time is transferred to said first storage.
3. The file storage system of claim 2, wherein said second memory is an MRAM and said first memory is a hard disk, and wherein data is written to said hard disk sequentially.
4. The file storage system of claim 2, wherein the second memory is an MRAM and the first memory is a NAND.
5. The file storage system according to claim 4, wherein the main control unit or the flash memory control unit runs NAND management software.
6. The file storage system according to claim 5, wherein a part of the storage space of the NAND is divided into large file areas, and another part of the storage space of the NAND is divided into small file areas.
7. The file storage system according to claim 6, wherein for said large file area, logical addresses and physical addresses are translated on a block basis; for the cell file region, logical addresses and physical addresses are translated on a page basis.
8. The file storage system according to claim 7, wherein said NAND management software includes a bulk write interface, and when a write operation of said large file is performed, said bulk write interface is called, and data of said large file is written to said large file area.
9. A file storage system according to claim 8, wherein for a storage block not full in the large file area, a free page in the storage block can still accept the storage of the small file.
CN201810519947.1A 2018-05-28 2018-05-28 File storage system capable of preventing file fragmentation Pending CN110609817A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201810519947.1A CN110609817A (en) 2018-05-28 2018-05-28 File storage system capable of preventing file fragmentation

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201810519947.1A CN110609817A (en) 2018-05-28 2018-05-28 File storage system capable of preventing file fragmentation

Publications (1)

Publication Number Publication Date
CN110609817A true CN110609817A (en) 2019-12-24

Family

ID=68887539

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201810519947.1A Pending CN110609817A (en) 2018-05-28 2018-05-28 File storage system capable of preventing file fragmentation

Country Status (1)

Country Link
CN (1) CN110609817A (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120173795A1 (en) * 2010-05-25 2012-07-05 Ocz Technology Group, Inc. Solid state drive with low write amplification
CN105589912A (en) * 2015-03-27 2016-05-18 上海磁宇信息科技有限公司 Method and storage structure for processing frequently written document with MRAM (Magnetic Random Access Memory)
CN105630697A (en) * 2015-04-15 2016-06-01 上海磁宇信息科技有限公司 Storage structure for storing small file by MRAM
CN105630403A (en) * 2015-03-27 2016-06-01 上海磁宇信息科技有限公司 File storage system and file storage control method and apparatus therefor

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120173795A1 (en) * 2010-05-25 2012-07-05 Ocz Technology Group, Inc. Solid state drive with low write amplification
CN105589912A (en) * 2015-03-27 2016-05-18 上海磁宇信息科技有限公司 Method and storage structure for processing frequently written document with MRAM (Magnetic Random Access Memory)
CN105630403A (en) * 2015-03-27 2016-06-01 上海磁宇信息科技有限公司 File storage system and file storage control method and apparatus therefor
CN105630697A (en) * 2015-04-15 2016-06-01 上海磁宇信息科技有限公司 Storage structure for storing small file by MRAM

Similar Documents

Publication Publication Date Title
US11967369B2 (en) Lifetime mixed level non-volatile memory system
CN105718530B (en) File storage system and file storage control method thereof
US10915475B2 (en) Methods and apparatus for variable size logical page management based on hot and cold data
US9720616B2 (en) Data-retention controller/driver for stand-alone or hosted card reader, solid-state-drive (SSD), or super-enhanced-endurance SSD (SEED)
US8055873B2 (en) Data writing method for flash memory, and controller and system using the same
CN110955384B (en) Data storage device and non-volatile memory control method
US20070094445A1 (en) Method to enable fast disk caching and efficient operations on solid state disks
WO2014074449A2 (en) Wear leveling in flash memory devices with trim commands
KR20100065786A (en) Cache synchronization method and system for fast power-off
US9176866B2 (en) Active recycling for solid state drive
CN110674056B (en) Garbage recovery method and device
KR101403922B1 (en) Apparatus and method for data storing according to an access degree
US8429339B2 (en) Storage device utilizing free pages in compressed blocks
US20090132757A1 (en) Storage system for improving efficiency in accessing flash memory and method for the same
CN112130749B (en) Data storage device and non-volatile memory control method
TWI718710B (en) Data storage device and non-volatile memory control method
CN105607862A (en) Solid state disk capable of combining DRAM (Dynamic Random Access Memory) with MRAM (Magnetic Random Access Memory) and being provided with backup power
US20200310669A1 (en) Optimized handling of multiple copies in storage management
CN110727470B (en) Hybrid nonvolatile memory device
CN113986773A (en) Write amplification optimization method and device based on solid state disk and computer equipment
JP5259257B2 (en) Storage device
KR102589609B1 (en) Snapshot management in partitioned storage
CN107765989B (en) Storage device control chip, storage device and storage device management method
CN115878033A (en) Solid state disk and mapping table management method thereof
CN109002265B (en) Data processing method and related device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20191224

RJ01 Rejection of invention patent application after publication