CN111273868A - Method for garbage recovery and write amplification reduction of full flash memory array - Google Patents
Method for garbage recovery and write amplification reduction of full flash memory array Download PDFInfo
- Publication number
- CN111273868A CN111273868A CN202010059623.1A CN202010059623A CN111273868A CN 111273868 A CN111273868 A CN 111273868A CN 202010059623 A CN202010059623 A CN 202010059623A CN 111273868 A CN111273868 A CN 111273868A
- Authority
- CN
- China
- Prior art keywords
- data
- flash memory
- extent
- written
- logical volume
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000003321 amplification Effects 0.000 title claims abstract description 19
- 238000003199 nucleic acid amplification method Methods 0.000 title claims abstract description 19
- 238000000034 method Methods 0.000 title claims abstract description 18
- 230000009467 reduction Effects 0.000 title claims abstract description 12
- 238000011084 recovery Methods 0.000 title description 5
- 238000013500 data storage Methods 0.000 claims description 3
- 230000006870 function Effects 0.000 claims description 3
- 230000003247 decreasing effect Effects 0.000 claims description 2
- 230000008569 process Effects 0.000 abstract description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000009977 dual effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000005055 memory storage Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 230000003449 preventive effect Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/064—Management of blocks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for garbage collection and write amplification reduction of a full flash memory array, which comprises the write operation of a logical volume and the read operation of the logical volume, and comprises the following steps: s1: firstly, calculating the hash value of the data object and the VBN of the logical volume; s2: then, the data object is written into the extent of the NVDIMM, the object is written from the front section of the extent to the back section in sequence, and mPu points to the position of the next object to be written; s3: the objectrecord is written from the back end of the extension in sequence, and mDown points to the position of the objectrecord to be written next; s4: when mDown minus mUp is smaller than 4KB +32B, the extent is full, a background thread is started, and the content in the extent is flushed to a back-end flash memory; according to the invention, the NVDIMM is used for writing and caching, so that the server is ensured not to lose power-off data in the writing process of data, the logical volume of the NVDIMM can continuously write the memory data into the rear-end flash memory, and the safety and integrity of the data are ensured.
Description
Technical Field
The invention relates to the technical field of flash memory storage systems, in particular to a method for garbage collection and write amplification reduction of a full flash memory array.
Background
With the cost reduction and the technical progress, the flash memory is increasingly used as a storage medium by an enterprise-level data center, and compared with the traditional magnetic disk medium, the advantages of the flash memory are obvious: high speed, low power consumption and space saving. Flash memory has an overwhelming advantage, particularly in terms of speed.
Flash memory is an electronic medium, and the writing mode of the flash memory processing data is different from the overlay writing of the traditional magnetic medium (magnetic disk or magnetic tape): the block to which new data is written must be a blank block, and if the block contains old data, the old data must be "erased" (Erase). Particularly, when the total amount of data written reaches all the capacity provided by the flash memory, and then the update is written again, the "erasure" is inevitably generated.
Compared to conventional read and write operations, erasing has 2 features:
firstly, the erasing speed is slower than the reading and writing speed, the erasing time can be more than 10 times of the writing time, and the ms level is basically achieved;
secondly, the processing unit of erasing is larger than that of reading and writing, and the processing unit of reading and writing is page, which is generally 4KB in size; the unit of erasure is block, which is composed of multiple pages, and is generally not less than 512 KB. If 4KB of data is to be written to a block that already has 4KB of old data, then the block space of 512KB of its periphery needs to be erased first.
This is the write amplification of the garbage collection of the flash memory
Flash memory write amplification is the inherent characteristic of flash memory media, cannot be avoided, and flash memory write amplification can be controlled to a certain range only by means of optimization of media materials, algorithms and architectures.
Such as: the Over-Provisioning approach, by providing a greater number of particles, is used to slow down the wear of the overall flash life and achieve more flexible preventive garbage recovery, with the drawback of purchasing flash of capacity 40TB, while the actual physical capacity exceeds 67TB, which is costly and wasteful.
Therefore, a method for garbage collection and write amplification reduction of a full flash memory array is provided.
Summary of the invention
1. Technical problem to be solved
Aiming at the problems in the prior art, the invention aims to provide a method for garbage collection and write amplification reduction of a full flash memory array, wherein NVDIMM is used for writing cache, so that the server power-off data can not be lost in the writing process of data, and after the service is powered on, the logical volume of the NVDIMM can continuously write memory data into a rear-end flash memory, so that the safety and the integrity of the data are ensured.
2. Technical scheme
In order to solve the above problems, the present invention adopts the following technical solutions.
A method for garbage collection and write amplification reduction of a full flash memory array comprises the steps of writing operation of a logical volume and reading operation of the logical volume;
the write operation of the logical volume comprises the following steps:
s1: when write operation is performed, the hash value of the data object and the VBN of the logical volume, namely the logical block number, are calculated and recorded;
s2: then, the data object is written into the extent of the NVDIMM, the object is written from the front section of the extent to the back section in sequence, and mPu points to the position of the next object to be written;
s3: the object metadata object is written from the back end of the extent to the front end in sequence, and mDown points to the position of the next object to be written;
s4: when mDown minus mUp is less than 4KB +32B, the extent will start a background thread because it is full;
s5: directly writing the data of the extension 16MB to a back-end flash memory at a corresponding position;
the read operation of the logical volume comprises the following steps:
s1: when a read operation is performed, the hash value of the data object of the VBN is inquired through the VBN of the logical volume, namely the logical block number;
s2: retrieving the metadata objectrecord through the hash value;
s3: data is read by the PBN in objectrecord locating the object to the NVDIMM.
Furthermore, the full flash memory array uses NVDIMM memory, i.e. non-volatile dual in-linear module, with power-off data storage function, as a write operation auxiliary module for an entire column of the flash memory, data may be in units of 4KB, each 4KB data is called an object, each object has a metadata object of 32 bytes, a hash value of the object is recorded, and a calculated physical location PBN of the object stored in the flash memory, i.e. a physical block number, NVDIMM is divided into 16 MB-sized blocks, each block is called an extension, each extension has its own data structure, and is divided into an extension header and an extension header, the extension header records an extension header with a size of 4KB, and the extension header stores the extension object and the extension record.
Further, mcackendendex indicates to which back-end flash memory device the object belongs, mcentindex indicates the location of the extent on the back-end flash memory, mUp indicates the subscript location of the object stored from front to back in the extent, increasing upwards, mDown indicates the subscript location of the object stored from back to front in the extent, decreasing downwards, the object data stored in mData and the object record.
Further, the back-end flash memory device is also divided into blocks of the same size as the extensions.
3. Advantageous effects
Compared with the prior art, the invention has the advantages that:
the invention aims to provide a method for garbage recovery and write amplification reduction of a full flash memory array, and 1, an NVDIMM is used for writing cache, so that the server power-off data can be ensured not to be lost in the data writing process, and after the service is powered on, a logic volume of the NVDIMM can continuously write memory data into a back-end flash memory, so that the safety and the integrity of the data are ensured.
2. The NVDIMM is used as a write operation auxiliary module of the flash memory array, so that the performance of write operation can be improved, and a cache effect is achieved.
3. The NVDIMM is divided into an extent with equal size and a flash memory physical medium to correspond to each other, and after the extent of the NVDIMM is written fully, data of the extent is written into the flash memory in a flashing mode, so that a plurality of small writing blocks of the flash memory are aggregated into a large writing block operation, and the problem of flash memory write amplification is solved.
Drawings
FIG. 1 is a flow chart illustrating a logical volume write operation of the present invention;
FIG. 2 is a flow chart illustrating a logical volume read operation according to the present invention;
FIG. 3 is a schematic diagram of an ExtentPoyload storage structure according to the present invention;
fig. 4 is a schematic structural diagram of the patent of the present invention.
Detailed Description
The technical solution in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention; it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all embodiments, and all other embodiments obtained by those skilled in the art without any inventive work are within the scope of the present invention.
Example 1:
referring to fig. 1 and fig. 2, a method for garbage collection and write amplification reduction of a full flash array includes a write operation of a logical volume and a read operation of the logical volume;
the write operation of the logical volume comprises the following steps:
s1: when write operation is performed, the hash value of the data object and the VBN of the logical volume, namely the logical block number, are calculated and recorded;
s2: then, the data object is written into the extent of the NVDIMM, the object is written from the front section of the extent to the back section in sequence, and mPu points to the position of the next object to be written;
s3: object metadata object is written from the back end of the extension in sequence, and mDown points to the position of the next object to be written;
s4: when mDown minus mUp is less than 4KB +32B, the extent will start a background thread because it is full;
s5: directly writing the data of the extension 16MB to a back-end flash memory at a corresponding position;
the read operation of the logical volume comprises the following steps:
s1: when a read operation is performed, the hash value of the data object of the VBN is inquired through the VBN of the logical volume, namely the logical block number;
s2: retrieving the metadata objectrecord through the hash value;
s3: data is read by the PBN in objectrecord locating the object to the NVDIMM.
Referring to fig. 3, the NVDIMM memory using the full flash memory array, i.e., non-volatile dual in-memory module, has a power-off data storage function, and serves as a write operation auxiliary module for an entire flash memory array, data is stored in 4KB units, each 4KB data is called an object, each object has a metadata object of 32 bytes, a hash value of the object is recorded, and a calculated physical location PBN of the object stored in the flash memory, i.e., a physical block number, the NVDIMM is divided into 16 MB-sized blocks, each block is called an extent, each extent has its own data structure, and is divided into an extent header and an extent header, the extent header records an extent header with a size of 4KB, and the extent header stores the extent object and the extent object,
referring to fig. 3, mcackendendex indicates to which back-end flash memory device the object belongs, mcentindex indicates the location of the extent on the back-end flash memory, mUp indicates the subscript location of the object stored from front to back in the extent, which is incremented upward, mDown indicates the subscript location of the object record stored from back to front in the extent, which is decremented downward, the object data and the object record stored in mData, and the back-end flash memory device is also divided into blocks with the same size as the extent, and please refer to fig. 4 for specific structure.
The invention aims to provide a method for garbage recovery and write amplification reduction of a full flash memory array, and 1, an NVDIMM is used for writing cache, so that the server power-off data can be ensured not to be lost in the data writing process, and after the service is powered on, a logic volume of the NVDIMM can continuously write memory data into a back-end flash memory, so that the safety and the integrity of the data are ensured.
2. The NVDIMM is used as a write operation auxiliary module of the flash memory array, so that the performance of write operation can be improved, and a cache effect is achieved.
3. The NVDIMM is divided into an extent with equal size and a flash memory physical medium to correspond to each other, and after the extent of the NVDIMM is written fully, data of the extent is written into the flash memory in a flashing mode, so that a plurality of small writing blocks of the flash memory are aggregated into a large writing block operation, and the problem of flash memory write amplification is solved.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.
Claims (4)
1. A method for garbage collection and write amplification reduction of a full flash memory array is characterized by comprising the steps of writing operation of a logical volume and reading operation of the logical volume;
the write operation of the logical volume comprises the following steps:
s1: when write operation is performed, the hash value of the data object and the VBN of the logical volume, namely the logical block number, are calculated and recorded;
s2: then, the data object is written into the extent of the NVDIMM, the object is written from the front section of the extent to the back section in sequence, and mPu points to the position of the next object to be written;
s3: the object metadata object is written from the back end of the extent to the front end in sequence, and mDown points to the position of the next object to be written;
s4: when mDown minus mUp is less than 4KB +32B, the extent will start a background thread because it is full;
s5: directly writing the data of the extension 16MB to a back-end flash memory at a corresponding position;
the read operation of the logical volume comprises the following steps:
s1: when a read operation is performed, the hash value of the data object of the VBN is inquired through the VBN of the logical volume, namely the logical block number;
s2: retrieving the metadata objectrecord through the hash value;
s3: data is read by the PBN in objectrecord locating the object to the NVDIMM.
2. The rule according to claim 1 for a method of full flash array garbage collection with reduced write amplification, wherein: the NVDIMM memory used by the full flash memory array, namely non-volatile in-line memory module, has a power-off data storage function, and serves as a write operation auxiliary module of a flash memory whole column, data takes 4KB as a unit, each 4KB of data is called object, each object has 32 bytes of metadata Objectrecord, the hash value of the object is recorded, the calculated physical position PBN (physical block number) of the object stored in the flash memory is divided into 16 MB-sized sub-blocks, each sub-block is called extension, each extension has a data structure, the sub-block has a header information size of 4KB of the extension, the extension header records the extension, and the extension stores the object and the object.
3. The rule according to claim 2 for a method of full flash array garbage collection with reduced write amplification, wherein: mBackendIndex indicates to which back-end flash memory device the object belongs, mExtentIndex indicates the position of the extent on the back-end flash memory, mUp indicates the subscript position of the object stored from front to back in the extent, increasing upwards, mDown indicates the subscript position of the object stored from back to front in the extent, decreasing downwards, and the object data and the object record stored in mData.
4. The rule according to claim 3 for a method of full flash array garbage collection with reduced write amplification, wherein: the back-end flash devices are also divided into blocks of the same size as the extensions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010059623.1A CN111273868A (en) | 2020-01-19 | 2020-01-19 | Method for garbage recovery and write amplification reduction of full flash memory array |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010059623.1A CN111273868A (en) | 2020-01-19 | 2020-01-19 | Method for garbage recovery and write amplification reduction of full flash memory array |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN111273868A true CN111273868A (en) | 2020-06-12 |
Family
ID=70997465
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010059623.1A Pending CN111273868A (en) | 2020-01-19 | 2020-01-19 | Method for garbage recovery and write amplification reduction of full flash memory array |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111273868A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI795249B (en) * | 2022-03-29 | 2023-03-01 | 睿寬智能科技有限公司 | Balanced method for controlling a speed of writing in a flash memory |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101526922A (en) * | 2009-04-03 | 2009-09-09 | 深圳市宝捷信科技有限公司 | Flash data access method and device thereof |
| CN104794070A (en) * | 2015-04-23 | 2015-07-22 | 南京道熵信息技术有限公司 | Solid-state flash memory write cache system and method based on dynamic non-covering RAID technology |
| CN106469119A (en) * | 2015-08-10 | 2017-03-01 | 北京忆恒创源科技有限公司 | A kind of data write buffer method based on NVDIMM and its device |
| CN107256196A (en) * | 2017-06-13 | 2017-10-17 | 北京中航通用科技有限公司 | The caching system and method for support zero-copy based on flash array |
| CN110557964A (en) * | 2018-03-30 | 2019-12-10 | 华为技术有限公司 | Data writing method, client server and system |
-
2020
- 2020-01-19 CN CN202010059623.1A patent/CN111273868A/en active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101526922A (en) * | 2009-04-03 | 2009-09-09 | 深圳市宝捷信科技有限公司 | Flash data access method and device thereof |
| CN104794070A (en) * | 2015-04-23 | 2015-07-22 | 南京道熵信息技术有限公司 | Solid-state flash memory write cache system and method based on dynamic non-covering RAID technology |
| CN106469119A (en) * | 2015-08-10 | 2017-03-01 | 北京忆恒创源科技有限公司 | A kind of data write buffer method based on NVDIMM and its device |
| CN107256196A (en) * | 2017-06-13 | 2017-10-17 | 北京中航通用科技有限公司 | The caching system and method for support zero-copy based on flash array |
| CN110557964A (en) * | 2018-03-30 | 2019-12-10 | 华为技术有限公司 | Data writing method, client server and system |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| TWI795249B (en) * | 2022-03-29 | 2023-03-01 | 睿寬智能科技有限公司 | Balanced method for controlling a speed of writing in a flash memory |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10915475B2 (en) | Methods and apparatus for variable size logical page management based on hot and cold data | |
| US10810127B2 (en) | Solid-state hard disk and data access method for use with solid-state hard disk | |
| EP2646919B1 (en) | Transaction log recovery | |
| CN102508788B (en) | SSD (solid state drive) and SSD garbage collection method and device | |
| CN105718530B (en) | File storage system and file storage control method thereof | |
| US20140122774A1 (en) | Method for Managing Data of Solid State Storage with Data Attributes | |
| CN101981551B (en) | Apparatus and method for cache utilization | |
| CN110674056B (en) | Garbage recovery method and device | |
| CN103257935A (en) | Cache management method and application thereof | |
| CN103246609B (en) | The method of cold and hot data separation management in a kind of flash memory device and device | |
| CN107092563B (en) | Garbage recovery method and device | |
| CN101702139B (en) | Method and device for accessing data of Nand flash memory | |
| CN101645043A (en) | Methods for reading and writing data and memory device | |
| CN104424110A (en) | Active recovery of solid state drive | |
| CN108874574B (en) | Garbage recycling method and related device | |
| US20140258591A1 (en) | Data storage and retrieval in a hybrid drive | |
| CN110321081B (en) | Flash memory read caching method and system | |
| US11061598B2 (en) | Optimized handling of multiple copies in storage management | |
| CN109783019B (en) | Intelligent data storage management method and device | |
| CN108519858B (en) | Memory chip hardware hit method | |
| CN108491166B (en) | Read data cache management method for optical disk library | |
| CN111273868A (en) | Method for garbage recovery and write amplification reduction of full flash memory array | |
| CN113190180A (en) | Storage device based on mixed media and distributed storage system | |
| CN105138277A (en) | Cache management method for solid-state disc array | |
| CN110275678B (en) | STT-MRAM-based solid state memory device random access performance improvement method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200612 |