CN109800182A - It is a kind of to reduce the data storage handling method and its system for writing amplification - Google Patents
It is a kind of to reduce the data storage handling method and its system for writing amplification Download PDFInfo
- Publication number
- CN109800182A CN109800182A CN201910048977.3A CN201910048977A CN109800182A CN 109800182 A CN109800182 A CN 109800182A CN 201910048977 A CN201910048977 A CN 201910048977A CN 109800182 A CN109800182 A CN 109800182A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- compression
- processing
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention relates to data storage handling methods and its system that amplification is write in a kind of reduction；Wherein, processing method, comprising the following steps: S1, to the specific format of the file information and processing strategie initialized with it is synchronous；File is divided into different types according to file suffixes by S2, and is equipped with corresponding ID number；S3 carries out the transmission of order and data by PCIe interface and NVMe agreement, and parses to bit corresponding in order, obtains file type ID；S4, inquiry obtains compression algorithm type corresponding to the ID, and carries out data compression；S5 carries out address of cache and other common processing to compressed file, and writes in NAND memory bank.The present invention formulates corresponding Compression Strategies according to different types of data file, obtains optimal compression effectiveness, so that data volume of the storage into Flash is smaller than original data volume, amplification is write in reduction；To expand the utilization rate of Flash storage, the service life of solid state hard disk is improved.
Description
Technical field
The present invention relates to solid state hard disk design Storage technical fields, more specifically refer to a kind of data for reducing and writing amplification
Storage processing method and its system.
Background technique
In existing solid state hard disk, the data of input can be compressed, be restored again into NAND Flash memory bank later；But
Use degree of compression when different compression algorithms different due to same formatted file；The file of some formats can be effectively compressed,
And the file of some formats cannot compress；Such as: the formatted files such as Word, TXT, Excel, compression ratio are relatively high；But mp3
The video file of the formats such as formatted audio files, JPG/H264 but can not almost be compressed；If be not added differentiation to data into
Row compression processing then can not compress those or be difficult the data processing of compressed file, will result in temporal waste, together
When also will increase power consumption, therefore be unable to satisfy demand.
Summary of the invention
It is an object of the invention to overcome the deficiencies of existing technologies, a kind of data storage processing side for reducing and writing amplification is provided
Method and its system.
To achieve the above object, the present invention is used in lower technical solution:
It is a kind of to reduce the data storage handling method for writing amplification, comprising the following steps:
S1, to the specific format of the file information and processing strategie initialized with it is synchronous；
File is divided into different types according to file suffixes by S2, and is equipped with corresponding ID number；
S3 carries out the transmission of order and data by PCIe interface and NVMe agreement, and to bit corresponding in order
It is parsed, obtains file type ID；
S4, inquiry obtains compression algorithm type corresponding to the ID, and carries out data compression；
S5 carries out address of cache and other common processing to compressed file, and writes in NAND memory bank.
Its further technical solution are as follows: the S1 includes: handle or using by the format and strategy arranged in advance
Dynamic synchronization and self strategy matching are carried out in the process, formulates the Compression Strategies of file type ID and use, and form fixed pair
It should be related to.
Its further technical solution are as follows: the S2 further include: when issuing data write command, by text belonging to current file
Part type ID is bundled in NVMe order.
Its further technical solution are as follows: the S5 further include: compressed data carry out random number filling, to adapt to address
Suitable data package size needed for mapping.
Its further technical solution are as follows: after the S5, further includes: carry out read operation order, read command is converted into
The read operation of NAND identification reads data from NAND storage body.
Its further technical solution are as follows: after the S5, further includes: when processing read command, according in compression control information
Compression mark and compression algorithm ID obtain decompression algorithm type corresponding to the ID, processing is unziped it to current data,
And pass host side back.
It is a kind of to reduce the data storage processing system for writing amplification, comprising: initialization unit is divided into unit, and transmission parsing is single
Member inquires compression unit and map processing unit；
The initialization unit, for the specific format of the file information and processing strategie initialized with it is synchronous；
It is described to be divided into unit, for file to be divided into different types, and be equipped with corresponding ID number according to file suffixes；
The transmission resolution unit, for carrying out the transmission of order and data by PCIe interface and NVMe agreement, and it is right
Corresponding bit is parsed in order, obtains file type ID；
The inquiry compression unit obtains compression algorithm type corresponding to the ID for inquiring, and carries out data compression；
The map processing unit for carrying out address of cache and other common processing to compressed file, and is write
In NAND memory bank.
Its further technical solution are as follows: the initialization unit includes: to be handled by the format and strategy arranged in advance
Or dynamic synchronization and self strategy matching are carried out in use, formulate the Compression Strategies of file type ID and use, and shape
At fixed correspondence.
Its further technical solution are as follows: described to be divided into unit further include: when issuing data write command, by current file institute
The file type ID of category is bundled in NVMe order.
Its further technical solution are as follows: further include: read operation unit and decompression processing unit；
Read command is converted into the read operation of NAND identification for carrying out read operation order by the read operation unit, from
Data are read in NAND storage body；
The decompression processing unit, when for handling read command, according to the compression mark and pressure in compression control information
Compression algorithm ID obtains decompression algorithm type corresponding to the ID, unzips it processing to current data, and pass host side back.
Compared with the prior art, the invention has the advantages that: according to different types of data file, formulate corresponding pressure
Contracting strategy, obtains optimal compression effectiveness, so that the data volume in storage Flash is smaller than original data volume, amplification is write in reduction；From
And expand the utilization rate of Flash storage, improve the service life of solid state hard disk；The initial address of data file is also utilized simultaneously
And end address information, the write-in block of reasonable arrangement data, to a little remaining space, using the measure of filling random number, thus
Software is reduced to the processing difficulty of data address of cache, it being capable of preferably meet demand.
The invention will be further described in the following with reference to the drawings and specific embodiments.
Detailed description of the invention
Fig. 1, which is that the present invention is a kind of, reduces the flow chart for writing the data storage handling method of amplification；
Fig. 2, which is that the present invention is a kind of, reduces the block diagram for writing the data storage processing system of amplification.
10 initialization units 20 are divided into unit
30 transmission resolution units 40 inquire compression unit
50 map processing unit, 60 read operation unit
70 decompression processing units
Specific embodiment
In order to more fully understand technology contents of the invention, combined with specific embodiments below to technical solution of the present invention into
One step introduction and explanation, but not limited to this.
Such as Fig. 1 to specific embodiment shown in Fig. 2, wherein as shown in Figure 1, the invention discloses a kind of reductions to write amplification
Data storage handling method, comprising the following steps:
S1, to the specific format of the file information and processing strategie initialized with it is synchronous；
File is divided into different types according to file suffixes by S2, and is equipped with corresponding ID number；
S3 carries out the transmission of order and data by PCIe interface and NVMe agreement, and to bit corresponding in order
It is parsed, obtains file type ID；
S4, inquiry obtains compression algorithm type corresponding to the ID, and carries out data compression；
S5 carries out address of cache and other common processing to compressed file, and writes in NAND memory bank.
Wherein, the S1 includes: to carry out processing by the format and strategy arranged in advance or carry out dynamic in use
Synchronous and self strategy matching, formulates the Compression Strategies of file type ID and use, and forms fixed correspondence.
Further, after solid state hard disk firmware design is good, the card initialization stage is being opened, is being carried out according to different user demands
Policy synchronization meets more demands with the enlargement of application environment.
Further, solid state hard disk in use, carries out dynamic synchronization and self strategy matching, expands utilization
Scene.
Wherein, the S2 further include: when issuing data write command, file type ID belonging to current file is bundled to
In NVMe order.
Wherein, different compression algorithms is different the compression effectiveness of files in different types；It is being aware of rising for file
Beginning address and end address, so that it may more effectively handle these data.Such as: these data are write one relatively continuously
In address, to reduce the probability of garbage reclamation, amplification is write in reduction.
Wherein, host side knows the file type of data, initial address and end address to data file
Know.If solid state hard disk similarly recognizes these information of data file, so that it may targetedly be handled:
Compression is not still compressed, and uses which kind of most effective Compression Strategies.
Wherein, PCIe solid state hard disk at present carries out order and data transmission using NVMe agreement；And current NVMe read-write
It is available free reserved bit in command format, the data file class and data file initial address message (IAM) of host side can be with
Solid state hard disk is passed to by these reserved bits, is handled by controller chip and its firmware.
Wherein, the data type of host side is classified, and is equipped with different ID numbers, each order carries files classes
The identification informations such as type ID and file state pause judgments.Solid state hard disk also includes representated by file type ID allocation table and each ID
File type use compression algorithm.Certainly, the file type without being compressed is also contained in allocation table and its strategy.
Different lossless compression Processing Algorithms is used to the data type of different ID.And specified audio-video format is no longer pressed
Contracting, such as MP3 audio file, the formatted files such as MP4/H264/H265 directly carry out data storage without compression.
Further, to the new type file that do not arrange, the adaptation of different compression algorithms is carried out, optimal compression is found out
Algorithm notifies firmware；And this file type and matched compression algorithm are formed into a fixed strategy, it is saved in data compression
In policy unit, when subsequent same type of data file operation, corresponding strategy is directly used.
Wherein, the S5 further include: compressed data carry out random number filling, suitable needed for address of cache to adapt to
Data package size；Utilize the initial address and end address information of data file, the write-in block of reasonable arrangement data, to a little
Remaining space, using the measure of filling random number, to reduce software to the processing difficulty of data address of cache.
Wherein, after the S5, further includes: carry out read operation order, according to file suffixes, control strategy and file type
File is divided into different types by ID allocation table, and is equipped with corresponding ID number, when issuing data read command, by current file
Affiliated file type ID is bundled to together in NVMe order, and the biography of order and data is carried out by PCIe interface and NVMe agreement
It is defeated；When solid state hard disk is when receiving NVMe order, bit corresponding in order is parsed, obtains file type ID,
Read operation order and file type ID are respectively transmitted, and read command is converted into the read operation of NAND identification, stores up body from NAND
Middle reading data；When write operation, address of cache processing unit can generate compression control information, protect together with address mapping information
It deposits.Compression control information includes compression mark and compression algorithm ID, and whether instruction current data packet compresses and compression algorithm used
ID number；When read operation, address of cache processing unit parses the compression information in address mapping information, determines current
Whether data packet, which needs, decompresses, and decompresses required compression algorithm.
Further, after the S5, further includes: when processing read command, according to the compression mark in compression control information
And compression algorithm ID obtains decompression algorithm type corresponding to the ID, unzips it processing to current data, and pass host back
End.
It wherein, include data compression flag in compression control information, whether instruction current file is compressed；Meanwhile
Also the ID number comprising compression algorithm used in current file.When data are read, according to the compression mark in compression control information
And compression algorithm ID, the data read from Flash are handled, decompresses or does not decompress.
As shown in Fig. 2, the invention also discloses the data storage processing systems that amplification is write in a kind of reduction, comprising: initialization
Unit 10 is divided into unit 20, transmits resolution unit 30, inquires compression unit 40 and map processing unit 50；
The initialization unit 10, for the specific format of the file information and processing strategie initialized with it is synchronous；
It is described to be divided into unit 20, for file to be divided into different types, and be equipped with corresponding ID according to file suffixes
Number；
The transmission resolution unit 30, for carrying out the transmission of order and data by PCIe interface and NVMe agreement, and
Bit corresponding in order is parsed, file type ID is obtained；
The inquiry compression unit 40 obtains compression algorithm type corresponding to the ID for inquiring, and carries out data pressure
Contracting；
The map processing unit 50 for carrying out address of cache and other common processing to compressed file, and is write
Into NAND memory bank.
Wherein, the initialization unit 10 includes: handle or in use process by the format and strategy arranged in advance
Middle progress dynamic synchronization and self strategy matching, formulate the Compression Strategies of file type ID and use, and form fixed corresponding pass
System.
Wherein, described to be divided into unit 20 further include: when issuing data write command, by file type belonging to current file
ID is bundled in NVMe order.
Wherein, the data storage processing system for writing amplification is reduced further include: read operation unit 60 and decompression processing unit
70；
Read command is converted into the read operation of NAND identification for carrying out read operation order by the read operation unit 60, from
Data are read in NAND storage body；
The decompression processing unit 70, when for handling read command, according in compression control information compression mark and
Compression algorithm ID obtains decompression algorithm type corresponding to the ID, unzips it processing to current data, and pass host side back.
The information such as the type of data file and file initial address are passed to solid state hard disk, solid-state by host by the present invention
Hard disk utilizes these information, formulates Compression Strategies, different compression algorithms is used to different types of data file, to improve number
According to the compression ratio and compression efficiency of compression, so that reduce data writes amplification, while chip power-consumption is reduced；The same text will be belonged to
The data of part are write as far as possible in a relatively continuous address, to reduce the probability of garbage reclamation, write amplification to reduce.
The present invention formulates Compression Strategies, obtains optimal compression effectiveness according to different types of data file, so that storage
Data volume in Flash is smaller than original data volume, and amplification is write in reduction；To expand the utilization rate of Flash storage, solid-state is improved
The service life of hard disk；Utilize the initial address and end address information of data file, the write-in block of reasonable arrangement data.To few
Perhaps remaining space, using the measure of filling random number, to reduce software to the processing difficulty of data address of cache；Different numbers
It can achieve more excellent effect using different lossless compression algorithms according to type.
It is above-mentioned that technology contents of the invention are only further illustrated with embodiment, in order to which reader is easier to understand, but not
It represents embodiments of the present invention and is only limitted to this, any technology done according to the present invention extends or recreation, by of the invention
Protection.Protection scope of the present invention is subject to claims.
Claims (10)
1. a kind of reduce the data storage handling method for writing amplification, which comprises the following steps:
S1, to the specific format of the file information and processing strategie initialized with it is synchronous；
File is divided into different types according to file suffixes by S2, and is equipped with corresponding ID number；
S3 is carried out the transmission of order and data by PCIe interface and NVMe agreement, and carried out to bit corresponding in order
Parsing obtains file type ID；
S4, inquiry obtains compression algorithm type corresponding to the ID, and carries out data compression；
S5 carries out address of cache and other common processing to compressed file, and writes in NAND memory bank.
2. the data storage handling method that amplification is write in a kind of reduction according to claim 1, which is characterized in that the S1 packet
It includes: carrying out processing by the format and strategy arranged in advance or carry out dynamic synchronization and self strategy matching, system in use
Determine the Compression Strategies of file type ID and use, and forms fixed correspondence.
3. the data storage handling method that amplification is write in a kind of reduction according to claim 1, which is characterized in that the S2 is also
It include: that file type ID belonging to current file is bundled in NVMe order when issuing data write command.
4. the data storage handling method that amplification is write in a kind of reduction according to claim 1, which is characterized in that the S5 is also
It include: that compressed data carry out random number filling, to adapt to suitable data package size needed for address of cache.
5. a kind of reduce according to claim 1 writes the data storage handling method of amplification, which is characterized in that the S5 it
Afterwards, further includes: carry out read operation order, read command is converted into the read operation of NAND identification, read data from NAND storage body.
6. a kind of reduce according to claim 5 writes the data storage handling method of amplification, which is characterized in that the S5 it
Afterwards, further includes: processing read command when, according in compression control information compression mark and compression algorithm ID obtain corresponding to the ID
Decompression algorithm type, processing is unziped it to current data, and pass host side back.
7. a kind of reduce the data storage processing system for writing amplification characterized by comprising initialization unit is divided into unit, passes
Defeated resolution unit inquires compression unit and map processing unit；
The initialization unit, for the specific format of the file information and processing strategie initialized with it is synchronous；
It is described to be divided into unit, for file to be divided into different types, and be equipped with corresponding ID number according to file suffixes；
The transmission resolution unit, for carrying out the transmission of order and data by PCIe interface and NVMe agreement, and to order
In corresponding bit parsed, obtain file type ID；
The inquiry compression unit obtains compression algorithm type corresponding to the ID for inquiring, and carries out data compression；
The map processing unit for carrying out address of cache and other common processing to compressed file, and writes NAND
In memory bank.
8. the data storage processing system that amplification is write in a kind of reduction according to claim 7, which is characterized in that described initial
Changing unit includes: to carry out processing by the format and strategy arranged in advance or carry out dynamic synchronization and self strategy in use
Matching, formulates the Compression Strategies of file type ID and use, and forms fixed correspondence.
9. the data storage processing system that amplification is write in a kind of reduction according to claim 7, which is characterized in that described to be divided into
Unit further include: when issuing data write command, file type ID belonging to current file is bundled in NVMe order.
10. the data storage processing system that amplification is write in a kind of reduction according to claim 7, which is characterized in that further include:
Read operation unit and decompression processing unit；
Read command is converted into the read operation of NAND identification for carrying out read operation order by the read operation unit, is stored up from NAND
Data are read in body；
The decompression processing unit when for handling read command, according to the compression mark in compression control information and compresses calculation
Method ID obtains decompression algorithm type corresponding to the ID, unzips it processing to current data, and pass host side back.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910048977.3A CN109800182A (en) | 2019-01-18 | 2019-01-18 | It is a kind of to reduce the data storage handling method and its system for writing amplification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910048977.3A CN109800182A (en) | 2019-01-18 | 2019-01-18 | It is a kind of to reduce the data storage handling method and its system for writing amplification |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109800182A true CN109800182A (en) | 2019-05-24 |
Family
ID=66559669
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910048977.3A Pending CN109800182A (en) | 2019-01-18 | 2019-01-18 | It is a kind of to reduce the data storage handling method and its system for writing amplification |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109800182A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949621A (en) * | 2020-07-22 | 2020-11-17 | 金钱猫科技股份有限公司 | Scene switching-based file compression storage method and terminal |
| CN112100143A (en) * | 2020-09-25 | 2020-12-18 | 平安科技（深圳）有限公司 | File compression storage method, device, equipment and storage medium |
| CN113641434A (en) * | 2021-08-12 | 2021-11-12 | 上海酷栈科技有限公司 | Cloud desktop data compression self-adaptive encoding method and system and storage device |
| CN114666406A (en) * | 2022-02-24 | 2022-06-24 | 国电南瑞科技股份有限公司 | Object model-based power internet of things data compression method and device |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1851691A (en) * | 2005-04-22 | 2006-10-25 | 北京九州软件有限公司 | Database back-up data compression and search method |
| CN101957836A (en) * | 2010-09-03 | 2011-01-26 | 清华大学 | Configurable real-time transparent compressing method in file system |
| CN102893580A (en) * | 2012-07-04 | 2013-01-23 | 华为技术有限公司 | Anti-virus method and device and firewall device |
| CN103020157A (en) * | 2012-11-23 | 2013-04-03 | 山东电力集团公司 | High-reliability real-time file generation method spanning physical isolation |
| US20140218220A1 (en) * | 1998-12-11 | 2014-08-07 | Realtime Data, Llc | Data compression systems and methods |
| US20140236908A1 (en) * | 2013-02-20 | 2014-08-21 | Verizon Patent And Licensing Inc. | Method and apparatus for providing enhanced data retrieval with improved response time |
-
2019
- 2019-01-18 CN CN201910048977.3A patent/CN109800182A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140218220A1 (en) * | 1998-12-11 | 2014-08-07 | Realtime Data, Llc | Data compression systems and methods |
| CN1851691A (en) * | 2005-04-22 | 2006-10-25 | 北京九州软件有限公司 | Database back-up data compression and search method |
| CN101957836A (en) * | 2010-09-03 | 2011-01-26 | 清华大学 | Configurable real-time transparent compressing method in file system |
| CN102893580A (en) * | 2012-07-04 | 2013-01-23 | 华为技术有限公司 | Anti-virus method and device and firewall device |
| CN103020157A (en) * | 2012-11-23 | 2013-04-03 | 山东电力集团公司 | High-reliability real-time file generation method spanning physical isolation |
| US20140236908A1 (en) * | 2013-02-20 | 2014-08-21 | Verizon Patent And Licensing Inc. | Method and apparatus for providing enhanced data retrieval with improved response time |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111949621A (en) * | 2020-07-22 | 2020-11-17 | 金钱猫科技股份有限公司 | Scene switching-based file compression storage method and terminal |
| CN111949621B (en) * | 2020-07-22 | 2023-12-29 | 金钱猫科技股份有限公司 | File compression storage method and terminal based on scene switching |
| CN112100143A (en) * | 2020-09-25 | 2020-12-18 | 平安科技（深圳）有限公司 | File compression storage method, device, equipment and storage medium |
| CN112100143B (en) * | 2020-09-25 | 2023-03-21 | 平安科技（深圳）有限公司 | File compression storage method, device, equipment and storage medium |
| CN113641434A (en) * | 2021-08-12 | 2021-11-12 | 上海酷栈科技有限公司 | Cloud desktop data compression self-adaptive encoding method and system and storage device |
| CN114666406A (en) * | 2022-02-24 | 2022-06-24 | 国电南瑞科技股份有限公司 | Object model-based power internet of things data compression method and device |
| CN114666406B (en) * | 2022-02-24 | 2023-11-21 | 国电南瑞科技股份有限公司 | Electric power Internet of things data compression method and device based on object model |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109800182A (en) | It is a kind of to reduce the data storage handling method and its system for writing amplification | |
| US8782018B2 (en) | Storage device and data processing device utilizing determined dictionary compression | |
| US6490649B2 (en) | Memory device | |
| CN102063267B (en) | Data storage system comprising a mapping bridge for aligning host block size with physical block size of a data storage device | |
| CN103136109B (en) | A kind of solid-state memory system FTL write with compression function and read method | |
| US10042576B2 (en) | Method and apparatus for compressing addresses | |
| JP2004362530A (en) | Storage device with best compression management mechanism | |
| CN113873255B (en) | Video data transmission method, video data decoding method and related devices | |
| JP2017527877A (en) | Method and apparatus for reading / writing data from / to flash memory and user equipment | |
| EP3574411A1 (en) | Bus encoding using on-chip memory | |
| KR20040107343A (en) | Storage device for improving transmission speed | |
| US9430327B2 (en) | Data access method, memory control circuit unit and memory storage apparatus | |
| CN105723320A (en) | Data arrangement method, storage apparatus, storage controller and storage array | |
| WO1997043764A1 (en) | Memory device | |
| CN114466196B (en) | Video data processing method, system, device and computer readable storage medium | |
| WO2020135411A1 (en) | Data backup and recovery method for nvdimm, nvdimm controller and nvdimm | |
| CN102591737B (en) | Data writing and reading method, memory controller and memory storage device | |
| CN1979475A (en) | Compressed file processing method | |
| CN114003169B (en) | Data compression method for SSD | |
| CN110633225A (en) | Apparatus and method for generating entity storage comparison table | |
| US11693820B2 (en) | Cooperative access method, system, and architecture of external storage | |
| CN107122312A (en) | A kind of solid-state disk address mapping method | |
| JP2002312250A (en) | Transmission system, device and method | |
| CN103870779B (en) | A kind of card reader with data compression function | |
| CN209803776U (en) | NVDIMM controller and NVDIMM |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20190524 |