CN109245772A - A kind of method, system, device and readable storage medium storing program for executing compressing I/O data - Google Patents
A kind of method, system, device and readable storage medium storing program for executing compressing I/O data Download PDFInfo
- Publication number
- CN109245772A CN109245772A CN201810935726.2A CN201810935726A CN109245772A CN 109245772 A CN109245772 A CN 109245772A CN 201810935726 A CN201810935726 A CN 201810935726A CN 109245772 A CN109245772 A CN 109245772A
- Authority
- CN
- China
- Prior art keywords
- data
- sub
- available thread
- complete
- compression
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 56
- 238000003860 storage Methods 0.000 title claims abstract description 16
- 238000013144 data compression Methods 0.000 claims abstract description 79
- 238000007906 compression Methods 0.000 claims description 45
- 230000006835 compression Effects 0.000 claims description 41
- 230000008569 process Effects 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000012216 screening Methods 0.000 claims description 8
- 238000001514 detection method Methods 0.000 claims description 4
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 230000006837 decompression Effects 0.000 description 12
- 239000007787 solid Substances 0.000 description 6
- 230000005540 biological transmission Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005194 fractionation Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000001680 brushing effect Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000000151 deposition Methods 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 230000014759 maintenance of location Effects 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000008520 organization Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 230000001737 promoting effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
-
- H—ELECTRICITY
- H03—ELECTRONIC CIRCUITRY
- H03M—CODING; DECODING; CODE CONVERSION IN GENERAL
- H03M7/00—Conversion of a code where information is represented by a given sequence or number of digits to a code where the same, similar or subset of information is represented by a different sequence or number of digits
- H03M7/30—Compression; Expansion; Suppression of unnecessary data, e.g. redundancy reduction
- H03M7/60—General implementation details not specific to a particular type of compression
- H03M7/6017—Methods or arrangements to increase the throughput
- H03M7/6023—Parallelization
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Compression, Expansion, Code Conversion, And Decoders (AREA)
Abstract
This application discloses a kind of methods for compressing I/O data, comprising: receives the complete I/O data issued；The complete I/O data are split as to the sub- I/O data of quantity identical as available thread number；It controls each available thread and data compression step is executed to corresponding sub- I/O data respectively.This method is by being split as multiple sub- I/O data for original complete I/O data, and data compression is realized by different available threads in a parallel fashion respectively, original single thread, serial data compression scheme are changed into multithreading provided by the present application, parallel data compression method, can be obviously improved that data compression performance, to shorten data compression time-consuming.The application further simultaneously discloses a kind of system, device and computer readable storage medium for compressing I/O data, has above-mentioned beneficial effect.
Description
Technical field
This application involves technical field of data compression, in particular to a kind of method for compressing I/O data, system, device and
Computer readable storage medium.
Background technique
Under big data, the trend of cloud computing, the importance of data storage is increasingly prominent.And due to data transmission bauds
It is promoted, restricts data-storage system handling capacity and IOPS (Input/Output Operations Per Second, progress per second
Read and write (I/O) operation number, for measuring the performance of random access) performance bottleneck become from data transmission media
Data rate memory, therefore, the mechanical hard disk slower compared to traditional read or write speed makes (Flash core using full flash chip
Piece) solid state hard disk gradually start to show up prominently.
It is different from mechanical hard disk, although solid state hard disk possesses faster read or write speed, but its service life is by the erasable longevity
The limitation of life, therefore in order to extend the service life of solid state hard disk as far as possible, industry is now mainly by reducing the number read and write
Erasable number is reduced according to the mode of amount, an important mode for reducing read-write data volume is data compression, such as will be original
The erasable number of half is just enough reduced when the compressing file that file size is 10GB is 5GB.
But present-day data compression is time-consuming still partially long, is that single thread is still used to execute data compression step because of it,
Or data manipulation (the core cpu of rotation execution data manipulation, so that the load of each core cpu is equal is executed using pseudo- multithreading
Weighing apparatus), therefore, how to overcome items technological deficiency existing for existing data compression scheme, provides a kind of data compression time-consuming more
Short, the higher data compression scheme of compression performance is those skilled in the art's urgent problem to be solved.
Summary of the invention
The purpose of the application is to provide a kind of method for compressing I/O data, by being split as original complete I/O data
Multiple sub- I/O data, and data compression is realized by different available threads in a parallel fashion respectively, by original single thread,
Serial data compression scheme changes into multithreading provided by the present application, parallel data compression method, can be obviously improved number
According to compression performance, it is time-consuming to shorten data compression.
The another object of the application is the provision of a kind of system, device and computer-readable storage for compressing I/O data
Medium.
To achieve the above object, the application provides a kind of method for compressing I/O data, this method comprises:
Receive the complete I/O data issued；
The complete I/O data are split as to the sub- I/O data of quantity identical as available thread number；
It controls each available thread and data compression step is executed to corresponding sub- I/O data respectively.
Optionally, the method for compression I/O data further include:
Detect the depth of the I/O data queue of each available thread；Wherein, it is arranged in the I/O data queue
The I/O data to be processed of preset quantity；
The available thread that the depth is lower than default queue depth is screened, preferred available thread is obtained；
It is corresponding, the complete I/O data are split as to the sub- I/O data of quantity identical as available thread number, specifically:
The complete I/O data are split as to the sub- I/O data of quantity identical as the preferred available thread number；
It is corresponding, it controls each available thread and data compression step is executed to corresponding sub- I/O data respectively, specific:
It controls each preferred available thread and data compression step is executed to corresponding sub- I/O data.
Optionally, the method for compression I/O data further include:
It is when the complete I/O data are split as multiple sub- I/O data, the complete I/O data are corresponding
Complete metadata is also split as the sub- metadata of quantity identical as the sub- I/O data；
After the data compression step of each available thread is performed both by, according to each sub- metadata again group
Each sub- I/O data of compression are closed to obtain completely compressing I/O data；Wherein, the sub- I/O data of compression are the sub- I/O data warp
The I/O data obtained after the data compression step.
Optionally, it controls each available thread and data compression step, packet is executed to corresponding sub- I/O data respectively
It includes:
It controls each available thread and is utilized respectively QAT data compression technique to corresponding sub- I/O data execution data
Squeeze operation.
To achieve the above object, present invention also provides a kind of system for compressing I/O data, which includes:
Complete I/O data receipt unit, for receiving the complete I/O data issued；
By available thread number split cells, for the complete I/O data to be split as quantity identical as available thread number
Sub- I/O data；
Parallelly compressed unit executes data compression behaviour to corresponding sub- I/O data respectively for controlling each available thread
Make.
Optionally, the system of compression I/O data further include:
Queue depth's detection unit, the depth of the I/O data queue for detecting each available thread；Wherein, institute
State the I/O data to be processed that preset quantity is arranged in I/O data queue；
Preference thread screening unit, the available thread for being lower than default queue depth for screening the depth, obtains preferably
Available thread；
It is corresponding, it is described to press available thread number split cells specifically:
The complete I/O data are split as to the sub- I/O data of quantity identical as the preferred available thread number；
Corresponding, the parallelly compressed unit is specific:
It controls each preferred available thread and data compression step is executed to corresponding sub- I/O data.
Optionally, the system of compression I/O data further include:
Metadata while deconsolidation process unit, for the complete I/O data to be split as multiple sub- I/O data
When, the corresponding complete metadata of the complete I/O data is also split as to the first number of son of quantity identical as the sub- I/O data
According to；
Recomposition unit, for after the data compression step of each available thread is performed both by, according to each described
Sub- metadata reconfigures each sub- I/O data of compression and obtains completely compressing I/O data；Wherein, the sub- I/O data of compression are institute
State the I/O data that sub- I/O data obtain after the data compression step.
Optionally, described to include: by available thread number split cells
QAT compresses subelement, is utilized respectively QAT data compression technique to corresponding for controlling each available thread
Sub- I/O data execute data compression step.
To achieve the above object, present invention also provides a kind of device for compressing I/O data, which includes:
Memory, for storing computer program；
Processor realizes the side that I/O data are compressed as described in above content when for executing the computer program
The step of method.
To achieve the above object, described computer-readable to deposit present invention also provides a kind of computer readable storage medium
It is stored with computer program on storage media, the pressure as described in above content is realized when the computer program is executed by processor
The step of method of contracting I/O data.
Obviously, the method for a kind of compression I/O data provided herein, by splitting original complete I/O data
For multiple sub- I/O data, and data compression is realized by different available threads in a parallel fashion respectively, by original single line
Journey, serial data compression scheme change into multithreading provided by the present application, parallel data compression method, can be obviously improved
It is time-consuming to shorten data compression for data compression performance.The application additionally provide simultaneously it is a kind of compress the systems of I/O data, device and
Computer readable storage medium has above-mentioned beneficial effect, and details are not described herein.
Detailed description of the invention
In order to illustrate the technical solutions in the embodiments of the present application or in the prior art more clearly, to embodiment or will show below
There is attached drawing needed in technical description to be briefly described, it should be apparent that, the accompanying drawings in the following description is only this
The embodiment of application for those of ordinary skill in the art without creative efforts, can also basis
The attached drawing of offer obtains other attached drawings.
Fig. 1 is a kind of flow chart of method for compressing I/O data provided by the embodiments of the present application；
Fig. 2 is the flow chart of the method for another compression I/O data provided by the embodiments of the present application；
Fig. 3 is the logical flow chart of the method for another compression I/O data provided by the embodiments of the present application；
Fig. 4 is a kind of structural block diagram of system for compressing I/O data provided by the embodiments of the present application.
Specific embodiment
The core of the application is to provide a kind of method, system, device and computer-readable storage medium for compressing I/O data
Matter, by the way that original complete I/O data are split as multiple sub- I/O data, and respectively by different available threads with parallel
Mode realizes data compression, and original single thread, serial data compression scheme are changed into multithreading provided by the present application, simultaneously
Capable data compression method can be obviously improved data compression performance, and it is time-consuming to shorten data compression.
To keep the purposes, technical schemes and advantages of the embodiment of the present application clearer, below in conjunction with the embodiment of the present application
In attached drawing, the technical scheme in the embodiment of the application is clearly and completely described, it is clear that described embodiment is
Some embodiments of the present application, instead of all the embodiments.Based on the embodiment in the application, those of ordinary skill in the art
All other embodiment obtained without making creative work, shall fall in the protection scope of this application.
Embodiment one
Below in conjunction with Fig. 1, Fig. 1 is a kind of flow chart of method for compressing I/O data provided by the embodiments of the present application, tool
Body the following steps are included:
S101: the complete I/O data issued are received；
This step is intended to receive the complete I/O data that issue, and the source of these complete I/O data is varied, can be with
For the dump I/O data from other equipment, data file can be obtained to download under internet address, can be from depositing
Local original I/O data are stored in, herein and are not specifically limited, purpose is only that obtain a untreated complete I/
O data so that subsequent step is compressed, and I/O data after compressed, small volume compression is stored, with logical
The mode for crossing reduction size of data reduces the erasable number of solid state hard disk, and deflecting extends the service life of solid state hard disk.
S102: complete I/O data are split as to the sub- I/O data of quantity identical as available thread number；
On the basis of S101, this step is intended to for complete I/O data being split as the son of quantity identical as available thread number
I/O data.Wherein, which refers to the quantity of available thread, such as is currently being widely used Intel CPU,
Maximum thread can be twice of CPU core calculation, such as 4 core, 8 thread, thread are a single sequential control flows in program
Journey.There is a relatively independent, schedulable execution unit in process, is the basic unit that CPU was independently dispatched and assigned to system
The thread of program when instruction operation, running multiple threads completes different work simultaneously in single program, referred to as more
Thread.
According to present load number, the creation of the property of can choose, deleting thread, (new thread can be by Geju City thread
Creation obtains), therefore before the execution of this step, it is also necessary to the available thread number under current system environment is determined, due to each
Cross-thread can be performed simultaneously different tasks, from principle by being no doubt that the Thread Count that uses simultaneously is finally completed data
The time-consuming of compression is shorter.
Illustrate, it is assumed that currently available Thread Count is 5, then can be classified as under the guidance of this step 5 parts it is different
Sub- I/O data, and this 5 parts different sub- I/O data are handed down to different available threads respectively, with it is parallel simultaneously make this 5
The sub- I/O data that a thread is received by simultaneously execute data compression step.
Further, it is contemplated that before the complete I/O data that processing newly receives, system load is higher, some lines
Journey has stored the queue of I/O data to be processed very long (it is longer to can be described as depth when measuring with queue depth), if will one
One's share of expenses for a joint undertaking I/O data distributing may tie down the compression progress of entire I/O data to the thread for such situation occur, because one
Whether complete I/O data, which are compressed, is completed to be to see whether compression is completed for all sub- I/O data that it splits out, therefore such
Situation can also filter out a part of preferred thread according to queue length (depth), and be carried out according to the quantity of preference thread
The fractionation of quantity, to optimize entire compression process.
Alternatively, can be carried out by the way of dividing equally during complete I/O data are split as each sub- I/O data
It splits, this mode is suitble to the situation unanimous on the whole in each available thread present load (queue length or queue depth), can
Realize load balancing to a certain extent, can also using it is non-be by the way of split, this mode is suitble to can be used each
The case where thread present load differs greatly also is used to realize load balancing, can flexibly select according to the actual situation.
S103: it controls each available thread and data compression step is executed to corresponding sub- I/O data respectively.
On the basis of S102, this step is intended to control each available thread and is connected to the sub- I/O data that body receives respectively
Data compression is executed, to accelerate data procedures by parallel form, increases the efficiency of data compression and shortens data compression consumption
When.
It is completed in the above process using the processor of Intel, the QAT technology that can be provided based on Intel completes this
Data compression step: it controls each available thread and is utilized respectively QAT data compression technique to corresponding sub- I/O data execution data
Squeeze operation.
QAT,QuickAssist Technology is a kind of accelerator art that Intel is provided.
Further, after needing each fractionation and in the case that the sub- I/O data through overcompression are recombinated again, may be used also
When complete I/O data are split as multiple sub- I/O data, the corresponding complete metadata of complete I/O data to be also split as
The sub- metadata of quantity identical as sub- I/O data；With after the data compression step of each available thread is performed both by, according to
Each sub- metadata reconfigures each sub- I/O data of compression and obtains completely compressing I/O data.
Wherein, compressing sub- I/O data is the I/O data that sub- I/O data obtain after data compression step.
Further, foregoing present a kind of modes of data compression, further include corresponding decompression side
Formula, since decompression process is consistent with compression process principle, step is on the contrary, art technology can be based on the compressed of above-mentioned offer
Journey step is readily attained corresponding decompression process, this does not need to pay creative effort, does not also repeat one by one herein.
Based on the above-mentioned technical proposal, a kind of method for compressing I/O data provided by the embodiments of the present application, by will be original
Complete I/O data are split as multiple sub- I/O data, and realize data pressure in a parallel fashion by different available threads respectively
Original single thread, serial data compression scheme are changed into multithreading provided by the present application, parallel data compression side by contracting
Method can be obviously improved data compression performance, and it is time-consuming to shorten data compression.
Embodiment two
Below in conjunction with Fig. 2, Fig. 2 is the flow chart of the method for another compression I/O data provided by the embodiments of the present application, this
Embodiment on the basis of example 1, is split as each sub- I/O data by the way of carrying out complete I/O data respectively,
On the basis of this, screened to obtain some load similar in lower, situation preferably by the way of based on each thread queue depth
Available thread, specific implementation step are as follows:
S201: the complete I/O data issued are received；
S202: detecting the depth of the I/O data queue of each available thread, and screening depth can lower than default queue depth
With thread, preferred available thread is obtained；
It is different from embodiment one, embodiment adds S202 come when often receiving the complete I/O data issued,
Carry out the depth detection of primary each available thread I/O data queue, wherein be stored with quantity in the I/O data queue and be no more than
Queue accommodates the I/O data to be processed of upper limit number, and screens to obtain actual arrays depth lower than this based on preset queue depth
The preferred available thread of default queue depth, be based in the next steps the quantity of the preferred available thread split and
Data compression step is executed respectively.
S203: complete I/O data are split as to the sub- I/O data of quantity identical as preferred available thread number；
S204: each preferably available thread of control executes data compression step to corresponding sub- I/O data.
It should be noted that being different from the non-respectively mode divided equally in the another kind that embodiment one refers to, can also be based on should
The mode that embodiment two provides carries out, naturally it is also possible to be directly based upon and carry out without the mode of screening, because non-divide equally mode
The case where being originally adapted to different loads.It can flexible choice according to the actual situation.
Embodiment three
On the basis of the various embodiments described above, the present embodiment will provide a kind of for QAT compression in conjunction with specifically used scene
The I/O data that equipment carries out data compression split mode, to realize the mesh for promoting QAT compression device overall compression performance and being promoted
's.
When receiving the I/O data issued, according to the queue depth inside current multithreading, the respectively I/O data,
The data prediction that I/O data are completed inside multithreading works (metadata etc.), and I/O data organization at QAT compression device
The form needed is sent to QAT compression device and carries out data compression.Finally current hierarchy compression together with compressed data
Under brush in SSD, decompression operation use same multithread scheduling method.
The compiling work of drive module is completed in the QAT driving provided using Intel, and generating can be in general processor operation
Host side program, and complete the installation of the program, then increase corresponding function code within the storage system and create multithreading
Carrying out the pretreatment works such as metadata to I/O, (multithreading quantity can be according to cpu performance, nucleus number, compressing card performance and actual measurement number
According to comprehensively considering).When there is I/O to issue, IO is divided equally according to the IO depth inside each thread.Multithreading completes data prediction
Afterwards, it calls the api function of QAT driving to complete compression, and completes the writing disk manipulation of compressed data inside thread.Master when decompression
Thread reads compressed data from storage system, assigns to decompression thread, the data processing work after completing data decompression.
Step just will be implemented to compression process and decompression procedure respectively below to be described, compression process can join simultaneously
See Fig. 3:
1. creation multithreading and each thread are respectively intended to store the circle queue of untreated I/O data；
2. dividing equally I/O request according to the queue depth inside current each thread when new I/O data arrive；
3. completing the work of the data predictions such as metadata inside thread；
4. the API of QAT driving is called to execute squeeze operation；
5.QAT compression device is moved using DMA (Direct Memory Access direct memory access) from designated position
Data to be compressed are cached to equipment, returned data after the completion of compression；
6. respectively compression thread is the designated position for brushing SSD under compressed data.
IO buffer in Fig. 3 refers to I/O buffer, and tail is for identifying queue tail, and head is for identifying queue
Head, tail-head are queue length, and maxlen is to preset maximum queue length, such use (tail-head+maxlen)/
The ratio of maxlen measures whether queue depth is excessive, and selects the queue that ratio is less than default ratio, QAT
Compression refers to QAT compression device.
Decompression process mainly includes following steps:
(1) data to be decompressed are read from SSD；
(2) decompression thread is assigned into data decompression request；
(3) it calls the API of QAT driving to realize decompression operation, data is returned.
The present embodiment is the number of threads by increasing data compression, is reduced from I/O data distributing to QAT compression device
Access delay, plays the maximum performance of QAT compression device as far as possible, full flash memory products (is all made of flash chip
Solid state hard disk) for, higher handling capacity and IOPS still are able in data compression.
Because situation is complicated, it can not enumerate and be illustrated, those skilled in the art should be able to recognize according to the application
The basic skills principle combination actual conditions of offer may exist many examples, in the case where not paying enough creative works,
It should within the scope of protection of this application.
Fig. 4 is referred to below, Fig. 4 is a kind of structural block diagram of system for compressing I/O data provided by the embodiments of the present application,
The system may include:
Complete I/O data receipt unit 100, for receiving the complete I/O data issued；
By available thread number split cells 200, for complete I/O data to be split as quantity identical as available thread number
Sub- I/O data；
Parallelly compressed unit 300 executes data compression to corresponding sub- I/O data respectively for controlling each available thread
Operation.
Further, the system of compression I/O data further include:
Queue depth's detection unit, the depth of the I/O data queue for detecting each available thread；Wherein, I/O data
The I/O data to be processed of preset quantity are arranged in queue；
Preference thread screening unit, the available thread for being lower than default queue depth for screening depth, preferably can be used
Thread；
It is corresponding, by available thread number split cells 200 specifically:
Complete I/O data are split as to the sub- I/O data of quantity identical as preferred available thread number；
Corresponding, parallelly compressed unit 300 is specific:
The each preferably available thread of control executes data compression step to corresponding sub- I/O data.
Further, the system of compression I/O data further include:
Deconsolidation process unit will be complete for when complete I/O data are split as multiple sub- I/O data simultaneously for metadata
The corresponding complete metadata of whole I/O data is also split as the sub- metadata of quantity identical as sub- I/O data；
Recomposition unit, for after the data compression step of each available thread is performed both by, according to each sub- metadata
Each sub- I/O data of compression are reconfigured to obtain completely compressing I/O data；Wherein, compressing sub- I/O data is sub- I/O data through number
According to the I/O data obtained after squeeze operation.
Wherein, include: by available thread number split cells
QAT compresses subelement, is utilized respectively QAT data compression technique to corresponding sub- I/ for controlling each available thread
O data executes data compression step.
Based on the above embodiment, present invention also provides a kind of device for compressing I/O data, the apparatus may include storages
Device and processor, wherein have computer program in the memory, which calls the computer program in the memory
When, step provided by above-described embodiment may be implemented.Certainly, which can also include various necessary network interfaces, electricity
Source and other components etc..
Present invention also provides a kind of computer readable storage mediums, have computer program thereon, the computer program
Step provided by above-described embodiment may be implemented when being performed terminal or processor execution.The storage medium may include: U
Disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access
Memory, RAM), the various media that can store program code such as magnetic or disk.
Specific examples are used herein to illustrate the principle and implementation manner of the present application, and between each embodiment
For progressive relationship, each embodiment focuses on the differences from other embodiments, identical between each embodiment
Similar portion may refer to each other.For the device disclosed in the embodiment, reference can be made to corresponding method part illustration.The above reality
The explanation for applying example is merely used to help understand the present processes and its core concept.For the ordinary skill people of the art
Member for, under the premise of not departing from the application principle, can also to the application, some improvement and modification can also be carried out, these improve and
Modification is also fallen into the protection scope of the claim of this application.
It should also be noted that, in the present specification, relational terms such as first and second and the like be used merely to by
One entity or operation are distinguished with another entity or operation, without necessarily requiring or implying these entities or operation
Between there are any actual relationship or orders.Moreover, the terms "include", "comprise" or its any other variant meaning
Covering non-exclusive inclusion, so that the process, method, article or equipment for including a series of elements not only includes that
A little elements, but also other elements including being not explicitly listed, or further include for this process, method, article or
The intrinsic element of equipment.In the absence of more restrictions, the element limited by sentence "including a ...", is not arranged
Except there is also other identical elements in the process, method, article or equipment for including element.
Claims (10)
1. a kind of method for compressing I/O data characterized by comprising
Receive the complete I/O data issued；
The complete I/O data are split as to the sub- I/O data of quantity identical as available thread number；
It controls each available thread and data compression step is executed to corresponding sub- I/O data respectively.
2. the method according to claim 1, wherein further include:
Detect the depth of the I/O data queue of each available thread；Wherein, it is arranged in the I/O data queue default
The I/O data to be processed of quantity；
The available thread that the depth is lower than default queue depth is screened, preferred available thread is obtained；
It is corresponding, the complete I/O data are split as to the sub- I/O data of quantity identical as available thread number, specifically:
The complete I/O data are split as to the sub- I/O data of quantity identical as the preferred available thread number；
It is corresponding, it controls each available thread and data compression step is executed to corresponding sub- I/O data respectively, specific:
It controls each preferred available thread and data compression step is executed to corresponding sub- I/O data.
3. the method according to claim 1, wherein further include:
It is when the complete I/O data are split as multiple sub- I/O data, the complete I/O data are corresponding complete
Metadata is also split as the sub- metadata of quantity identical as the sub- I/O data；
After the data compression step of each available thread is performed both by, reconfigured respectively according to each sub- metadata
Sub- I/O data are compressed to obtain completely compressing I/O data；Wherein, the sub- I/O data of compression are described in the sub- I/O data warp
The I/O data obtained after data compression step.
4. method according to any one of claims 1 to 3, which is characterized in that control each available thread respectively to correspondence
Sub- I/O data execute data compression step, comprising:
It controls each available thread and is utilized respectively QAT data compression technique to corresponding sub- I/O data execution data compression
Operation.
5. a kind of system for compressing I/O data characterized by comprising
Complete I/O data receipt unit, for receiving the complete I/O data issued；
By available thread number split cells, for the complete I/O data to be split as to the son of quantity identical as available thread number
I/O data；
Parallelly compressed unit executes data compression step to corresponding sub- I/O data respectively for controlling each available thread.
6. system according to claim 5, which is characterized in that further include:
Queue depth's detection unit, the depth of the I/O data queue for detecting each available thread；Wherein, the I/O
The I/O data to be processed of preset quantity are arranged in data queue；
Preference thread screening unit, the available thread for being lower than default queue depth for screening the depth, preferably can be used
Thread；
It is corresponding, it is described to press available thread number split cells specifically:
The complete I/O data are split as to the sub- I/O data of quantity identical as the preferred available thread number；
Corresponding, the parallelly compressed unit is specific:
It controls each preferred available thread and data compression step is executed to corresponding sub- I/O data.
7. system according to claim 5, which is characterized in that further include:
Metadata deconsolidation process unit simultaneously, for when the complete I/O data are split as multiple sub- I/O data,
The corresponding complete metadata of the complete I/O data is also split as to the sub- metadata of quantity identical as the sub- I/O data；
Recomposition unit, for after the data compression step of each available thread is performed both by, according to each son member
Data reconfigure each sub- I/O data of compression and obtain completely compressing I/O data；Wherein, the sub- I/O data of compression are the son
The I/O data that I/O data obtain after the data compression step.
8. according to the described in any item systems of claim 5 to 7, which is characterized in that described to press available thread number split cells packet
It includes:
QAT compresses subelement, is utilized respectively QAT data compression technique to corresponding sub- I/ for controlling each available thread
O data executes data compression step.
9. a kind of device for compressing I/O data characterized by comprising
Memory, for storing computer program；
Processor realizes such as Claims 1-4 described in any item compression I/O data when for executing the computer program
Method the step of.
10. a kind of computer readable storage medium, which is characterized in that be stored with computer on the computer readable storage medium
Program is realized when the computer program is executed by processor such as the described in any item compression I/O data of Claims 1-4
The step of method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810935726.2A CN109245772A (en) | 2018-08-16 | 2018-08-16 | A kind of method, system, device and readable storage medium storing program for executing compressing I/O data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810935726.2A CN109245772A (en) | 2018-08-16 | 2018-08-16 | A kind of method, system, device and readable storage medium storing program for executing compressing I/O data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN109245772A true CN109245772A (en) | 2019-01-18 |
Family
ID=65070560
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810935726.2A Withdrawn CN109245772A (en) | 2018-08-16 | 2018-08-16 | A kind of method, system, device and readable storage medium storing program for executing compressing I/O data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109245772A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113741991A (en) * | 2021-11-04 | 2021-12-03 | 苏州浪潮智能科技有限公司 | Method and device for improving SSD service quality, computer equipment and storage medium |
| CN114153383A (en) * | 2021-11-05 | 2022-03-08 | 广州文远知行科技有限公司 | Data compression storage method and device for automatic vehicle driving system |
-
2018
- 2018-08-16 CN CN201810935726.2A patent/CN109245772A/en not_active Withdrawn
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113741991A (en) * | 2021-11-04 | 2021-12-03 | 苏州浪潮智能科技有限公司 | Method and device for improving SSD service quality, computer equipment and storage medium |
| CN113741991B (en) * | 2021-11-04 | 2022-02-18 | 苏州浪潮智能科技有限公司 | Method and device for improving SSD service quality, computer equipment and storage medium |
| CN114153383A (en) * | 2021-11-05 | 2022-03-08 | 广州文远知行科技有限公司 | Data compression storage method and device for automatic vehicle driving system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10838642B2 (en) | Efficient modification of storage system metadata | |
| US11106393B2 (en) | Latency-based scheduling of command processing in data storage devices | |
| US20160110121A1 (en) | Resource reservation for storage system metadata updates | |
| US8706962B2 (en) | Multi-tier storage system configuration adviser | |
| JP5500256B2 (en) | Storage system | |
| CN105204781B (en) | Compression method, device and equipment | |
| US9712646B2 (en) | Automated client/server operation partitioning | |
| US9684512B2 (en) | Adaptive Map-Reduce pipeline with dynamic thread allocations | |
| EP2378410A2 (en) | Method and apparatus to manage tier information | |
| US9886310B2 (en) | Dynamic resource allocation in MapReduce | |
| CN103914516A (en) | Method and system for layer-management of storage system | |
| CN105574141B (en) | Method and device for carrying out data migration on database | |
| CN103577454A (en) | Document merging method and document merging device | |
| US20210181986A1 (en) | Storage device, storage system and operating method thereof | |
| US20170003911A1 (en) | Information processing device | |
| CN109245772A (en) | A kind of method, system, device and readable storage medium storing program for executing compressing I/O data | |
| CN103353850A (en) | Virtual machine thermal migration memory processing method, device and system | |
| CN103984528A (en) | Multithread concurrent data compression method based on FT processor platform | |
| CN103927215B (en) | Optimization method and system based on ram disk and the kvm scheduling virtual machines of SSD hard disks | |
| CN104040512A (en) | Method and device for processing storage space and non-volatile computer readable storage medium | |
| US8713278B2 (en) | System and method for stranded file opens during disk compression utility requests | |
| US10380066B2 (en) | File system with multi-class in situ tiered archiving | |
| CN102184141A (en) | Method and device for storing check point data | |
| US10180901B2 (en) | Apparatus, system and method for managing space in a storage device | |
| CN105630410B (en) | The method for managing file storage area using non-volatile random read-write storage chip dynamic |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20190118 |
|
| WW01 | Invention patent application withdrawn after publication |