CN110162561A - Offline compression method, device, computer equipment and storage medium - Google Patents
Offline compression method, device, computer equipment and storage medium Download PDFInfo
- Publication number
- CN110162561A CN110162561A CN201910320212.0A CN201910320212A CN110162561A CN 110162561 A CN110162561 A CN 110162561A CN 201910320212 A CN201910320212 A CN 201910320212A CN 110162561 A CN110162561 A CN 110162561A
- Authority
- CN
- China
- Prior art keywords
- data
- compressed
- compression
- locking
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000007906 compression Methods 0.000 title claims abstract description 200
- 230000006835 compression Effects 0.000 title claims abstract description 200
- 238000000034 method Methods 0.000 title claims abstract description 39
- 238000003860 storage Methods 0.000 title claims abstract description 22
- 230000006870 function Effects 0.000 claims abstract description 29
- 238000004590 computer program Methods 0.000 claims description 18
- 238000013144 data compression Methods 0.000 claims description 12
- 235000019580 granularity Nutrition 0.000 description 32
- 238000010586 diagram Methods 0.000 description 3
- 230000003321 amplification Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000003199 nucleic acid amplification method Methods 0.000 description 2
- 241000208340 Araliaceae Species 0.000 description 1
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 1
- 235000003140 Panax quinquefolius Nutrition 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 230000006837 decompression Effects 0.000 description 1
- 238000007599 discharging Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000002708 enhancing effect Effects 0.000 description 1
- 235000008434 ginseng Nutrition 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000005303 weighing Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2453—Query optimisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of offline compression method, device, computer equipment and storage mediums.This method comprises: obtaining data list to be compressed by data-interface, data to be compressed are sequentially obtained based on data list to be compressed；Data to be compressed are sent to corresponding lock function by lock port, obtain lock function treated locking compressed data；Locking compressed data is compressed, compressed locking compressed data is obtained；Call memory interface that compressed locking compressed data is stored on hard disk；Solution lock interface is called to be unlocked compressed locking compressed data, and pass through the space before calling space reclamation interface release locking compressed data to compress, both it may be implemented to treat compressed data and be compressed, it will not increase burden to business access, guarantee normal business access speed and access efficiency.
Description
Technical field
The present invention relates to computer development field more particularly to a kind of offline compression method, device, computer equipment and deposit
Storage media.
Background technique
Existing ceph provides line compression function, but without offline compression function.Compressing to data can be effective
Memory space is saved, reduces enterprise's hardware cost, but compress to data cpu resource can be lost, if compression procedure is busy,
Application system certainly will be influenced to the access efficiency of storage；It needs first to decompress to return again to when in addition, reading compressed data and not press
Contracting data will affect client to the access speed of compressed data in this way, increase access delay to client.Since consumption is compared in compression
When, line compression will increase time delay, so that data access becomes the efficiency for slowly influencing data interaction.
Summary of the invention
The embodiment of the present invention provides a kind of offline compression method, device, computer equipment and storage medium, online to solve
Compression influences the problem of data access efficiency.
A kind of offline compression method, comprising:
Data list to be compressed is obtained by data-interface, data to be compressed are sequentially obtained based on data list to be compressed；
The data to be compressed are sent to corresponding lock function by lock port, obtaining lock function, treated
Lock compressed data；
The locking compressed data is compressed, compressed locking compressed data is obtained；
Call memory interface that the compressed locking compressed data is stored on hard disk；
Solution lock interface is called to be unlocked the compressed locking compressed data, and by calling space reclamation interface
Space before discharging the locking compressed data compression.
A kind of offline compression set, comprising:
Data acquisition module, for obtaining data list to be compressed by data-interface, based on data list to be compressed according to
Sequence obtains data to be compressed；
Data interlock module is obtained for the data to be compressed to be sent to corresponding lock function by lock port
Take lock function treated locking compressed data；
Data compressing module obtains compressed locking compressed data for compressing to the locking compressed data；
Data memory module, for calling memory interface that the compressed locking compressed data is stored in hard disk；
Unlocking data and space reclamation module, for calling solution lock interface to carry out the compressed locking compressed data
Unlock, and pass through the space before calling space reclamation interface to discharge the locking compressed data compression.
A kind of computer equipment, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, the processor realize above-mentioned offline compression method when executing the computer program.
A kind of computer readable storage medium, the computer-readable recording medium storage have computer program, the meter
Calculation machine program realizes above-mentioned offline compression method when being executed by processor.
Above-mentioned offline compression method, device, computer equipment and storage medium, offline compressor is by obtaining number to be compressed
According to list, so that sequence of the offline compressor according to the data to be compressed stored in data list to be compressed, sequentially to be compressed
Data are compressed.After obtaining data to be compressed, compressing offline compressor offline can be by lock port by data to be compressed
Locking, to guarantee that data to be compressed can be well on compression.Then determine that compression Thread Count, compression initial time, compression are temporary
Between stopping time and the compression of compressed data is treated in compression granularity, completion.After data to be compressed are completed to compress, offline compressor will be to
Compressed data is stored on hard disk, and is unlocked to the data to be compressed for completing compression, to facilitate subsequent use.Using from
Wire compression device is treated compressed data and is compressed, and both may be implemented to treat compressed data and compressed, will not be to business access
It increases burden, guarantees normal business access speed and access efficiency.
Detailed description of the invention
In order to illustrate the technical solution of the embodiments of the present invention more clearly, below by institute in the description to the embodiment of the present invention
Attached drawing to be used is needed to be briefly described, it should be apparent that, the accompanying drawings in the following description is only some implementations of the invention
Example, for those of ordinary skill in the art, without any creative labor, can also be according to these attached drawings
Obtain other attached drawings.
Fig. 1 is an application scenario diagram of offline compression method in one embodiment of the invention；
Fig. 2 is a flow chart of offline compression method in one embodiment of the invention；
Fig. 3 is a specific flow chart of step S30 in Fig. 2；
Fig. 4 is a specific flow chart of step S32 in Fig. 3；
Fig. 5 is a specific flow chart of step S33 in Fig. 3；
Fig. 6 is another flow chart of offline compression method in one embodiment of the invention；
Fig. 7 is a functional block diagram of offline compression set in one embodiment of the invention；
Fig. 8 is a schematic diagram of computer equipment in one embodiment of the invention.
Specific embodiment
Following will be combined with the drawings in the embodiments of the present invention, and technical solution in the embodiment of the present invention carries out clear, complete
Site preparation description, it is clear that described embodiments are some of the embodiments of the present invention, instead of all the embodiments.Based on this hair
Embodiment in bright, every other implementation obtained by those of ordinary skill in the art without making creative efforts
Example, shall fall within the protection scope of the present invention.
Offline compression method provided by the present application, can be applicable in the application environment such as Fig. 1, wherein terminal device passes through
Network is communicated with offline compressor.Wherein, terminal device can be, but not limited to various personal computers, laptop,
Smart phone, tablet computer and portable wearable device.Offline compressor can use independent server either multiple clothes
The server cluster of business device composition is realized.
In one embodiment, it as shown in Fig. 2, providing a kind of offline compression method, applies in this way offline in Fig. 1
It is illustrated, includes the following steps: for compressor
S10: obtaining data list to be compressed by data-interface, sequentially obtains number to be compressed based on data list to be compressed
According to.
Wherein, data-interface refers to the interface for obtaining data.Data-interface in the present embodiment refers to for obtaining number to be compressed
According to the interface of list.Data list to be compressed refers to the list for storing data to be compressed, and data to be compressed, which refer to, needs compressed number
According to.
Specifically, offline compressor obtains data list to be compressed by data-interface, is obtaining data list to be compressed
Afterwards, data to be compressed are sequentially obtained according to putting in order for data to be compressed in data list to be compressed.According to data to be compressed
List sequentially obtains data to be compressed, treats compressed data according to the sequence for obtaining data to be compressed so as to subsequent step and is pressed
Contracting.
S20: data to be compressed are sent to corresponding lock function by lock port, obtaining lock function, treated
Lock compressed data.
Lock port refers to the interface for the object that locking to be compressed.Object in the present embodiment refers specifically to data to be compressed.
Specifically, after obtaining data to be compressed, data to be compressed are sent to lock by lock port by offline compressor
Determine function, lock function is treated compressed data and is locked, so that data to be compressed are not used during being compressed,
Guarantee the smooth compression of data to be compressed.Wherein, lock function refers to the function for locking data to be compressed.Lock compressed data
Refer to the data to be compressed locked using lock function.
S30: locking compressed data is compressed, compressed locking compressed data is obtained.
Specifically, after by data interlock to be compressed, memory can be consumed due to compressing to locking compressed data, so that
CPU consumption is excessive, influences normal business access, and therefore, offline compressor, can be periodically when compressing to locking data
Ground detects current business amount of access, determines compression Thread Count according to current business amount of access.Wherein, current business amount of access, which refers to, works as
Preceding moment, the quantity of the business access of offline compressor statistics.
Since normal business access can be interfered with when compressing to locking compressed data, in the present embodiment also
The current business amount of access of real-time statistics is needed, to determine compression initial time and compression time out.When current business accesses
When measuring larger (number of services that such as current business amount of access is greater than pre-set standard), stop the pressure to locking compressed data
Contracting；(number of services that such as current business amount of access is less than pre-set standard), starting when current business amount of access is smaller
Compression to locking compressed data.Wherein compression initial time refers to the time for starting to be compressed to locking compressed data；Compression
Time out refers to the time that locking compressed data is compressed in pause.
Offline compressor is after determining compression Thread Count, compression initial time and compression time out, it is also necessary to according to
The data attribute for locking compressed data determines the compression granularity of locking compressed data, and the compression to locking compressed data is completed.
Wherein, data attribute refers to that determining whether the locking compressed data belongs to according to the service life of locking compressed data is commonly used
The attribute of data.Compress the compression unit of granularity finger pressure contracting locking compressed data, such as 64K, 4M.
S40: call memory interface that compressed locking compressed data is stored on hard disk.
Wherein, memory interface refers to the interface for receiving and saving the data after the completion of compressing.Specifically, offline compressor
After completing compression to locking compressed data, memory interface can be called, compressed locking compressed data is stored in corresponding hard
On disk, it is completed and compressed locking data is stored.Hard disk in the present embodiment refers to the hard of storage locking compressed data
Disk.
S50: solution lock interface is called to be unlocked compressed locking compressed data, and by calling space reclamation interface
Space before release locking compressed data compression.
Wherein, solution lock interface refers to the interface of unlocking objects.Object in the present embodiment refers to compressed locking compressed data.
Specifically, after it will lock compressed data and be stored on hard disk, offline compressor calls solution lock interface to locking compressed data
It is unlocked, guarantees that compressed locking compressed data is unlocked, facilitate subsequent use.
In order to save memory space, the position ginseng that offline compressor also needs to carry by locking compressed data is compressed offline
Number finds the storage location of data to be compressed before compression, and by calling space reclamation interface release locking compressed data pressing
Space before contracting.Wherein, space reclamation interface refers to the interface for realizing the release to space occupied by the data before compression.
Step S10- step S50, offline compressor by obtaining data list to be compressed so that offline compressor according to
The sequence of the data to be compressed stored in compressed data list is sequentially treated compressed data and is compressed.Obtaining number to be compressed
According to rear, compressing offline compressor offline can be by lock port by data interlock to be compressed, to guarantee that data to be compressed can be into
Row smoothly compression.Then determine that pressure is treated in compression Thread Count, compression initial time, compression time out and compression granularity, completion
The compression of contracting data.After data to be compressed are completed to compress, data to be compressed are stored on hard disk by offline compressor, and to complete
It is unlocked at the data to be compressed of compression, to facilitate subsequent use.Compressed data is treated using offline compressor to be pressed
Contracting both may be implemented to treat compressed data and be compressed, will not increase burden to business access, guarantee normal business access
Speed and access efficiency.
In one embodiment, as shown in figure 3, in step S30, locking compressed data is compressed, is specifically included as follows
Step:
S31: according to preset period of time detecting current business amount of access, is adjusted according to current business amount of access dynamic
Compression Thread Count in offline compressor.
Wherein, preset time refer to pre-set for detect current business amount of access have determined that compression Thread Count when
Between.If preset time is set as 15 minutes, offline compressor can detect primary current industry at interval of 15 minutes according to preset time
Business amount of access, to adjust the quantity of compression Thread Count.
Specifically, due to compression Thread Count number will affect the access efficiency of current business amount of access, compress Thread Count
Quantity it is more, will be bigger on the influence of the access efficiency of business access；Correspondingly, the quantity for compressing Thread Count is fewer, to industry
The access efficiency influence of business access will be smaller.Therefore, in order to guarantee that the access efficiency of business access is not affected, this implementation
Offline compressor in example can according to preset period of time detect current business amount of access, dynamic according to current business amount of access
State adjusts the compression Thread Count in offline compressor.Current business amount of access is more, and the compression Thread Count in offline compressor is just
It correspondingly reduces, current business amount of access is fewer, and the compression Thread Count in offline compressor just correspondingly increases.It is understood that
Ground, compression Thread Count is more, and the compression speed for locking compressed data is faster；It is fewer to compress Thread Count, locks compressed data
Compression speed is slower, selects suitable compression Thread Count that compression efficiency can be improved, and will not be excessive because of compression Thread Count
Influence normal business access.
Further, in this embodiment the calculation formula for calculating compression Thread Count is provided with, for calculating offline compressor
Compression Thread Count.Specifically: compression Thread Count=(cpu utilization rate * is offline in offline compressor cpu Thread Count-preset time
Compressor cpu Thread Count) * compression performance coefficient；Wherein, compression performance coefficient is configurable parameter, is worth between 0 to 1, numerical value
Bigger expression treat compressed data compress occupied compression Thread Count quantity it is more.
S32: obtaining current business amount of access in real time, when setting the compression of locking compressed data by current business amount of access
Between.
Specifically, when current business amount of access is larger, data will be frequently accessed, and cause memory or CPU excessively high,
If being compressed at this time to locking compressed data, due to needing first to read the number compressed from memory before the compression
According to needs write back the data in memory again upon compression, continually internally deposit and access, will lead to CPU or memory disappears
It consumes excessively high, influences normal business access.Therefore, it is provided that in the present embodiment by offline compressor, obtains current industry in real time
Be engaged in amount of access, with realize according to current business amount of access number, in time determine compression time, with ensure business access normally into
Row, it is unaffected.
S33: the data attribute by locking compressed data determines the compression granularity of locking compressed data.
Data attribute refers to that determining whether the locking compressed data belongs to according to the service life of locking compressed data often makes
The attribute of data.Service life in the present embodiment is referred to according to current time and the period being calculated using the time.
When service life is in pre-set threshold value (such as one month) range, it is determined that the data attribute of the locking compressed data is frequent
It uses；When service life is more than pre-set threshold value (as more than one month), it is determined that the data category of the locking compressed data
Property to be not frequently used.
Specifically, the service life of offline compressor statistics locking compressed data, in the use for obtaining locking compressed data
After period, locking compressed data service life is compared with pre-set threshold value, determine lock compressed data whether be
The data being frequently used.Then the compression granularity of the locking compressed data is determined according to the data attribute of locking compressed data.
Such as presetting threshold value is one month, and the service life for locking compressed data is 20 days, in one month, corresponding data attribute
For commonly using according to the corresponding compression granularity of data attribute lookup, the data of the present embodiment setting being commonly used are corresponding
Compression granularity is 64K.It determines compression granularity, offline compressor can be made to press according to compression granularity locking compressed data
Contracting.
S34: locking compressed data is compressed based on compression time, compression Thread Count and compression granularity.
Specifically, then offline compressor determines number to be compressed according to dynamic adjustment compression Thread Count according to the actual situation
According to compression initial time and compression time out, finally according to compression granularity to locking compressed data compress, realize
Locking compressed data is independently compressed using offline compressor, is not needed manual intervention, is realized Intelligent Compression.
In one embodiment, compression time includes compression initial time and compression time out.As shown in figure 4, step
S32 obtains current business amount of access in real time, the compression time of locking compressed data is set by current business amount of access, specifically
Include the following steps:
S321: obtaining current business amount of access in real time, when current business amount of access is less than standard access amount, when will be current
Carve the compression initial time for being set as locking compressed data.
Wherein, standard access amount refers to that is pre-set is used to judge the business whether offline compressor can be compressed
Amount of access.Specifically, offline compressor can obtain current business amount of access in real time, and compare current business amount of access and standard visit
The size for the amount of asking, determination compress whether offline compressor can execute squeeze operation offline.If current business amount of access is less than mark
Quasi- amount of access, then it represents that current CPU or memory can permit offline compressor and compress to locking compressed data, press at this time
Contracting, will not affect business access.Offline compressor originates the compression that current time is set as locking compressed data
Time starts to execute the squeeze operation to locking compressed data.
S322: when current business amount of access is more than or equal to standard access amount, locking pressure will be set as current time
The compression time out of contracting data.
Specifically, when current business amount of access is more than or equal to standard access amount, then it represents that current CPU or memory are not
Allow offline compressor to compress locking compressed data, if compressing at this time to locking compressed data, will increase CPU
Or the consumption of memory, influence normal business access.In order to not influence normal business access, offline compression is gone when will be current
The compression time out for being set as locking compressed data is carved, no longer execution squeeze operation.
Step S321- step S322, offline compressor determine current business amount of access by obtaining current business amount of access
Whether standard access amount is less than, when current business amount of access is more than or equal to standard access amount, then it represents that business is visited at this time
Ask excessive, unsuitable progress squeeze operation, the consumption of increase memory or CPU；When current business amount of access is less than standard access
When amount, then it represents that carrying out compression to locking compressed data at this time will not influence normal business access.Determine compression initial time
The management whether compressed to locking compressed data may be implemented with compression time out.
In one embodiment, each data to be compressed have a corresponding Data Identification, which refers to for identification
The mark of data to be compressed treats the data after compressed data is locked since locking compressed data refers to, locking compression
Data can also carry Data Identification.As shown in figure 5, step S33, the data attribute by locking compressed data determines locking pressure
The compression granularity of contracting data, specifically comprises the following steps:
S331: obtaining the service life of locking compressed data, and the service life by locking compressed data determines locking pressure
The data attribute of contracting data.
Specifically, when offline compressor obtains the use of the corresponding locking compressed data of the Data Identification according to Data Identification
Between, according to using time and current time to be calculated, obtain service life.If service life be it is multiple, choose the smallest
Data attribute of the service life (what i.e. locking compressed data was the last uses the time) as locking compressed data, to determine lock
Determine the data attribute of compressed data, provides data source for the subsequent compression granularity for obtaining locking compressed data.
S332: the compression granularity of locking compressed data is obtained by data attribute.
Specifically, offline compressor is after the data attribute for obtaining locking compressed data, according to the number of locking compressed data
The compression granularity of the locking compressed data is obtained according to attribute.The compression granularity being arranged in the present embodiment according to data attribute is specific
Are as follows: when the service life of locking compressed data is in one week, then it represents that the locking compressed data is the data frequently used, corresponding
Compression granularity be 0；When the service life of locking compressed data more than one week and in one month, is then indicating locking compression
Data be commonly using data, corresponding compression granularity be 64K；When locking compressed data service life more than one month,
Then indicate that the locking compressed data is the data being not frequently used, corresponding compression granularity is 4M.
Under normal circumstances, the bigger corresponding compression efficiency of compression granularity is higher, and the memory headroom of saving is also bigger, but meeting
The efficiency for influencing the reading of locking compressed data and writing.Because needing entirely lock compression number when carrying out locking compressed data and reading
According to reading out together, if compression granularity is 4M, entire 4M compression space can be decompressed reading the locking compressed data of 4K, is deposited
The problem of reading amplification；If carrying out locking compressed data when writing, the partial data in locking compressed data is only write, it is also desirable to will
Entire compression blocks decompression, then the part of modification is merged re-write again, if only covering the 4K in 4M compression blocks, it is also desirable to
4M data are decompressed, 4M data are written again after merging, there are problems that writing amplification.In addition, compression granularity is excessive to will lead to CPU urgency
Speed increases, and the memory overhead so as to cause entire ceph cluster is excessive, and compression undersized can reduce compression efficiency.Therefore, right
The locking compressed data of different data attribute selects different compression granularities, and the flat of compression performance and memory headroom may be implemented
Weighing apparatus.
Step S331- step S332, offline compressor lock the data attribute of compressed data by obtaining, and determine compression grain
Degree, may be implemented the balance of compression performance and memory headroom.
In one embodiment, offline compressor is before obtaining data list to be compressed by data-interface, it is also necessary to open
Offline compressor is moved, locks compressed data using offline compressor compresses convenient for subsequent.In addition, realizing subsequent step for convenience
Suddenly, it is also necessary to corresponding interface be configured to offline compressor in advance.As shown in fig. 6, the offline compression method further include:
S101: start offline compressor automatically by the progress of guarding.
The progress of guarding in the present embodiment is specially Daemon program, and Daemon program is the server-side journey run always
Sequence, also known as finger daemon.Usually in running background, without controlling terminal, information exchange, Daemon program are not carried out with foreground
It services and uses generally as system (such as operation system).Daemon is the process of long-play, usually after system start-up
Operation just terminates when system is closed.Generally Daemon program is not have controlling terminal because of it in running background, can not
With user's interaction on foreground.Daemon program is generally all used as service routine, and client-side program is waited to communicate with it.Specifically
Ground, when backstage, offline compressor starting brings into operation, deamon program will start offline compressor automatically.
S102: the interface that offline compressor needs, including data-interface, lock port, memory interface, solution lock interface are configured
With space reclamation interface.
Specifically, different interfaces is configured to offline compressor in advance, is used for subsequent step.Wherein, data-interface, which refers to, obtains
The interface for waiting compressed data is taken, corresponding interface statement is get_compress_source.Lock port refers to that locking is wanted
The interface of the object of compression, corresponding interface statement are lock_compress_obj.Memory interface refers to for receiving and saving pressure
The interface of data after the completion of contracting, corresponding interface statement are save_compressed_data.Solution lock interface refers to unlocking objects
Interface, corresponding interface statement be unlock_compress_obj.Space reclamation interface refers to realization to the data institute before compression
The interface of the release of occupied space, corresponding interface statement are release_uncompressed_space.
Step S101- step S102, offline compressor start offline compressor by the progress of guarding automatically, without artificial pipe
Reason can voluntarily start, and be pre-configured with the interface that offline compressor needs, provide necessary condition for subsequent processing.
Offline compression method provided by the invention executes following steps by the pre-configured interface of offline compressor.
Data list to be compressed is obtained, so that sequence of the offline compressor according to the data to be compressed stored in data list to be compressed,
Compressed data is sequentially treated to be compressed.After obtaining data to be compressed, lock port can be passed through by compressing offline compressor offline
By data interlock to be compressed, to guarantee that data to be compressed can be well on compression.Then it is determined according to current business amount of access
Thread Count, compression initial time, compression time out are compressed, and compression grain is determined according to the data attribute of locking compressed data
Degree completes the compression for treating compressed data.After data to be compressed are completed to compress, data to be compressed are stored on hard disk, and
The data to be compressed for completing compression are unlocked, to facilitate subsequent use.Using offline compressor treat compressed data into
Row compression, both may be implemented to treat compressed data and be compressed, will not increase burden to business access, guarantee normal business
Access speed and access efficiency.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
In one embodiment, a kind of offline compression set is provided, which presses offline with above-described embodiment
Contracting method corresponds.As shown in fig. 7, the offline compression set includes data acquisition module 10, data interlock module 20, data
Compression module 30, data memory module 40 and unlocking data and space reclamation module 50.Detailed description are as follows for each functional module:
Data acquisition module 10 is based on data list to be compressed for obtaining data list to be compressed by data-interface
Sequentially obtain data to be compressed.
Data interlock module 20 is obtained for data to be compressed to be sent to corresponding lock function by lock port
Lock function treated locking compressed data.
Data compressing module 30 obtains compressed locking compressed data for compressing to locking compressed data.
Data memory module 40, for calling memory interface that compressed locking compressed data is stored in hard disk.
Unlocking data and space reclamation module 50, for calling solution lock interface to solve compressed locking compressed data
Lock, and pass through the space before calling the release of space reclamation interface to lock compressed data compression.
Further, data compressing module includes compression Thread Count determination unit, compression time determination unit, compression granularity
Determination unit and data compression unit.
Thread Count determination unit is compressed, for according to preset period of time detecting current business amount of access, according to working as
Preceding business access amount dynamic adjusts the compression Thread Count in offline compressor.
Compression time determination unit is set by current business amount of access and is locked for obtaining current business amount of access in real time
Determine the compression time of compressed data.
Granularity determination unit is compressed, for determining the compression of locking compressed data by the data attribute of locking compressed data
Granularity.
Data compression unit, for being carried out based on compression time, compression Thread Count and compression granularity to locking compressed data
Compression.
Further, compression time determination unit includes initial time determination unit and time out determination unit.
Initial time determination unit, for obtaining current business amount of access in real time, when current business amount of access is less than standard
When amount of access, it sets current time to the compression initial time of locking compressed data.
Time out determination unit is used for when current business amount of access is more than or equal to standard access amount, will be current
Moment is set as the compression time out of locking compressed data.
Further, compression granularity determination unit includes that data attribute obtains subelement and compression granularity acquisition subelement.
Data attribute obtains subelement, for obtaining the service life of locking compressed data, passes through locking compressed data
Service life determines the data attribute of locking compressed data.
It compresses granularity and obtains subelement, for obtaining the compression granularity of locking compressed data by data attribute.
Further, offline compression set further includes offline compressor start unit and interface configuration unit.
Offline compressor start unit, for starting offline compressor automatically by the progress of guarding.
Interface configuration unit, the interface needed for configuring offline compressor, including data-interface, lock port, storage
Interface, solution lock interface and space reclamation interface.
Specific about offline compression set limits the restriction that may refer to above for offline compression method, herein not
It repeats again.Modules in above-mentioned offline compression set can be realized fully or partially through software, hardware and combinations thereof.On
Stating each module can be embedded in the form of hardware or independently of in the processor in computer equipment, can also store in a software form
In memory in computer equipment, the corresponding operation of the above modules is executed in order to which processor calls.
In one embodiment, a kind of computer equipment is provided, which can be offline compressor, in
Portion's structure chart can be as shown in Figure 8.The computer equipment includes that the processor, memory, network connected by system bus connects
Mouth and database.Wherein, the processor of the computer equipment is for providing calculating and control ability.The storage of the computer equipment
Device includes non-volatile memory medium, built-in storage.The non-volatile memory medium be stored with operating system, computer program and
Database.The built-in storage provides environment for the operation of operating system and computer program in non-volatile memory medium.It should
The database of computer equipment is for storing the data that offline compression method is related to.The network interface of the computer equipment is used for
It is communicated with external terminal by network connection.To realize a kind of offline compression side when the computer program is executed by processor
Method.
In one embodiment, a kind of computer equipment is provided, including memory, processor and storage are on a memory
And the computer program that can be run on a processor, processor realize the offline compression of above-described embodiment when executing computer program
Method, such as step shown in the step of step S10- shown in Fig. 2 S50 or Fig. 3 to Fig. 6, to avoid repeating, here no longer
It repeats.Alternatively, processor realizes the function of each module/unit in this embodiment of offline compression set when executing computer program
Can, such as data acquisition module shown in Fig. 7 10, data interlock module 20, data compressing module 30,40 and of data memory module
The function of unlocking data and space reclamation module 50, to avoid repeating, which is not described herein again.
In one embodiment, a kind of computer readable storage medium is provided, computer program is stored thereon with, is calculated
Machine program realizes the offline compression method of above-described embodiment, such as the step S50 of step S10- shown in Fig. 2 when being executed by processor,
Or step shown in Fig. 3 to Fig. 6, to avoid repeating, which is not described herein again.Alternatively, computer program is executed by processor
The function of each module/unit in this embodiment of the offline compression set of Shi Shixian, such as data acquisition module shown in Fig. 7
10, data interlock module 20, data compressing module 30, data memory module 40 and unlocking data and space reclamation module 50 function
Can, to avoid repeating, which is not described herein again.
Those of ordinary skill in the art will appreciate that realizing all or part of the process in above-described embodiment method, being can be with
Relevant hardware is instructed to complete by computer program, the computer program can be stored in a non-volatile computer
In read/write memory medium, the computer program is when being executed, it may include such as the process of the embodiment of above-mentioned each method.Wherein,
To any reference of memory, storage, database or other media used in each embodiment provided herein,
Including non-volatile and/or volatile memory.Nonvolatile memory may include read-only memory (ROM), programming ROM
(PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM) or flash memory.Volatile memory may include
Random access memory (RAM) or external cache.By way of illustration and not limitation, RAM is available in many forms,
Such as static state RAM (SRAM), dynamic ram (DRAM), synchronous dram (SDRAM), double data rate sdram (DDRSDRAM), enhancing
Type SDRAM (ESDRAM), synchronization link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM
(RDRAM), direct memory bus dynamic ram (DRDRAM) and memory bus dynamic ram (RDRAM) etc..
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that: it still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (10)
1. a kind of offline compression method, which is characterized in that the following steps executed including offline compressor:
Data list to be compressed is obtained by data-interface, data to be compressed are sequentially obtained based on data list to be compressed；
The data to be compressed are sent to corresponding lock function by lock port, obtain lock function treated locking
Compressed data；
The locking compressed data is compressed, compressed locking compressed data is obtained；
Call memory interface that the compressed locking compressed data is stored on hard disk；
Solution lock interface is called to be unlocked the compressed locking compressed data, and by calling the release of space reclamation interface
Space before the locking compressed data compression.
2. offline compression method as described in claim 1, which is characterized in that described to press the locking compressed data
Contracting, comprising:
Current business amount of access according to preset period of time is detected, is adjusted according to the current business amount of access dynamic offline
Compression Thread Count in compressor；
Current business amount of access is obtained in real time, and the compression time of locking compressed data is set by the current business amount of access；
Data attribute by locking compressed data determines the compression granularity of locking compressed data；
The locking compressed data is compressed based on the compression time, the compression Thread Count and compression granularity.
3. offline compression method as claimed in claim 2, which is characterized in that the compression time include compression initial time and
Compress time out；
The real-time acquisition current business amount of access, when setting the compression of locking compressed data by the current business amount of access
Between, comprising:
Current business amount of access is obtained in real time, when the current business amount of access is less than standard access amount, will be set at current time
It is set to the compression initial time of locking compressed data；
When the current business amount of access is more than or equal to the standard access amount, locking compression will be set as current time
The compression time out of data.
4. offline compression method as claimed in claim 2, which is characterized in that the locking compressed data carries data mark
Know；
The data attribute by locking compressed data determines the compression granularity of locking compressed data, comprising:
The service life for obtaining the locking compressed data determines the locking by the service life of the locking compressed data
The data attribute of compressed data；
The compression granularity of the locking compressed data is obtained by the data attribute.
5. offline compression method as described in claim 1, which is characterized in that obtain number to be compressed by data-interface described
Before list, the offline compression method further include:
Start offline compressor automatically by the progress of guarding；
The interface that offline compressor needs is configured, including data-interface, lock port, memory interface, solution lock interface and space are returned
Receive interface.
6. a kind of offline compression set characterized by comprising
Data acquisition module is sequentially obtained for obtaining data list to be compressed by data-interface based on data list to be compressed
Take data to be compressed；
Data interlock module obtains lock for the data to be compressed to be sent to corresponding lock function by lock port
Determine function treated locking compressed data；
Data compressing module obtains compressed locking compressed data for compressing to the locking compressed data；
Data memory module, for calling memory interface that the compressed locking compressed data is stored in hard disk；
Unlocking data and space reclamation module, for calling solution lock interface to solve the compressed locking compressed data
Lock, and pass through the space before calling space reclamation interface to discharge the locking compressed data compression.
7. offline compression set as claimed in claim 6, which is characterized in that the data compressing module includes:
Compression Thread Count determination unit is worked as according to preset period of time detecting current business amount of access according to described
Preceding business access amount dynamic adjusts the compression Thread Count in offline compressor；
Compression time determination unit is set by the current business amount of access and is locked for obtaining current business amount of access in real time
Determine the compression time of compressed data；
Granularity determination unit is compressed, for determining the compression grain of locking compressed data by the data attribute of locking compressed data
Degree；
Data compression unit, for being compressed based on the compression time, the compression Thread Count and compression granularity to the locking
Data are compressed.
8. offline compression set as claimed in claim 6, which is characterized in that the compression time determination unit includes:
Initial time determination unit, for obtaining current business amount of access in real time, when the current business amount of access is less than standard
When amount of access, it sets current time to the compression initial time of locking compressed data；
Time out determination unit is used for when the current business amount of access is more than or equal to the standard access amount, will
Current time is set as the compression time out of locking compressed data.
9. a kind of computer equipment, including memory, processor and storage are in the memory and can be in the processor
The computer program of upper operation, which is characterized in that the processor realized when executing the computer program as claim 1 to
Any one of 5 offline compression methods.
10. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In realization offline compression method as described in any one of claim 1 to 5 when the computer program is executed by processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910320212.0A CN110162561B (en) | 2019-04-19 | 2019-04-19 | Offline compression method, offline compression device, computer equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910320212.0A CN110162561B (en) | 2019-04-19 | 2019-04-19 | Offline compression method, offline compression device, computer equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110162561A true CN110162561A (en) | 2019-08-23 |
| CN110162561B CN110162561B (en) | 2024-03-01 |
Family
ID=67638636
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910320212.0A Active CN110162561B (en) | 2019-04-19 | 2019-04-19 | Offline compression method, offline compression device, computer equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110162561B (en) |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090006728A1 (en) * | 2007-06-26 | 2009-01-01 | Microsoft Corporation | Virtual machine state snapshots |
| JP2010171609A (en) * | 2009-01-21 | 2010-08-05 | Hitachi Ltd | Image processing apparatus |
| CN108563719A (en) * | 2018-04-02 | 2018-09-21 | 郑州云海信息技术有限公司 | A kind of LZO compression methods and system for abnormal log dump |
-
2019
- 2019-04-19 CN CN201910320212.0A patent/CN110162561B/en active Active
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090006728A1 (en) * | 2007-06-26 | 2009-01-01 | Microsoft Corporation | Virtual machine state snapshots |
| JP2010171609A (en) * | 2009-01-21 | 2010-08-05 | Hitachi Ltd | Image processing apparatus |
| CN108563719A (en) * | 2018-04-02 | 2018-09-21 | 郑州云海信息技术有限公司 | A kind of LZO compression methods and system for abnormal log dump |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110162561B (en) | 2024-03-01 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN106484331B (en) | A kind of data processing method, device and flash memory device | |
| CN108304553B (en) | Data migration method and device, computer equipment and storage medium | |
| CN111796856B (en) | Differential upgrading method and device, storage medium and computer equipment | |
| CN109766349A (en) | The anti-weighing method of task, device, computer equipment and storage medium | |
| CN108196959B (en) | Resource management method and device of ETL system | |
| CN110895524B (en) | Composite overdue method, device, server and storage medium of full-load redis time key | |
| CN110781214A (en) | Database reading and writing method and device, computer equipment and storage medium | |
| CN111324623A (en) | Resource transfer data checking method and device, computer equipment and storage medium | |
| CN108958891B (en) | Virtual machine memory allocation method, device and terminal | |
| CN109359809A (en) | Task distribution method, system, computer equipment and storage medium | |
| CN115269544A (en) | Database cluster upgrading method and device, electronic equipment and storage medium | |
| CN106603125B (en) | Method and device for data interaction with Bluetooth equipment | |
| CN114237505A (en) | Batch processing method and device of business data and computer equipment | |
| CN111008181A (en) | Method, system, terminal and storage medium for switching storage strategies of distributed file system | |
| CN110162561A (en) | Offline compression method, device, computer equipment and storage medium | |
| CN113419672A (en) | Storage capacity management method, system and storage medium | |
| CN110413206A (en) | Method of controlling operation thereof, equipment and computer program product in storage system | |
| CN108363719A (en) | The transparent compressing method that can configure in distributed file system | |
| CN112506698A (en) | Small file data reconstruction recovery method, system, terminal and storage medium | |
| CN114610243B (en) | Method, system, storage medium and equipment for converting thin volume | |
| CN114721814A (en) | Task allocation method and device based on shared stack and computer equipment | |
| CN115269558A (en) | Data storage method, apparatus, device, storage medium and program product | |
| CN110851433A (en) | Key optimization method for key value storage system, storage medium, electronic device and system | |
| CN112631771B (en) | Parallel processing method of big data system | |
| CN111399622B (en) | Server power consumption control method and system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |