CN104348490A - Combined data compression algorithm based on effect optimization - Google Patents
Combined data compression algorithm based on effect optimization Download PDFInfo
- Publication number
- CN104348490A CN104348490A CN201410647747.6A CN201410647747A CN104348490A CN 104348490 A CN104348490 A CN 104348490A CN 201410647747 A CN201410647747 A CN 201410647747A CN 104348490 A CN104348490 A CN 104348490A
- Authority
- CN
- China
- Prior art keywords
- compression
- data
- algorithm
- compression algorithm
- data block
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Abstract
The invention relates to a combined data compression algorithm based on effect optimization. The combined data compression algorithm comprises the following steps of forming a compression algorithm set by selecting multiple compression algorithms according to to-be-compressed data types, main characteristics and/or time limiting factors, and obtaining an average decompression speed Di of decompression algorithms to which the multiple compression algorithms correspond; analyzing to-be-compressed data requirements, dividing to-be-compressed data into multiple data blocks, and obtaining public parameters of a computer system; calculating the compression effect values of the multiple compression algorithms used by each data block; comparing the compression effect values of the multiple compression algorithms, and selecting the algorithms with smallest compression effect value as optimal algorithms; implementing data compression calculation according to the optimal algorithm to which each data block corresponds. According to the combined data compression algorithm based on the effect optimization, the advantages of the multiple compression algorithms are fused; meanwhile, the influences of data reading speeds and disk space resources are considered; the data compression effect can be furthest improved, and the performance of practical data reading can also be improved.
Description
Technical field
The present invention relates to computerized algorithm technical field, particularly relate to a kind of based on the preferred data splitting compression algorithm of effect.
Background technology
At present, the programmed algorithm principle property of there are differences, different compression algorithms is not identical to the compression effectiveness of same type data, and in like manner, identical compression algorithm is not identical for the compression effectiveness of different types of data yet.Unpredictable due to concrete data characteristics, causes any single data compression algorithm all cannot reach optimal compression effect.For the compression of different types of data, how just can reach desirable compression effectiveness, be the hot issue of current people research and exploitation always.
We show the data of experiment statistics, and the combination compression algorithm of feature based value can make the compression algorithm of the data selection of about 15% can not reach optimal compression effect, and its compression effectiveness is greatly between 40% ~ 80% of optimal compression effect.Therefore, with regard to conceptual data, there is the optimization space of 15%* (1-60%)=6% in the combination compression algorithm of feature based value.
Under compression and the little scene of the number of times difference of decompress(ion), the optimization in the optimization space of above-mentioned 6% is worth and little, and in the scene of decompress(ion) number of times much larger than compression number of times, as analytic type database, sacrifice certain compression performance, thus improve the compression effectiveness of above-mentioned 6%, in repeatedly decompression procedure, overall decompress(ion) reading performance can be improved greatly.
The optimization space that the combination compression algorithm that the present invention is directed to feature based value exists, propose and preferably combine compression algorithm based on effect, this algorithm is by the contrast to actual compression result, from alternative compression algorithms selection optimal compression algorithm, the actual compression effect of data can be improved to greatest extent.
Summary of the invention
Technical problem to be solved by this invention is that the compression algorithm how solving current feature based value exists partial data and cannot reach the optimized key issue of compression.
For this purpose, the present invention proposes a kind of based on the preferred data splitting compression algorithm of effect, comprise following concrete steps:
S1: choose multiple compression algorithm according to data type to be compressed, principal character and/or time restriction factor, forms compression algorithm collection, and
Obtain the average decompress(ion) speed D of decompression algorithm corresponding to described multiple compression algorithm
i;
S2: analyze described demand data to be compressed, be divided into multiple data block, and
Obtain the common parameter of described computer system;
S3: calculate the compression effectiveness value that described each data block uses described multiple compression algorithm;
S4: the described compression effectiveness value of more described multiple compression algorithm, chooses the minimum algorithm of compression effectiveness value as optimal algorithm;
S5: the described optimal algorithm corresponding according to described each data block carries out data compression calculating.
Further, described step S3 also comprises:
S31: for described multiple data block, preset data size is C
0;
S32: travel through described compression algorithm collection described multiple compression algorithm wherein and compression calculating is carried out to described each data block, and add up the size CR of compression result
i.
Further, described step S3 also comprises:
S31 ': calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i;
S32 ': the reading total time T when data calculated in described each data block are not compressed
0and compression effectiveness value.
Particularly, formula is passed through:
T
i＝(CR
i/D
i)+(CR
i/V)
Calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i, wherein, the size CR of described compression result
i, the average decompression rate of often kind of algorithm in described multiple compression algorithm is D
i, the reading speed of average I/O is V.
Particularly, formula is passed through:
T
0＝C
0/V
Calculate reading total time T when described multiple compression algorithm is not compressed for the data in described each data block
0.
Particularly, formula is passed through:
CEV
0＝(T
0/T
0)+C
0*DSR＝1+C
0*DSR
Calculate compression effectiveness value when described multiple compression algorithm is not compressed for the data in described each data block, wherein, described preset data size is C
0, disk coefficient of sensitivity is DSR.
Further, described step S3 also comprises:
Pass through formula:
CEV
i＝(T
i/T
0)+CR
i*DSR
Calculate the compression effectiveness value that described each data block uses described multiple compression algorithm, wherein, calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i, the reading total time T when data calculated in described each data block are not compressed
0, the size CR of described compression result
i, described disk coefficient of sensitivity is DSR.
Further, also comprise after described step S5: record described each data block compression result, and described optimal algorithm information.
The invention discloses a kind of based on the preferred data splitting compression algorithm of effect, combination compression algorithm based on effect has merged the advantage of multiple compression algorithm, consider the impact of data reading speed and disk space resource simultaneously, can maximized raising data compression effects, and the performance of the actual reading of data can be improved; Further, the advantage that combination compression algorithm combines LZ4, Huffman encoding algorithm, gzip compress three kinds of compression algorithms, can maximized raising data compression effects, thus the effective reading performance improving data.
Accompanying drawing explanation
Can understanding the features and advantages of the present invention clearly by reference to accompanying drawing, accompanying drawing is schematic and should not be construed as and carry out any restriction to the present invention, in the accompanying drawings:
Fig. 1 shows a kind of flow chart of steps based on the preferred data splitting compression algorithm of effect in the embodiment of the present invention;
Fig. 2 shows a kind of flow chart of steps based on the preferred data splitting compression algorithm of effect in another embodiment of the present invention.
Embodiment
First following noun is explained in detail: 1) compression algorithm collection: the set of multiple compression algorithm composition, according to different data characteristicses, scene etc., different compression algorithms can be selected to form compression algorithm collection; 2) average decompression rate: when the decompression algorithm using compression algorithm corresponding performs decompression operations, the average amount of the packed data that can process in the unit interval; DSR (Disk Sensitivity Ratio, disk coefficient of sensitivity): the coefficient of sensitivity of data when disk stores of some, the value of this coefficient is the inverse of TDV (Total Disk Volume: disk total capacity) and the long-pending of ES (Expert Score: expert estimation): DSR=ES/TDV.Wherein, TDV is the parameter of system hardware, and ES is the marking provided after carrying out overall merit by expert to application scenarios and system.DSR is less, represents more insensitive for coefficient.Under extreme case, when disk is infinitely great, DSR is 0; 4) numerical value of evaluation compression effectiveness of CEV (Compress Effect Value: compression effectiveness value) by calculating, unit is 1.Below in conjunction with accompanying drawing, embodiments of the present invention is described in detail.
As shown in Figure 1, the invention provides a kind of based on the preferred data splitting compression algorithm of effect, comprise following concrete steps:
Step S1: choose multiple compression algorithm according to data type to be compressed, principal character and/or time restriction factor, forms compression algorithm collection, and obtains the average decompress(ion) speed D of decompression algorithm corresponding to multiple compression algorithm
i.
Data to be compressed are divided into multiple data block by step S2: analyze demand data to be compressed, and obtain the common parameter of computer system.
Step S3: calculate the compression effectiveness value that each data block uses multiple compression algorithm.
Particularly, step S3 also comprises:
Step S31: for multiple data block, preset data size is C
0.
Step S32: traversal compression algorithm collection multiple compression algorithm is wherein carried out compression to each data block and calculated, and adds up the size CR of compression result
i.
Further, step S3 also comprises:
Step S31 ': calculate the total reading time T of multiple compression algorithm for the compression result of multiple data block
i.
Step S32 ': the reading total time T when data calculated in multiple data block are not compressed
0and compression effectiveness value.
Particularly, formula is passed through: T
i=(CR
i/ D
i)+(CR
i/ V)
Calculate the total reading time T of multiple compression algorithm for the compression result of each data block
i, wherein, the size CR of compression result
i, the average decompression rate of often kind of algorithm in multiple compression algorithm is D
i, the reading speed of average I/O is V.
Further, formula is passed through: T
0=C
0/ V
Calculate reading total time T when multiple compression algorithm is not compressed for the data in each data block
0.
Further, formula is passed through:
CEV
0＝(T
0/T
0)+C
0*DSR＝1+C
0*DSR
Calculate compression effectiveness value when multiple compression algorithm is not compressed for the data in each data block, wherein, preset data size is C
0, disk coefficient of sensitivity is DSR.
Further, final purpose due to data compression is to read available data faster, when data are not compressed, although data volume is larger than data after compression, but do not need decompress(ion) when reading, there is less possibility of total reading time, namely when Selective Pressure compression algorithm, need the data of not packed data that always the time of reading adds with reference in Alternative algorithms.
Particularly, formula is passed through:
CEV
i＝(T
i/T
0)+CR
i*DSR
Calculate the compression effectiveness value that each data block uses multiple compression algorithm, wherein, calculate the total reading time T of multiple compression algorithm for the compression result of each data block
i, the reading total time T when data calculated in each data block are not compressed
0, the size CR of compression result
i, disk coefficient of sensitivity is DSR.
Further, the time of digital independent comprises the time of data after disk reading compression and data is carried out to the time of decompress(ion).
Step S4: the compression effectiveness value of more multiple compression algorithm, chooses the minimum algorithm of compression effectiveness value as optimal algorithm.
Step S5: the optimal algorithm corresponding according to each data block carries out data compression calculating.
Further, also comprise after step S5: record each data block compression result, and described optimal algorithm information, thus when the data of this compression algorithm of decompress(ion), decompression algorithm can use corresponding algorithm to carry out decompress(ion), effectively, accurately obtains decompressed data.
In order to understand better with the one that proposes of application the present invention based on the preferred data splitting compression algorithm of effect, composition graphs 2 couples of the present invention carry out following example, and the present invention not only limits to following example.
Particularly, with large table data instance conventional in row deposit data storehouse, in the list table data that communication common carrier is detailed, be divided into three class data according to data characteristics: take exchange hour as a large amount of repetition ordered datas of representative, take user area as a large amount of repetition non-ordered data of representative and the less repetition non-ordered data that is representative with tranaction costs (be namely accurate to point).
In conventional compressed encoding model, the Run-Length Coding belonging to dictionary model is in close relations with the ordering of initial data, and irrelevant with its frequency of occurrences; The huffman coding and the initial data frequency of occurrences that belong to statistical model are in close relations, and have nothing to do with its ordering.Therefore, take exchange hour as the feature of the ordered data of representative, select LZ4 algorithm; According to the feature of radix compared with small data that user area is representative, select Huffman encoding algorithm; According to the feature that tranaction costs (be namely accurate to point) are the less repetition non-ordered data of representative, select the gzip algorithm of comprehensive two kinds of algorithm characteristics.Therefore, in this example, compression algorithm collection is made up of LZ4 algorithm, Huffman encoding algorithm, gzip algorithm three kinds.
Further, a large amount of repetition ordered data uses the LZ4 compression algorithm of Run-Length Coding, this algorithm uses Run-Length Coding to realize, the number of times that this algorithm repeats by digital data of description, when data repeat in a large number in order, this algorithm has high compression ratio, and all data compressions can be applicable in the primitive stored to one, therefore all data to be compressed only need to be divided into 1 data block.Repeat unordered data selection in a large number and use Huffman encoding algorithm, when radix value is larger, this algorithm needs to use more space to store all radixes.Therefore, optimum organization compression algorithm only has the data total amount suitably reducing single compression, data compression result could be saved in the applicable data cell stored, thus realize optimum compression effectiveness.About in this example, in the large table of database, data to be compressed need to be divided into 10 ~ 20 data blocks.Unordered data selection use gzip algorithm is repeated, because its average compression ratio is less than the algorithm of other two types data use for less.Therefore, about the list table in this example needs total data to be divided into 50 ~ 100 data blocks, guarantee compression result can be stored in primitive.
Particularly, first use LZ4 algorithm to carry out compression to data block and calculate, obtain the size CR compressing rear data
1, then use Huffman encoding algorithm to carry out compression to data block and calculate, obtain the size CR compressing rear data
2, finally use gzip compression algorithm to carry out compression to data block and calculate, obtain the size of data CR after compressing
3.
Further, the average decompression rate D of LZ4 algorithm, Huffman encoding algorithm, gzip algorithm is obtained respectively
1, D
2, D
3and average I/O reading speed V, and in conjunction with the size of data CR after the compression of the three kinds of algorithms obtained in aforesaid operations
1, CR
2, CR
3, calculate the total reading time T of three kinds of algorithms for the compression result of this data block
1, T
2, T
3.
Further, the CEV of LZ4 algorithm, Huffman encoding algorithm, gzip algorithm is calculated respectively
1, CEV
2, CEV
3.Wherein, the CEV that compression algorithm collection comprises LZ4, Huffman encoding algorithm, gzip compress three kinds of algorithms and do not compress, namely compression effectiveness value is respectively CEV
1, CEV
2, CEV
3, CEV
0.Therefrom choose minimum value, be assumed to CEV
2, the algorithm of its correspondence and Huffman encoding algorithm are exactly the optimal algorithm of this data block.The compression result that in preservation aforesaid operations process, Huffman encoding algorithm calculates is as the compression result of this data block.
Further, while recording data blocks compression result, need to record this data block and use Huffman Compress Algorithm, correct decompression algorithm could be used when decompress(ion) like this.
The invention discloses a kind of based on the preferred data splitting compression algorithm of effect, combination compression algorithm based on effect has merged the advantage of multiple compression algorithm, consider the impact of data reading speed and disk space resource simultaneously, while maximized raising data compression effects, the performance of the actual reading of data can be improved; Further, the advantage that combination compression algorithm combines LZ4, Huffman encoding algorithm, gzip compress three kinds of compression algorithms, can maximized raising data compression effects, thus the effective reading performance improving data; Further, although when loading, this combinational algorithm to data use respectively LZ4, Huffman encoding algorithm, gzip compress three kinds of compression algorithms carry out compression calculate, reduce Data import performance, but greatly improve digital independent performance, for once loading the use scenes repeatedly read, such as, large data query analysis etc. its in the repeatedly reading of data, the lifting of reading performance is far longer than the reduction of loading performance, thus provides efficient data compression algorithm for this kind of scene.
Although describe embodiments of the present invention by reference to the accompanying drawings, but those skilled in the art can make various modifications and variations without departing from the spirit and scope of the present invention, such amendment and modification all fall into by within claims limited range.
Claims (8)
1., based on the preferred data splitting compression algorithm of effect, it is characterized in that, comprise following concrete steps:
S1: choose multiple compression algorithm according to data type to be compressed, principal character and/or time restriction factor, forms compression algorithm collection, and
Obtain the average decompress(ion) speed D of decompression algorithm corresponding to described multiple compression algorithm
i;
S2: analyze described demand data to be compressed, be divided into multiple data block, and
Obtain the common parameter of described computer system;
S3: calculate the compression effectiveness value that described each data block uses described multiple compression algorithm;
S4: the described compression effectiveness value of more described multiple compression algorithm, chooses the minimum algorithm of compression effectiveness value as optimal algorithm;
S5: the described optimal algorithm corresponding according to described each data block carries out data compression calculating.
2. as claimed in claim 1 a kind of based on the preferred data splitting compression algorithm of effect, it is characterized in that, described step S3 also comprises:
S31: for described multiple data block, preset data size is C
0;
S32: travel through described compression algorithm collection described multiple compression algorithm wherein and compression calculating is carried out to described each data block, and add up the size CR of compression result
i.
3. as claimed in claim 1 a kind of based on the preferred data splitting compression algorithm of effect, it is characterized in that, described step S3 also comprises:
S31 ': calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i;
S32 ': the reading total time T when data calculated in described each data block are not compressed
0and compression effectiveness value.
4. as claimed in claim 3 a kind of based on the preferred data splitting compression algorithm of effect, it is characterized in that, pass through formula:
T
i＝(CR
i/D
i)+(CR
i/V)
Calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i, wherein, the size CR of described compression result
i, the average decompression rate of often kind of algorithm in described multiple compression algorithm is D
i, the reading speed of average I/O is V.
5. as claimed in claim 3 a kind of based on the preferred data splitting compression algorithm of effect, it is characterized in that, pass through formula:
T
0＝C
0/V
Calculate reading total time T when described multiple compression algorithm is not compressed for the data in described each data block
0.
6. the one as described in claim 3 or 5, based on the preferred data splitting compression algorithm of effect, is characterized in that, passes through formula:
CEV
0＝(T
0/T
0)+C
0*DSR＝1+C
0*DSR
Calculate compression effectiveness value when described multiple compression algorithm is not compressed for the data in described each data block, wherein, described preset data size is C
0, disk coefficient of sensitivity is DSR.
7. as claimed in claim 1 a kind of based on the preferred data splitting compression algorithm of effect, it is characterized in that, described step S3 also comprises:
Pass through formula:
CEV
i＝(T
i/T
0)+CR
i*DSR
Calculate the compression effectiveness value that described each data block uses described multiple compression algorithm, wherein, calculate the total reading time T of described multiple compression algorithm for the compression result of described each data block
i, the reading total time T when data calculated in described each data block are not compressed
0, the size CR of described compression result
i, described disk coefficient of sensitivity is DSR.
8. one as claimed in claim 1 is based on the preferred data splitting compression algorithm of effect, it is characterized in that, also comprises after described step S5: record described each data block compression result, and described optimal algorithm information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410647747.6A CN104348490B (en) | 2014-11-14 | 2014-11-14 | A kind of data splitting compression method preferred based on effect |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410647747.6A CN104348490B (en) | 2014-11-14 | 2014-11-14 | A kind of data splitting compression method preferred based on effect |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104348490A true CN104348490A (en) | 2015-02-11 |
| CN104348490B CN104348490B (en) | 2017-09-19 |
Family
ID=52503430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410647747.6A Active CN104348490B (en) | 2014-11-14 | 2014-11-14 | A kind of data splitting compression method preferred based on effect |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104348490B (en) |
Cited By (15)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105630999A (en) * | 2015-12-28 | 2016-06-01 | 华为技术有限公司 | Data compressing method and device of server |
| CN105653698A (en) * | 2015-12-30 | 2016-06-08 | 北京奇艺世纪科技有限公司 | Data loading method and apparatus for database table Hive Table |
| CN106610616A (en) * | 2016-11-30 | 2017-05-03 | 交控科技股份有限公司 | Automatic performance tuning method and automatic performance tuning device based on big data analysis |
| WO2017084482A1 (en) * | 2015-11-16 | 2017-05-26 | 华为技术有限公司 | Data transmission method and device |
| CN107451266A (en) * | 2017-07-31 | 2017-12-08 | 北京京东尚科信息技术有限公司 | For processing data method and its equipment |
| WO2018036290A1 (en) * | 2016-08-26 | 2018-03-01 | 华为技术有限公司 | Data compression method and terminal |
| CN109710239A (en) * | 2018-12-29 | 2019-05-03 | 北京航天数据股份有限公司 | Industry pattern generation method and device, digital asset processing method and electronic equipment |
| WO2019114655A1 (en) * | 2017-12-14 | 2019-06-20 | 阿里巴巴集团控股有限公司 | Data compression method, electronic device and computer readable storage medium |
| CN110035299A (en) * | 2019-04-18 | 2019-07-19 | 雷欧尼斯（北京）信息技术有限公司 | The compression transmitting method and framework of immersion multi-object audio |
| CN110163367A (en) * | 2018-09-29 | 2019-08-23 | 腾讯科技（深圳）有限公司 | A kind of model compression method and apparatus |
| CN110311687A (en) * | 2019-07-09 | 2019-10-08 | 南京天数智芯科技有限公司 | A kind of time series data lossless compression method based on Integrated Algorithm |
| CN110334124A (en) * | 2018-06-15 | 2019-10-15 | 腾讯科技（深圳）有限公司 | Compression algorithm selection method, device and equipment |
| CN110535869A (en) * | 2019-09-05 | 2019-12-03 | 厦门市美亚柏科信息股份有限公司 | A kind of data transmission method based on compression algorithm, terminal device and storage medium |
| CN110704408A (en) * | 2019-09-10 | 2020-01-17 | 南京天数智芯科技有限公司 | Clustering-based time sequence data compression method and system |
| WO2022155921A1 (en) * | 2021-01-22 | 2022-07-28 | 京东方科技集团股份有限公司 | Display panel and driving method therefor, compensation data compression method, and compensation data decompression method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1251449A (en) * | 1998-10-18 | 2000-04-26 | 华强 | Combined use with reference of two category dictionary compress algorithm in data compaction |
| CN101459616A (en) * | 2009-01-04 | 2009-06-17 | 成都市华为赛门铁克科技有限公司 | Method, customer terminal and access system for network access |
| US7765346B2 (en) * | 2007-12-14 | 2010-07-27 | Bmc Software, Inc. | Dynamic compression of systems management data |
| WO2011129818A1 (en) * | 2010-04-13 | 2011-10-20 | Empire Technology Development Llc | Adaptive compression |
| CN102594360A (en) * | 2012-02-01 | 2012-07-18 | 浪潮(北京)电子信息产业有限公司 | Method and device for computer data compression |
| CN103166983A (en) * | 2011-12-09 | 2013-06-19 | 中国移动通信集团广东有限公司 | Network data optimizing method, device and network server |
-
2014
- 2014-11-14 CN CN201410647747.6A patent/CN104348490B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1251449A (en) * | 1998-10-18 | 2000-04-26 | 华强 | Combined use with reference of two category dictionary compress algorithm in data compaction |
| US7765346B2 (en) * | 2007-12-14 | 2010-07-27 | Bmc Software, Inc. | Dynamic compression of systems management data |
| CN101459616A (en) * | 2009-01-04 | 2009-06-17 | 成都市华为赛门铁克科技有限公司 | Method, customer terminal and access system for network access |
| WO2011129818A1 (en) * | 2010-04-13 | 2011-10-20 | Empire Technology Development Llc | Adaptive compression |
| CN103166983A (en) * | 2011-12-09 | 2013-06-19 | 中国移动通信集团广东有限公司 | Network data optimizing method, device and network server |
| CN102594360A (en) * | 2012-02-01 | 2012-07-18 | 浪潮(北京)电子信息产业有限公司 | Method and device for computer data compression |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10979520B2 (en) | 2015-11-16 | 2021-04-13 | Huawei Technologies Co., Ltd. | Data transmission method and apparatus |
| WO2017084482A1 (en) * | 2015-11-16 | 2017-05-26 | 华为技术有限公司 | Data transmission method and device |
| CN105630999A (en) * | 2015-12-28 | 2016-06-01 | 华为技术有限公司 | Data compressing method and device of server |
| CN105653698A (en) * | 2015-12-30 | 2016-06-08 | 北京奇艺世纪科技有限公司 | Data loading method and apparatus for database table Hive Table |
| WO2018036290A1 (en) * | 2016-08-26 | 2018-03-01 | 华为技术有限公司 | Data compression method and terminal |
| CN106610616B (en) * | 2016-11-30 | 2019-12-03 | 交控科技股份有限公司 | A kind of performance automated tuning method and device based on big data analysis |
| CN106610616A (en) * | 2016-11-30 | 2017-05-03 | 交控科技股份有限公司 | Automatic performance tuning method and automatic performance tuning device based on big data analysis |
| CN107451266A (en) * | 2017-07-31 | 2017-12-08 | 北京京东尚科信息技术有限公司 | For processing data method and its equipment |
| WO2019114655A1 (en) * | 2017-12-14 | 2019-06-20 | 阿里巴巴集团控股有限公司 | Data compression method, electronic device and computer readable storage medium |
| CN110334124B (en) * | 2018-06-15 | 2023-05-16 | 腾讯科技（深圳）有限公司 | Compression algorithm selection method, device and equipment |
| CN110334124A (en) * | 2018-06-15 | 2019-10-15 | 腾讯科技（深圳）有限公司 | Compression algorithm selection method, device and equipment |
| CN110163367A (en) * | 2018-09-29 | 2019-08-23 | 腾讯科技（深圳）有限公司 | A kind of model compression method and apparatus |
| CN110163367B (en) * | 2018-09-29 | 2023-04-07 | 腾讯科技（深圳）有限公司 | Terminal deployment method and device |
| CN109710239B (en) * | 2018-12-29 | 2022-08-16 | 北京航天数据股份有限公司 | Industrial model generation method and device, digital asset processing method and electronic equipment |
| CN109710239A (en) * | 2018-12-29 | 2019-05-03 | 北京航天数据股份有限公司 | Industry pattern generation method and device, digital asset processing method and electronic equipment |
| CN110035299A (en) * | 2019-04-18 | 2019-07-19 | 雷欧尼斯（北京）信息技术有限公司 | The compression transmitting method and framework of immersion multi-object audio |
| CN110311687A (en) * | 2019-07-09 | 2019-10-08 | 南京天数智芯科技有限公司 | A kind of time series data lossless compression method based on Integrated Algorithm |
| CN110311687B (en) * | 2019-07-09 | 2022-10-04 | 上海天数智芯半导体有限公司 | Time sequence data lossless compression method based on integration algorithm |
| CN110535869A (en) * | 2019-09-05 | 2019-12-03 | 厦门市美亚柏科信息股份有限公司 | A kind of data transmission method based on compression algorithm, terminal device and storage medium |
| CN110535869B (en) * | 2019-09-05 | 2021-11-02 | 厦门市美亚柏科信息股份有限公司 | Data transmission method based on compression algorithm, terminal equipment and storage medium |
| CN110704408A (en) * | 2019-09-10 | 2020-01-17 | 南京天数智芯科技有限公司 | Clustering-based time sequence data compression method and system |
| WO2022155921A1 (en) * | 2021-01-22 | 2022-07-28 | 京东方科技集团股份有限公司 | Display panel and driving method therefor, compensation data compression method, and compensation data decompression method |
Also Published As
| Publication number | Publication date |
|---|---|
| CN104348490B (en) | 2017-09-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104348490A (en) | Combined data compression algorithm based on effect optimization | |
| CN101241508B (en) | Structured data sequence compression method | |
| CN102970043B (en) | A kind of compression hardware system based on GZIP and accelerated method thereof | |
| CN105144157B (en) | System and method for the data in compressed data library | |
| CN102609491A (en) | Column-storage oriented area-level data compression method | |
| US20190140657A1 (en) | Data compression coding method, apparatus therefor, and program therefor | |
| CN105574212A (en) | Image retrieval method for multi-index disk Hash structure | |
| CN107273471A (en) | A kind of binary electric power time series data index structuring method based on Geohash | |
| WO2019076177A1 (en) | Gene sequencing data compression preprocessing, compression and decompression method, system, and computer-readable medium | |
| Gupta et al. | Faster as well as early measurements from big data predictive analytics model | |
| Kim et al. | SBH: Super byte-aligned hybrid bitmap compression | |
| CN105653698A (en) | Data loading method and apparatus for database table Hive Table | |
| CN105302915A (en) | High-performance data processing system based on memory calculation | |
| CN112463784A (en) | Data deduplication method, device, equipment and computer readable storage medium | |
| CN107291935B (en) | Spark and Huffman coding based CPIR-V nearest neighbor privacy protection query method | |
| CN104199821B (en) | A kind of flow data cube construction method based on Sketch | |
| CN104125475A (en) | Multi-dimensional quantum data compressing and uncompressing method and apparatus | |
| CN113078908B (en) | Simple encoding and decoding method suitable for time sequence database | |
| CN104133883B (en) | Telephone number ownership place data compression method | |
| Crume et al. | Compressing intermediate keys between mappers and reducers in scihadoop | |
| Duvignau et al. | Piecewise linear approximation in data streaming: Algorithmic implementations and experimental analysis | |
| CN104077272A (en) | Method and device for compressing dictionary | |
| CN202931290U (en) | Compression hardware system based on GZIP | |
| Yuan et al. | Parallel implementation of lossy data compression for temporal data sets | |
| CN105373583A (en) | Modeling method for support vector machine based on data compression |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |