CN116578230A - Optimization method and terminal for data compression - Google Patents

Optimization method and terminal for data compression Download PDF

Info

Publication number
CN116578230A
CN116578230A CN202310422591.0A CN202310422591A CN116578230A CN 116578230 A CN116578230 A CN 116578230A CN 202310422591 A CN202310422591 A CN 202310422591A CN 116578230 A CN116578230 A CN 116578230A
Authority
CN
China
Prior art keywords
data
written
value
calling party
compressed
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202310422591.0A
Other languages
Chinese (zh)
Inventor
刘德建
郭孟祺
郑坦
郭凤英
陈娟
陈宏�
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujian Tianquan Educational Technology Ltd
Original Assignee
Fujian Tianquan Educational Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujian Tianquan Educational Technology Ltd filed Critical Fujian Tianquan Educational Technology Ltd
Priority to CN202310422591.0A priority Critical patent/CN116578230A/en
Publication of CN116578230A publication Critical patent/CN116578230A/en
Pending legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
    • G06F3/0608—Saving storage space on storage systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0638—Organizing or formatting or addressing of data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
    • G06F3/0655—Vertical data movement, i.e. input-output transfer; data movement between one or more hosts and one or more storage devices
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
    • G06F3/0601—Interfaces specially adapted for storage systems
    • G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
    • G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
  • Compression, Expansion, Code Conversion, And Decoders (AREA)

Abstract

The invention discloses a data compression optimization method machine terminal, which comprises the following steps: s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer; s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster. By adding the proxy layer between the calling party and the Redis, the data to be read and written are compressed and processed according to different values, so that the memory of the Redis is saved, and the phenomenon of burrs in the Redis cluster caused by large key value data is avoided.

Description

Optimization method and terminal for data compression
Technical Field
The present invention relates to the field of internet systems, and in particular, to a data compression optimization method and a terminal.
Background
In order to realize the requirement, a server program uses a caching mechanism in a large amount, and the cache is used for relieving the concurrency of large-flow requests so as to realize high concurrency and high throughput. Since a large amount of cache is used, a cluster cache with a large memory system is required, and cache elements such as Redis clusters are generated. However, the use of the Redis cluster brings new problems, such as sporadic request burrs, so that the occupancy rate of memory and IO resources is instantaneously higher, and finally, the request delay phenomenon is caused. The main reason for this phenomenon is that the memory and IO resources are largely occupied due to the large key value data existing in the request process.
Disclosure of Invention
The technical problems to be solved by the invention are as follows: the optimization method and the terminal for data compression are provided, the occupancy rate of memory and IO resources is reduced, and request burrs are eliminated.
In order to solve the technical problems, the invention adopts a technical scheme that:
an optimization method for data compression, comprising the following steps:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster.
In order to solve the technical problems, the invention adopts another technical scheme that:
an optimized terminal for data compression, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the optimized method for data compression when executing the computer program.
The invention has the beneficial effects that: the invention judges whether the large key value data exists or not by adding a proxy layer, and compresses the large key value data. And aiming at the data of various special data structures in the Redis, respectively carrying out different processing, reducing the data request quantity by utilizing a flexible compression or partial compression mode, reducing the consumption of memory and IO resources by using the consumption of a proxy layer CPU, further reducing the influence of large key value data on the Redis cluster, and finally solving or relieving the request burr phenomenon.
Drawings
FIG. 1 is a flow chart of an optimization method for data compression according to an embodiment of the present invention;
fig. 2 is a diagram of an architecture of an optimized terminal for data compression according to an embodiment of the present invention.
Detailed Description
In order to describe the technical contents, the achieved objects and effects of the present invention in detail, the following description will be made with reference to the embodiments in conjunction with the accompanying drawings.
The embodiment of the invention provides a data compression optimization method, which comprises the following steps:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster. From the above description, the beneficial effects of the invention are as follows: the invention judges whether the large key value data exists or not by adding a proxy layer, and compresses the large key value data. And aiming at the data of various special data structures in the Redis, respectively carrying out different processing, reducing the data request quantity by utilizing a flexible compression or partial compression mode, reducing the consumption of memory and IO resources by using the consumption of a proxy layer CPU, further reducing the influence of large key value data on the Redis cluster, and finally solving or relieving the request burr phenomenon.
Further, the step S2 specifically includes:
when the data structure type of the data to be written is a character string, judging whether the value of the data to be written exceeds a preset threshold, if the value of the data to be written does not exceed the preset threshold, directly writing the data to be written into a Redis cluster, if the value of the data to be written exceeds the preset threshold, adopting a Huffman compression algorithm to compress the value of the data to be written, adding a compression suffix identifier after the character string is compressed, and writing the compression suffix identifier into the Redis cluster by the proxy layer.
As can be seen from the above description, for writing of the data of the string data structure type, only the data with the value exceeding the preset threshold value is compressed, so that the compression time and the compression calculation amount of the system resource are reduced.
Further, the compressing the data to be written includes:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
As can be seen from the above description, only the compressed content needs to be decompressed, and the uncompressed data is directly returned, so that the computing resources are saved and the data reading efficiency is improved.
Further, step S2 includes:
when the data structure type of the data to be written is a lookup table or a set, traversing all values in the lookup table or the set, judging whether the value of the data to be written exceeds a preset threshold, if not, directly writing the data to be written into a Redis cluster, if so, adopting a Huffman compression algorithm to compress the value of the data to be written, and after the lookup table or the set is compressed, adding a compression suffix identifier and writing the Redis cluster by the proxy layer.
As can be seen from the above description, for writing of the lookup table or the aggregate data structure type data, it is necessary to traverse all data, and then compress only the data exceeding the preset threshold, so as to reduce the compression time and the calculation amount of the system resource.
Further, the compressing the data to be written includes:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
As can be seen from the above description, only the compressed content needs to be decompressed, and the uncompressed data is directly returned, so that the computing resources are saved and the data reading efficiency is improved.
The embodiment of the invention also provides an optimized terminal for data compression, which comprises a memory, a processor and a computer program stored on the memory and capable of running on the processor, wherein the following steps are realized when the processor executes the computer program:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster. From the above description, the beneficial effects of the invention are as follows: the invention judges whether the large key value data exists or not by adding a proxy layer, and compresses the large key value data. And aiming at the data of various special data structures in the Redis, respectively carrying out different processing, reducing the data request quantity by utilizing a flexible compression or partial compression mode, reducing the consumption of memory and IO resources by using the consumption of a proxy layer CPU, further reducing the influence of large key value data on the Redis cluster, and finally solving or relieving the request burr phenomenon.
Further, the step S2 specifically includes:
when the data structure type of the data to be written is a character string, judging whether the value of the data to be written exceeds a preset threshold, if the value of the data to be written does not exceed the preset threshold, directly writing the data to be written into a Redis cluster, if the value of the data to be written exceeds the preset threshold, adopting a Huffman compression algorithm to compress the value of the data to be written, adding a compression suffix identifier after the character string is compressed, and writing the compression suffix identifier into the Redis cluster by the proxy layer.
As can be seen from the above description, for writing of the data of the string data structure type, only the data with the value exceeding the preset threshold value is compressed, so that the compression time and the compression calculation amount of the system resource are reduced.
Further, the compressing the data to be written includes:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
As can be seen from the above description, only the compressed content needs to be decompressed, and the uncompressed data is directly returned, so that the computing resources are saved and the data reading efficiency is improved.
Further, step S2 includes:
when the data structure type of the data to be written is a lookup table or a set, traversing all values in the lookup table or the set, judging whether the value of the data to be written exceeds a preset threshold, if not, directly writing the data to be written into a Redis cluster, if so, adopting a Huffman compression algorithm to compress the value of the data to be written, and after the lookup table or the set is compressed, adding a compression suffix identifier and writing the Redis cluster by the proxy layer.
As can be seen from the above description, for writing of the lookup table or the aggregate data structure type data, it is necessary to traverse all data, and then compress only the data exceeding the preset threshold, so as to reduce the compression time and the calculation amount of the system resource.
Further, the compressing the data to be written includes:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
As can be seen from the above description, only the compressed content needs to be decompressed, and the uncompressed data is directly returned, so that the computing resources are saved and the data reading efficiency is improved.
The optimization method and the terminal for data compression can reduce the occupancy rate of memory and IO resources and eliminate request burrs, and are described in the following specific embodiments:
example 1
Referring to fig. 1, an optimization method for data compression includes the following steps:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
specifically, the check and compression work of the data is performed by proxy, and thereafter, the Redis is operated again by client.
S2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written into the Redis cluster.
Specifically, when the data structure type is a character string, judging whether the value exceeds a preset threshold, if the value does not exceed the preset threshold, directly writing the data to be written into the Redis cluster, if the value exceeds the preset threshold, adopting a Huffman compression algorithm to compress the data to be written, and adding a compression suffix identifier after compressing the processed character string, and writing the compressed suffix identifier into the cluster by the proxy layer.
When reading data with the data structure type of character string (string), receiving a data reading request of a calling party through an agent layer, reading the data from a Redis cluster, judging whether a value has a compressed suffix identification or not, and if the value does not have the compressed suffix identification, directly returning the data to the calling party; if the compressed suffix identifier exists, decompression processing of a Huffman compression algorithm is carried out on the data to be read, and the decompressed data is returned to the calling party.
When the data structure type is a lookup table (map) or a set (list), traversing all data, judging whether the value exceeds a preset threshold, if not, directly writing the data to be written into a Redis cluster, if so, adopting a Huffman compression algorithm to compress the data to be written, and adding a compression suffix identification after compressing the character string, and writing the character string into the cluster by a proxy layer.
When reading data with the data structure type of a lookup table or a set, receiving a data reading request of a calling party through an agent layer, reading the data from a Redis cluster, judging whether a value has a compressed suffix identification, and if the value does not have the compressed suffix identification, directly returning the data to the calling party; if the compressed suffix identifier exists, decompression processing of a Huffman compression algorithm is carried out on the data to be read, and the decompressed data is returned to the calling party.
Through the scheme, the memory and IO consumption of the Redis cluster machine can be replaced by the CPU consumption of the agent layer machine. Compared with full data compression, the compression time is reduced, the transmission and storage of data volume are also reduced, and therefore the memory of Redis is saved, and the burr phenomenon in the Redis cluster is also reduced.
Example two
Referring to fig. 2, an optimization terminal for data compression includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements each step in the above-mentioned optimization method for data compression when executing the computer program.
In summary, according to the data compression optimization method and terminal provided by the invention, the proxy layer is added between the calling party and the Redis, and the data to be read and written are compressed and processed according to different value values, so that the occupancy rate of memory and IO resources is reduced, and the request burrs are eliminated.
It should be noted that, for the sake of simplicity of description, the foregoing method embodiments are all expressed as a series of combinations of actions, but it should be understood by those skilled in the art that the present invention is not limited by the order of actions described, as some steps may be performed in other order or simultaneously in accordance with the present invention. Further, those skilled in the art will appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily all required for the present invention.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to the related descriptions of other embodiments.
The foregoing description is only illustrative of the present invention and is not intended to limit the scope of the invention, and all equivalent structures or equivalent processes or direct or indirect application in other related technical fields are included in the scope of the present invention.

Claims (10)

1. An optimization method for data compression, comprising the steps of:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster.
2. The method for optimizing data compression according to claim 1, wherein step S2 is specifically:
when the data structure type of the data to be written is a character string, judging whether the value of the data to be written exceeds a preset threshold, if the value of the data to be written does not exceed the preset threshold, directly writing the data to be written into a Redis cluster, if the value of the data to be written exceeds the preset threshold, adopting a Huffman compression algorithm to compress the value of the data to be written, adding a compression suffix identifier after the character string is compressed, and writing the compression suffix identifier into the Redis cluster by the proxy layer.
3. The method for optimizing data compression according to claim 2, wherein the compressing the data to be written comprises:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
4. The method for optimizing data compression according to claim 1, wherein step S2 comprises:
when the data structure type of the data to be written is a lookup table or a set, traversing all values in the lookup table or the set, judging whether the value of the data to be written exceeds a preset threshold, if not, directly writing the data to be written into a Redis cluster, if so, adopting a Huffman compression algorithm to compress the value of the data to be written, and after the lookup table or the set is compressed, adding a compression suffix identifier and writing the Redis cluster by the proxy layer.
5. The method for optimizing data compression according to claim 4, wherein the compressing the data to be written comprises:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
6. An optimized terminal for data compression, comprising a memory, a processor and a computer program stored on said memory and executable on the processor, characterized in that said processor implements the following steps when executing said computer program:
s1, setting a proxy layer between a calling party and a Redis cluster, and receiving a data writing request of the calling party through the proxy layer;
s2, judging whether the value of the data to be written in the data writing request exceeds a preset threshold value, if so, compressing the data to be written, and writing the compressed data to be written in a Redis cluster.
7. The optimized terminal for data compression according to claim 6, wherein step S2 is specifically:
when the data structure type of the data to be written is a character string, judging whether the value of the data to be written exceeds a preset threshold, if the value of the data to be written does not exceed the preset threshold, directly writing the data to be written into a Redis cluster, if the value of the data to be written exceeds the preset threshold, adopting a Huffman compression algorithm to compress the value of the data to be written, adding a compression suffix identifier after the character string is compressed, and writing the compression suffix identifier into the Redis cluster by the proxy layer.
8. The optimized terminal for data compression according to claim 7, wherein said compressing the data to be written, after which comprises:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
9. The optimized terminal for data compression according to claim 6, wherein step S2 comprises:
when the data structure type of the data to be written is a lookup table or a set, traversing all values in the lookup table or the set, judging whether the value of the data to be written exceeds a preset threshold, if not, directly writing the data to be written into a Redis cluster, if so, adopting a Huffman compression algorithm to compress the value of the data to be written, and after the lookup table or the set is compressed, adding a compression suffix identifier and writing the Redis cluster by the proxy layer.
10. The optimized terminal for data compression according to claim 9, wherein said compressing the data to be written, after which comprises:
receiving a data reading request of the calling party through the proxy layer, reading data from a Redis cluster, judging whether a value of the data to be read has a compressed suffix identification, and if the value of the data to be read does not have the compressed suffix identification, directly returning the data to be read to the calling party; if the compressed suffix identifier exists, decompressing the data to be read by a Huffman compression algorithm, and returning the decompressed data to a calling party.
CN202310422591.0A 2023-04-19 2023-04-19 Optimization method and terminal for data compression Pending CN116578230A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202310422591.0A CN116578230A (en) 2023-04-19 2023-04-19 Optimization method and terminal for data compression

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202310422591.0A CN116578230A (en) 2023-04-19 2023-04-19 Optimization method and terminal for data compression

Publications (1)

Publication Number Publication Date
CN116578230A true CN116578230A (en) 2023-08-11

Family

ID=87542257

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202310422591.0A Pending CN116578230A (en) 2023-04-19 2023-04-19 Optimization method and terminal for data compression

Country Status (1)

Country Link
CN (1) CN116578230A (en)

Similar Documents

Publication Publication Date Title
JP4831785B2 (en) Adaptive session compression management method, compression manager, and session management system
US7181457B2 (en) System and method for utilizing compression in database caches to facilitate access to database information
US20130238574A1 (en) Cloud system and file compression and transmission method in a cloud system
US10649905B2 (en) Method and apparatus for storing data
US9836248B2 (en) In-memory data compression complementary to host data compression
WO2023279964A1 (en) Data compression method and apparatus, and computing device and storage medium
CN111857550A (en) Method, apparatus and computer readable medium for data deduplication
CN113296709B (en) Method and apparatus for deduplication
CN111723057A (en) File pre-reading method, device, equipment and storage medium
US20180270320A1 (en) Enhanced storage clients
CN113300715B (en) Data processing method, device, hardware compression equipment and medium
EP3848815B1 (en) Efficient shared bulk loading into optimized storage
CN111949681A (en) Data aggregation processing device and method and storage medium
CN116578230A (en) Optimization method and terminal for data compression
CN114065704A (en) Data compression method, electronic device and computer program product
US20220357980A1 (en) Selectively offloading the compression and decompression of files to a hardware controller
CN114282141A (en) Processing method and device for compression format data, electronic equipment and readable storage medium
WO2021253907A1 (en) Small-memory embedded device-based method and terminal for accelerating png display speed
CN114780501A (en) Data processing method, electronic device and computer program product
US9160820B2 (en) Large volume data transfer
CN113220981A (en) Method and device for optimizing cache
CN111352897A (en) Real-time data storage method, equipment and storage medium
Kato et al. Performance Enhancement of Stream-Based Decompression Process by Notifying Compression Buffer Size
US11435956B2 (en) Method, electronic device, and computer program product for data compression
US11966597B1 (en) Multi-domain configurable data compressor/de-compressor

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination