CN117857391A - Dynamic storage method and system based on CPU load and data packet compression efficiency - Google Patents

Dynamic storage method and system based on CPU load and data packet compression efficiency Download PDF

Info

Publication number
CN117857391A
CN117857391A CN202410095892.1A CN202410095892A CN117857391A CN 117857391 A CN117857391 A CN 117857391A CN 202410095892 A CN202410095892 A CN 202410095892A CN 117857391 A CN117857391 A CN 117857391A
Authority
CN
China
Prior art keywords
compression
cpu utilization
utilization rate
cpu
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN202410095892.1A
Other languages
Chinese (zh)
Inventor
吴颖
张广兴
姜海洋
田利荣
王嘉
廖志元
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Jiangsu Future Networks Innovation Institute
Original Assignee
Jiangsu Future Networks Innovation Institute
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Jiangsu Future Networks Innovation Institute filed Critical Jiangsu Future Networks Innovation Institute
Priority to CN202410095892.1A priority Critical patent/CN117857391A/en
Publication of CN117857391A publication Critical patent/CN117857391A/en
Pending legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/16—Threshold monitoring
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/50—Testing arrangements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L69/00—Network arrangements, protocols or services independent of the application payload and not provided for in the other groups of this subclass
    • H04L69/04—Protocols for data compression, e.g. ROHC

Landscapes

  • Engineering & Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • Environmental & Geological Engineering (AREA)
  • Signal Processing For Digital Recording And Reproducing (AREA)

Abstract

The invention relates to a dynamic storage method and a system based on CPU load and data packet compression efficiency. The method comprises the following steps: s1, loading an initial compression algorithm, a compression level of the compression algorithm, parameters of IO operation and a CPU utilization threshold; s2, starting a scanning thread, and collecting CPU utilization rate according to a preset time interval; s3, judging whether the CPU utilization rate collected in each time interval exceeds a CPU utilization rate threshold, and if the collected CPU utilization rate is higher than or equal to a CPU load threshold, reducing the compression level of the current compression algorithm; if the collected CPU utilization rate is lower than the CPU load threshold value, the CPU utilization rate is unchanged; s4, finishing data storage. According to the method, the compression strategy of the data packet is dynamically adjusted by monitoring the CPU load in real time, so that the balance between the storage efficiency and the CPU load is realized, and the overall performance is improved.

Description

Dynamic storage method and system based on CPU load and data packet compression efficiency
Technical Field
The invention relates to the technical field of computers, in particular to a dynamic storage method and a system based on CPU load and data packet compression efficiency.
Background
With the rapid development of network technology, full-flow storage systems face IO performance challenges when processing mass data. In conventional storage systems, in order to improve storage efficiency, the landing data files are typically compressed. However, the conventional compression method generally adopts a fixed compression strategy, and cannot be dynamically adjusted according to the CPU load and the specific situation of the data packet. When the load of the CPU is too high, the compression process occupies a large amount of CPU resources, so that the overall performance of the system is reduced. In addition, the conventional IO operation mode also has the problem of low efficiency, and cannot meet the requirement of large-scale data storage.
To solve these problems, researchers have proposed various optimization methods. One common approach is to employ asynchronous compression techniques. This technique can compress data in the background, thereby relieving the load of the CPU. However, asynchronous compression techniques may cause data latency and consistency issues. In addition, asynchronous compression techniques cannot be dynamically adjusted according to CPU load and packet specifics. Another common approach is to employ caching techniques. This technique may improve IO efficiency by caching a portion of the data. However, the cache technology does not fundamentally solve the problem of excessive CPU load. In addition, the cache technology also needs to consume certain memory resources, which may cause waste of system resources.
Therefore, how to reduce the CPU load while ensuring the storage efficiency is a problem to be solved.
Disclosure of Invention
The invention aims to provide a dynamic storage method and a system based on CPU load and data packet compression efficiency. According to the method, the compression strategy of the data packet is dynamically adjusted and IO operation is optimized by monitoring the CPU load and IO performance in real time, so that balance between storage efficiency and the CPU load is realized, and the overall performance is improved.
The solution of the invention is as follows:
a dynamic storage method based on CPU load and data packet compression efficiency comprises the following steps:
s1, loading an initial compression algorithm, a compression level of the compression algorithm, parameters of IO operation and a CPU utilization threshold;
s2, starting a scanning thread, and collecting CPU utilization rate according to a preset time interval;
s3, judging whether the CPU utilization rate collected in each time interval exceeds a CPU utilization rate threshold, and if the collected CPU utilization rate is higher than or equal to a CPU load threshold, reducing the compression level of the current compression algorithm; if the collected CPU utilization rate is lower than the CPU load threshold value, the CPU utilization rate is unchanged;
s4, finishing data storage;
the compression level of the compression algorithm is divided according to the compression rate of the compression algorithm, and the compression level of the initial compression algorithm is selected from the compression algorithm at a medium compression rate.
Further, in step S4, before the data storage, the method further includes adjusting the parameters of the IO operation according to the compression level of the compression algorithm reduced in step S3.
Further, in step S1, loading an initial disk write rate is further included; in step S2, the method further includes collecting the disk write rate at preset time intervals; in step S3, if the collected CPU utilization is lower than the CPU utilization threshold, the method further includes determining whether the disk write rate exceeds the disk write rate threshold, and according to the determination result, readjusting the compression level of the compression algorithm.
Further, in step S3, according to the determination result, readjusting the compression level of the compression algorithm includes: if the collected disk writing speed is lower than the disk writing speed threshold, the compression level of the compression algorithm is improved; if the collected disk write rate is equal to or higher than the disk write rate threshold, the current compression algorithm compression level is maintained.
Further, in step S4, before the data storage, the method further includes adjusting parameters of the IO operation according to the compression level of the compression algorithm reduced in step S3, including: if the compression level of the compression algorithm is reduced, increasing the data volume of each IO operation; if the compression level of the compression algorithm is increased, the data volume of each IO operation is reduced.
Further, after step S4, step S5 is further included, where it is determined whether the CPU utilization rate or the disk write rate exceeds the current CPU utilization rate threshold or the current disk write rate threshold in a plurality of consecutive monitoring periods, and if yes, the current CPU utilization rate threshold or the current disk write rate threshold is increased; if the CPU utilization rate or the disk writing rate is lower than the initial CPU utilization rate threshold value or the initial disk writing rate threshold value in a plurality of continuous monitoring periods, recovering the initial CPU utilization rate threshold value or the initial disk writing rate threshold value; if the CPU utilization rate or the disk writing rate is lower than or equal to the current CPU utilization rate threshold or the current disk writing rate threshold in any one period in a plurality of continuous monitoring periods, the current CPU utilization rate threshold or the initial disk writing rate threshold is maintained.
Further, if the CPU utilization rate is lower than the initial CPU utilization rate threshold value in a plurality of continuous monitoring periods, the method comprises switching to a deflate compression algorithm and an intermediate level 5 compression level of the deflate compression algorithm in addition to recovering the initial CPU utilization rate threshold value; if the CPU utilization rate exceeds the current CPU utilization rate threshold value in a plurality of continuous monitoring periods, switching to an initial compression algorithm and an initial compression algorithm compression level in addition to improving the current CPU utilization rate threshold value; if the CPU utilization rate is lower than or equal to the current CPU utilization rate threshold value in any period of a plurality of continuous monitoring periods, the current compression algorithm is maintained.
The "parameters of the IO operation" is the IO size stored.
The invention also provides a dynamic storage system based on the CPU load and the data packet compression efficiency: the system comprises an initialization module, a packet receiving module, a data processing module, a data compression module, an IO operation module and a monitoring module;
the initialization module is used for initializing the system, and storing and running default initial parameters of the system, including a compression algorithm, compression level of the compression algorithm, parameters of IO operation, monitoring period, threshold adjustment period, CPU utilization rate threshold and disk writing rate threshold;
the packet receiving module is used for receiving a network data packet from the network card;
the data processing module is used for data processing, including data packet decoding, classification and index creation;
the data compression module is used for compressing the data packet and is completed through a compression algorithm and a compression level;
the IO operation module is used for carrying out IO operation on the compressed data, completing the disc-falling work of the data and dynamically adjusting the parameters of the IO operation through the output of the data compression module;
the monitoring module is used for collecting the CPU utilization rate and the disk writing rate according to a preset time interval, comparing the CPU utilization rate and the disk writing rate with a threshold value, transmitting the judging result to the data compression module to complete compression, simultaneously judging whether the CPU utilization rate exceeds the threshold value again in a plurality of continuous monitoring periods, and transmitting the judging result to the data compression module according to the judging result.
The present invention also provides a computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform a method as described above.
The invention also provides a computer program product comprising instructions which, when run on a computer, cause the computer to perform a method as described above.
Compared with the prior art, the invention has the remarkable advantages that:
(1) the dynamic performance optimization, the invention can dynamically adjust compression parameters and algorithms by monitoring CPU service conditions and disk IO performance indexes in real time so as to adapt to the change of system load, thereby optimizing the overall performance;
(2) the invention can adaptively set the thresholds of CPU load and IO performance according to historical data and current workload, thereby avoiding frequent compression strategy adjustment and improving the stability and efficiency of the system;
(3) the invention can dynamically switch different compression algorithms according to the change of CPU load and IO performance and the preset algorithm, thereby not only ensuring the data compression rate, but also avoiding excessive consumption of CPU resources;
(4) IO operation optimization, the invention can optimize IO operation times, reduce unnecessary disk operation and improve IO performance of the system by dynamically adjusting the size of the IO block;
(5) the method has high flexibility, the implementation method of the invention can be adjusted and optimized according to different system environments and requirements,
the method has higher flexibility and can be suitable for various application scenes;
(6) the implementation method is easy to realize, is easy to realize and maintain based on the prior art and tools, and reduces the development cost and difficulty;
(7) the design of the invention has good expansibility, and can conveniently add new monitoring indexes, compression algorithms and optimization strategies to meet the continuously changing application demands.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic flow chart of the method of embodiment 1 of the present invention.
Fig. 2 is a detailed illustration of method steps S1 to S5 in embodiment 1 of the invention.
Fig. 3 is a schematic diagram of a dynamic storage system based on CPU load and packet compression efficiency according to embodiment 2 of the present invention.
Detailed Description
In order that the above-recited objects, features and advantages of the present invention will become more readily apparent, a more particular description of the invention will be rendered by reference to specific embodiments thereof which are illustrated in the appended drawings.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present invention, but the present invention may be practiced in other ways other than those described herein, and persons skilled in the art will readily appreciate that the present invention is not limited to the specific embodiments disclosed below.
In the present invention, the "CPU load condition" is determined using the "CPU utilization", and both represent the same meaning. The disk IO performance index is judged by using the disk writing rate index, and the two indexes have the same meaning.
Example 1
In this embodiment, the compression algorithm includes lz4 and deflate. The compression algorithm lz4 has 1-9 levels, 1 representing the fastest compression rate but the lowest compression rate and 9 representing the highest compression rate but the slowest compression rate, so level 5 is selected among the initial default compression levels. The deflate has 1-9 levels, 1 representing the fastest compression rate but the lowest compression rate, 9 representing the highest compression rate but the slowest compression rate, and the initial level is 5.
As shown in fig. 1 and 2, a dynamic storage method based on CPU load and data packet compression efficiency includes the following steps:
s1, loading an initial compression algorithm, a compression level of the compression algorithm, parameters of IO operation and a CPU utilization threshold; the compression level of the compression algorithm is divided according to the compression rate of the compression algorithm, and the compression level of the initial compression algorithm is selected from the compression algorithm at a medium compression rate.
As a specific example, in step S1, an initial compression algorithm lz4 (default lz 4), deflate, compression level (default level 5), IO block size (32 k), monitor period (every 5S), threshold adjustment period (5 monitor periods), CPU usage 80%, and disk write rate 500MB/S are loaded. These default values can be configured and adjusted according to actual requirements. Based on historical data and traffic demand, appropriate cpu usage thresholds (e.g., over 80%) and disk write rate thresholds (e.g., below 500 MB/s) are set.
S2, starting a scanning thread, and collecting CPU utilization rate according to a preset time interval, wherein the method further comprises the step of collecting disk writing rate according to the preset time interval as a specific embodiment. The scanning thread can start one or more at the same time, and the preset time interval (set to every 5s in this embodiment) periodically collects the CPU utilization and the disk writing rate.
S3, judging whether the CPU utilization rate collected in each time interval exceeds a CPU utilization rate threshold, and if the collected CPU utilization rate is higher than or equal to a CPU load threshold, reducing the compression level of the current compression algorithm; and if the collected CPU utilization rate is lower than the CPU load threshold value, the CPU utilization rate is unchanged. As a specific embodiment, if the collected CPU utilization is lower than the CPU utilization threshold, further comprising determining whether the disk write rate exceeds the disk write rate threshold (disk write rate 500 MB/s), and according to the determination result, readjusting the compression level of the compression algorithm, specifically: if the collected disk writing rate is lower than a threshold value (500 MB/s), the compression level of the compression algorithm is improved; if the collected disk write rate is equal to or higher than the disk write rate threshold, the current compression algorithm compression level is maintained.
When the CPU usage exceeds 80%, the compression level of the compression algorithm is lowered from the currently used compression level one step at a time until the CPU usage is lower than the threshold or lowered to 1 level, so as to relieve the load of the CPU. Under the condition that the CPU load is normal (below the threshold), judging whether the IO performance (disk writing rate) is lower than the initial threshold of 500MB/s, and when the IO performance (disk writing rate) is lower than the initial threshold, gradually increasing the compression level of the compression algorithm from the currently used compression level until the IO writing rate threshold is restored or increased to 9 levels so as to lighten the CPU load and improve the IO performance. When the CPU load is higher than the threshold value, the compression level is preferentially reduced so as to lighten the CPU load, and when the CPU load is reduced below the threshold value, the compression level is adjusted according to the disk writing rate (IO performance), so that the condition that the CPU load is extremely high and high-strength compression is simultaneously carried out can be avoided, and the system performance is sharply reduced.
S4, finishing data storage, wherein the method further comprises the step of adjusting parameters of IO operation according to the compression level of the compression algorithm reduced in the step S3 before the data storage, specifically: if the compression level of the compression algorithm is reduced, increasing the data volume of each IO operation; if the compression level of the compression algorithm is increased, the data volume of each IO operation is reduced.
The purpose of the above operation is to: (1) When the compression level decreases, that is, the compression rate decreases, it means that the space occupied by the same amount of data after compression becomes larger, and in order to reduce the number of IO operations, the amount of data per IO operation may be increased, specifically by increasing the size of the IO block, so that more data is read or written in one IO operation. In this embodiment, the compression level is below 5 based on a default level of 5, block size of 32k, and the block sizes are correspondingly adjusted to 64k,128k,256k,512k. (2) When the compression level is increased, that is, when the compression rate is increased, it means that the occupied space of the compressed data is reduced, in order to control the data more finely, the data amount of each IO operation can be reduced, specifically, by gradually reducing the size of the IO block, so that the system can perform the IO operation more frequently, thereby adapting to data storage and reading under high compression rate. In this embodiment, the default level 5, the block size 32k, and the compression level above 5, correspondingly adjust the block sizes to 16k,8k,4k, and 2k.
After the step S4, the method further comprises a step S5, wherein whether the CPU utilization rate or the disk writing rate exceeds the current CPU utilization rate threshold or the current disk writing rate threshold in a plurality of continuous monitoring periods is judged, and if yes, the current CPU utilization rate threshold or the current disk writing rate threshold is improved; if the CPU utilization rate or the disk writing rate is lower than the initial CPU utilization rate threshold value or the initial disk writing rate threshold value in a plurality of continuous monitoring periods, recovering the initial CPU utilization rate threshold value or the initial disk writing rate threshold value; if the CPU utilization rate or the disk writing rate is lower than or equal to the current CPU utilization rate threshold or the current disk writing rate threshold in any one period in a plurality of continuous monitoring periods, the current CPU utilization rate threshold or the initial disk writing rate threshold is maintained.
Further, the method further comprises the following steps: if the CPU utilization rate is lower than the initial CPU utilization rate threshold value in a plurality of continuous monitoring periods, switching to a deflate compression algorithm and an intermediate level 5 of the deflate compression algorithm in addition to recovering the initial CPU utilization rate threshold value; if the CPU utilization rate exceeds the current CPU utilization rate threshold value in a plurality of continuous monitoring periods, switching to an initial compression algorithm and an initial compression algorithm compression level in addition to improving the current CPU utilization rate threshold value; if the CPU utilization rate is lower than or equal to the current CPU utilization rate threshold value in any period of a plurality of continuous monitoring periods, the current compression algorithm is maintained.
When the CPU usage rate or the disk writing rate reaches the threshold value for 5 continuous monitoring periods, the threshold value is adjusted to avoid frequent compression strategy adjustment, the CPU usage rate or the disk writing rate is lifted by 1% on the basis of the initial threshold value each time until the upper limit of the system is reached, and the initial threshold value is saved for recovery. According to the CPU load condition (CPU utilization), different compression algorithms are automatically switched. The balance between the storage efficiency and the IO performance is conveniently realized through the switching of a compression algorithm lz4 with smaller calculation amount and a deflate with more calculation amount.
In this embodiment, the initial thresholds of the CPU utilization and the disk writing rate are loaded first, and if the initial thresholds are adjusted upward in consecutive monitoring periods (threshold adjustment periods) in step S5, the adjusted up thresholds are used to make a determination in the next running period.
Through implementation of the steps, the dynamic storage strategy of the flow storage system based on the CPU load and the data packet compression efficiency can effectively realize balance between the storage efficiency and the CPU load, so that the overall performance is improved.
Example 2
As shown in fig. 3, the dynamic storage system based on CPU load and data packet compression efficiency includes an initialization module, a packet receiving module, a data processing module, a data compression module, an IO operation module, and a monitoring module;
the initialization module is used for initializing the system, and storing and running default parameter values of the system, including compression algorithm, compression level, parameters of IO operation, monitoring period, threshold adjustment period, CPU usage threshold and disk writing rate threshold.
And the packet receiving module is used for receiving the network data packet from the network card.
And the data processing module is used for data processing and comprises data packet decoding, classification and index creation.
And the data compression module is used for compressing the data packet, and the data packet is compressed through a compression algorithm and a compression level.
And the IO operation module is used for carrying out IO operation on the compressed data, completing the disc-falling work of the data, and dynamically adjusting parameters of the IO operation, such as the size of a block, through the output of the data compression module so as to improve the IO performance.
The monitoring module is used for collecting the CPU utilization rate and the disk writing rate according to a preset time interval, comparing the CPU utilization rate and the disk writing rate with a threshold value, transmitting the judging result to the data compression module to complete compression, simultaneously judging whether the current CPU utilization rate or the current disk writing rate exceeds the threshold value again in a plurality of continuous monitoring periods, and transmitting the judging result to the data compression module according to the judging result.
As shown in fig. 3, the system firstly initializes the system by an initialization module, loads an initial compression algorithm, compression parameters, an IO block size and a CPU utilization rate;
the packet receiving module receives a network data packet from the network card;
and the data processing module is used for processing the received network data and comprises data packet decoding, classification and index creation.
Meanwhile, the monitoring module starts a scanning thread, collects CPU utilization rate and disk writing rate according to preset time intervals (monitoring periods), compares the CPU utilization rate and the disk writing rate with a threshold value, transmits a judging result to the data compression module to complete compression, and simultaneously judges whether the current CPU utilization rate or the current disk writing rate exceeds the threshold value again in a plurality of continuous monitoring periods (threshold value adjusting periods) and transmits the judging result to the data compression module according to the judging result.
Specifically, the monitoring module compares the collected CPU load condition (CPU utilization rate) with a threshold value, and reduces the compression level of the first-level compression algorithm each time if the collected CPU load condition is higher than or equal to the threshold value; if the collected CPU load condition is lower than the CPU load threshold, judging the collected disk writing rate and the threshold, and adjusting the compression algorithm level again according to the judging result, wherein the method specifically comprises the following steps: if the collected disk writing rate is lower than the threshold value, the compression level of the compression algorithm is improved; if the collected disk write rate is equal to or higher than the threshold, the current compression algorithm compression level is maintained. And then, the monitoring module sends the determined result to the compression module.
Further, the monitoring module also judges whether the CPU utilization rate or the disk writing rate exceeds the current CPU utilization rate threshold or the current disk writing rate threshold in a plurality of continuous monitoring periods (threshold adjustment period), if so, the current CPU utilization rate threshold or the current disk writing rate threshold is improved; if the CPU utilization rate or the disk writing rate is lower than the initial CPU utilization rate threshold value or the initial disk writing rate threshold value in a plurality of continuous monitoring periods, recovering the initial CPU utilization rate threshold value or the initial disk writing rate threshold value; if the CPU utilization rate or the disk writing rate is lower than or equal to the current CPU utilization rate threshold value or the current disk writing rate threshold value in any one period of a plurality of continuous monitoring periods, the current threshold value is maintained.
If the CPU utilization rate is lower than the initial CPU utilization rate threshold value in a plurality of continuous monitoring periods, the monitoring module sends an instruction comprising a switching algorithm to the data compression module besides recovering the initial CPU utilization rate threshold value; if the CPU utilization rate exceeds the current CPU utilization rate threshold value in a plurality of continuous monitoring periods, the CPU utilization rate is increased, and the CPU utilization rate control method further comprises a switching algorithm instruction to the data compression module; if the CPU utilization rate is lower than or equal to the current CPU utilization rate threshold value in any period of a plurality of continuous monitoring periods, the switching algorithm instruction is not sent.
And the compression module compresses the data packet of the landing disc according to the judgment result fed back by the monitoring module, and correspondingly compresses the algorithm and adjusts the compression level according to the judgment result fed back by the monitoring module, thereby completing the compression of the data packet.
And the IO operation module receives the IO operation for compressing the data, completes the disc-falling work of the data, and dynamically adjusts the parameters of the IO operation, such as the size of a block, through the output of the data compression module so as to improve the IO performance. The method specifically comprises the following steps: when the compression level is reduced, namely the compression rate is reduced, increasing the data amount of each IO operation, and reading or writing more data in one IO operation; when the compression level is increased, namely when the compression rate is increased, the data volume of each IO operation is reduced, namely by gradually reducing the IO block size, so that the system can perform IO operations more frequently, and the data storage and reading under high compression rate are suitable.
Correspondingly, the invention further provides a computer readable storage medium. The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied essentially or in a part contributing to the prior art or in a part of the technical solution, in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the method according to the embodiment of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
More specific examples (a non-exhaustive list) of the computer-readable storage medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

Claims (11)

1. A dynamic storage method based on CPU load and data packet compression efficiency is characterized in that: the method comprises the following steps:
s1, loading an initial compression algorithm, a compression level of the compression algorithm, parameters of IO operation and a CPU utilization threshold;
s2, starting a scanning thread, and collecting CPU utilization rate according to a preset time interval;
s3, judging whether the CPU utilization rate collected in each time interval exceeds a CPU utilization rate threshold, and if the collected CPU utilization rate is higher than or equal to a CPU load threshold, reducing the compression level of the current compression algorithm; if the collected CPU utilization rate is lower than the CPU load threshold value, the CPU utilization rate is unchanged;
s4, finishing data storage;
the compression level of the compression algorithm is divided according to the compression rate of the compression algorithm, and the compression level of the initial compression algorithm is selected from the compression algorithm at a medium compression rate.
2. The dynamic storage method based on CPU load and packet compression efficiency according to claim 1, wherein: in step S4, before the data storage, the method further includes adjusting parameters of the IO operation according to the compression level of the compression algorithm reduced in step S3.
3. The dynamic storage method based on CPU load and packet compression efficiency according to claim 2, wherein: in step S1, the method further comprises loading an initial disk writing rate; in step S2, the method further includes collecting the disk write rate at preset time intervals; in step S3, if the collected CPU utilization is lower than the CPU utilization threshold, the method further includes determining whether the disk write rate exceeds the disk write rate threshold, and according to the determination result, readjusting the compression level of the compression algorithm.
4. The dynamic storage method based on CPU load and packet compression efficiency according to claim 3, wherein: in step S3, the readjusting the compression level of the compression algorithm according to the determination result includes: if the collected disk writing speed is lower than the disk writing speed threshold, the compression level of the compression algorithm is improved; if the collected disk write rate is equal to or higher than the disk write rate threshold, the current compression algorithm compression level is maintained.
5. The dynamic storage method based on CPU load and packet compression efficiency according to claim 3 or 4, wherein: in step S4, before the data storage, the method further includes adjusting parameters of the IO operation according to the compression level of the compression algorithm reduced in step S3, including: if the compression level of the compression algorithm is reduced, increasing the data volume of each IO operation; if the compression level of the compression algorithm is increased, the data volume of each IO operation is reduced.
6. The dynamic storage method based on CPU load and packet compression efficiency according to claim 2, wherein: after the step S4, the method further comprises a step S5, wherein whether the CPU utilization rate or the disk writing rate exceeds the current CPU utilization rate threshold or the current disk writing rate threshold in a plurality of continuous monitoring periods is judged, and if yes, the current CPU utilization rate threshold or the current disk writing rate threshold is improved; if the CPU utilization rate or the disk writing rate is lower than the initial CPU utilization rate threshold value or the initial disk writing rate threshold value in a plurality of continuous monitoring periods, recovering the initial CPU utilization rate threshold value or the initial disk writing rate threshold value; if the CPU utilization rate or the disk writing rate is lower than or equal to the current CPU utilization rate threshold or the current disk writing rate threshold in any one period in a plurality of continuous monitoring periods, the current CPU utilization rate threshold or the initial disk writing rate threshold is maintained.
7. The dynamic storage method based on CPU load and packet compression efficiency according to claim 6, wherein: if the CPU utilization rate is lower than the initial CPU utilization rate threshold value in a plurality of continuous monitoring periods, switching to a deflate compression algorithm and an initial compression algorithm compression level in addition to recovering the initial CPU utilization rate threshold value; if the CPU utilization rate exceeds the current CPU utilization rate threshold value in a plurality of continuous monitoring periods, switching to an initial compression algorithm and an initial compression algorithm compression level in addition to improving the current CPU utilization rate threshold value; if the CPU utilization rate is lower than or equal to the current CPU utilization rate threshold value in any period of a plurality of continuous monitoring periods, the current compression algorithm is maintained.
8. The dynamic storage method based on CPU load and packet compression efficiency according to any one of claims 1 to 7, wherein: the parameter of the IO operation is the stored IO size.
9. A dynamic storage system based on CPU load and data packet compression efficiency is characterized in that: the system comprises an initialization module, a packet receiving module, a data processing module, a data compression module, an IO operation module and a monitoring module;
the initialization module is used for initializing the system, and storing and running default initial parameters of the system, including a compression algorithm, compression level of the compression algorithm, parameters of IO operation, monitoring period, threshold adjustment period, CPU utilization rate threshold and disk writing rate threshold;
the packet receiving module is used for receiving a network data packet from the network card;
the data processing module is used for data processing, including data packet decoding, classification and index creation;
the data compression module is used for compressing the data packet and is completed through a compression algorithm and a compression level;
the IO operation module is used for carrying out IO operation on the compressed data, completing the disc-falling work of the data and dynamically adjusting the parameters of the IO operation through the output of the data compression module;
the monitoring module is used for collecting the CPU utilization rate and the disk writing rate according to a preset time interval, comparing the CPU utilization rate and the disk writing rate with a threshold value, transmitting the judging result to the data compression module to complete compression, simultaneously judging whether the current CPU utilization rate or the current disk writing rate exceeds the threshold value again in a plurality of continuous monitoring periods, and transmitting the judging result to the data compression module according to the judging result.
10. A computer readable storage medium having instructions stored therein which, when run on a computer, cause the computer to perform the method of any of claims 1-8.
11. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any of claims 1-8.
CN202410095892.1A 2024-01-24 2024-01-24 Dynamic storage method and system based on CPU load and data packet compression efficiency Pending CN117857391A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202410095892.1A CN117857391A (en) 2024-01-24 2024-01-24 Dynamic storage method and system based on CPU load and data packet compression efficiency

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202410095892.1A CN117857391A (en) 2024-01-24 2024-01-24 Dynamic storage method and system based on CPU load and data packet compression efficiency

Publications (1)

Publication Number Publication Date
CN117857391A true CN117857391A (en) 2024-04-09

Family

ID=90547405

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202410095892.1A Pending CN117857391A (en) 2024-01-24 2024-01-24 Dynamic storage method and system based on CPU load and data packet compression efficiency

Country Status (1)

Country Link
CN (1) CN117857391A (en)

Similar Documents

Publication Publication Date Title
US9250968B2 (en) Method and memory manager for managing a memory in a multi-processing environment
US5390334A (en) Workstation power management by page placement control
US9733854B2 (en) Dynamic adaptive compression in network storage device
US8225044B2 (en) Storage system which utilizes two kinds of memory devices as its cache memory and method of controlling the storage system
US9727487B2 (en) Cache management method and apparatus for non-volatile storage device
EP4160379A1 (en) Storage system and energy-saving control method
US8438167B2 (en) Method and device for recording media
CN118672520B (en) Hierarchical storage method and device for file data, medium and electronic equipment
CN106502576A (en) Migration strategy adjustment method, capacity change suggestion method and device
CN1327197A (en) Method for supporting operation system of storage compression
CN101373445B (en) Method and apparatus for scheduling memory
CN114020416B (en) Large page memory dynamic management method, device and computer equipment
CN109032503B (en) Flow control method and device for data migration bandwidth of solid state disk
CN113722072A (en) Storage system file merging method and device based on intelligent distribution
CN116700634A (en) Distributed storage system garbage collection method, device and distributed storage system
CN117857391A (en) Dynamic storage method and system based on CPU load and data packet compression efficiency
CN115098039B (en) Flow control method and device for SSD command, storage medium and SSD device
CN111240825B (en) Docker cluster memory configuration method, storage media, computer equipment
CN111309480A (en) Method and equipment for dynamic power consumption capping regulation and control
CN112631512B (en) Method and equipment for adaptive steady state of SSD
WO2007046902A1 (en) Managing data for memory, a data store, and a storage device
CN118113214A (en) Dynamic allocation algorithm for memory cache space
CN118732932A (en) Storage management method and device
CN117032986A (en) Storage system flow control method and system
CN116244080A (en) A virtual memory management method based on memory request splitting

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination