CN114461138B - Scheduling method and device of storage resources, electronic equipment and readable storage medium - Google Patents
Scheduling method and device of storage resources, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114461138B CN114461138B CN202111600523.6A CN202111600523A CN114461138B CN 114461138 B CN114461138 B CN 114461138B CN 202111600523 A CN202111600523 A CN 202111600523A CN 114461138 B CN114461138 B CN 114461138B
- Authority
- CN
- China
- Prior art keywords
- storage
- log
- resource
- pool
- log storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 44
- 238000013508 migration Methods 0.000 claims description 68
- 230000005012 migration Effects 0.000 claims description 68
- 230000015654 memory Effects 0.000 claims description 33
- 238000012163 sequencing technique Methods 0.000 claims description 16
- 238000004364 calculation method Methods 0.000 claims description 3
- 238000013500 data storage Methods 0.000 abstract description 2
- 238000004891 communication Methods 0.000 description 9
- 238000012544 monitoring process Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 4
- 238000012423 maintenance Methods 0.000 description 3
- 238000012550 audit Methods 0.000 description 2
- 230000000977 initiatory effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000000802 evaporation-induced self-assembly Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000007726 management method Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention relates to the technical field of data storage, and discloses a scheduling method and device of storage resources, electronic equipment and a readable storage medium. Wherein the method comprises the following steps: acquiring a log storage resource and a log writing rate; calculating the exhaustion time of the log storage resources based on the log writing rate and the log storage resources; judging whether the depletion time is less than a preset time; and when the exhaustion time is less than the preset time, carrying out resource scheduling on the log storage resources. By implementing the method and the device, the risk of log storage resource exhaustion is eliminated, the problem that the cloud host is down or logs which cannot be written in are lost due to the fact that the log storage resource exhaustion is avoided, and the log writing capability of the cloud host is improved.
Description
Technical Field
The present invention relates to the field of data storage technologies, and in particular, to a method and apparatus for scheduling storage resources, an electronic device, and a readable storage medium.
Background
The log plays an important role in the operation management of the information system, particularly in the safety field, the log records the detailed information of all activities of a user so as to be capable of effectively tracing the time, place and process of the occurrence of the event after the occurrence of the event violating the safety rule, and the log is an important basis for the tracing and evidence obtaining analysis of the safety event. In the field of cloud computing, the internal operational details of cloud service providers should also be logged and monitored in real time.
With the continuous development of cloud computing service scenes and service scales, more and more large enterprises select to use a cloud computing platform as a main infrastructure for providing services to the outside, and in order to ensure the data security of cloud platform users, a log audit system is widely adopted by cloud service providers as an effective security means. The log audit system in the present stage usually stores the log based on the back end of the distributed storage system Ceph, in general, the distributed storage system Ceph can allocate a space with a specific size for each node according to the log writing amount of the user to perform log writing operation for all cloud hosts of the node, new logs are written continuously, and expired logs are deleted continuously, so that the remaining resources of the storage space are maintained in a relatively balanced state. However, if the user continues to perform the high-frequency operation in a specific time period, that is, the new log is continuously written, and the old log is not expired but cannot be deleted, so that the log storage space allocated to the node is continuously reduced, until the cloud host stops working or the log stops recording after the resources are exhausted, the cloud host is down, or the log which cannot be written is lost, and the like.
Disclosure of Invention
In view of the above, the embodiments of the present invention provide a method, an apparatus, an electronic device, and a readable storage medium for scheduling storage resources, so as to solve the problem that when a user continues to perform high-frequency operation, log storage resources are easy to be exhausted, so that a cloud host is down or a log which cannot be written in is lost.
According to a first aspect, an embodiment of the present invention provides a method for scheduling a storage resource, including: acquiring a log storage resource and a log writing rate; calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource; judging whether the depletion time is smaller than a preset time or not; and when the exhaustion time is smaller than the preset time, carrying out resource scheduling on the log storage resources.
According to the scheduling method of the storage resources, the log storage resources and the log writing rate are obtained, the exhaustion time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the exhaustion time is smaller than the preset time, the log storage resources are about to be exhausted, at the moment, the log storage resources are subjected to resource scheduling, the risk of log storage resource exhaustion is eliminated, the problem that a cloud host is down or logs which cannot be written are lost due to log storage resource exhaustion is avoided, and the log writing capacity of the cloud host is improved.
With reference to the first aspect, in a first implementation manner of the first aspect, the performing resource scheduling on the log storage resource includes: acquiring capacity expansion information of the log storage resource and the residual space of a storage pool corresponding to the log storage resource; and carrying out resource scheduling on the log storage resources based on the capacity expansion information and the residual space of the storage pool.
According to the scheduling method of the storage resources, the capacity expansion information of the log storage resources and the residual space of the storage pool corresponding to the log storage resources are obtained, and the log storage resources are scheduled based on the capacity expansion information and the residual space of the storage pool, so that the log storage resources can be scheduled in an actual use state, the problem that the log storage resources are exhausted during high-frequency operation is avoided, and the log writing capability during high-frequency operation is guaranteed.
With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and a remaining space of the storage pool includes: acquiring the current residual space of the storage pool after the capacity expansion of the log storage resource; judging whether the current residual space is larger than a preset space or not; and when the current residual space is larger than the preset space, expanding the capacity of the log storage resource based on the capacity expansion information.
With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: when the current residual space is smaller than the preset space, acquiring the storage state of each storage pool corresponding to the log storage resource and migration information of the log storage resource; judging whether a target migration storage pool exists in each storage pool or not based on the storage state of each storage pool; and when the target migration storage pool exists in the storage pools, migrating the log storage resources to the target migration storage pool based on the migration information.
With reference to the third implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: when the target migration storage pool does not exist in each storage pool, acquiring storage pool newly-built information and migration information of the log storage resource; generating a target storage pool corresponding to the storage pool newly-built information based on the storage pool newly-built information; and migrating the log storage resource to the target storage pool based on the migration information of the log storage resource.
According to the scheduling method of the storage resources, when the current residual space of the storage pool is larger than the preset space, the capacity of the log storage resources is expanded based on the capacity expansion information; and when the current residual space of the storage pool is smaller than the preset space, carrying out resource scheduling on the log storage resources based on the storage state of each storage pool and migration information of the log storage resources so as to ensure that the log of the high-frequency operation can be continuously written into the log storage resources. The risk of log storage resource exhaustion is eliminated through the capacity expansion mode and the migration mode, and reliable storage of the log is ensured while normal use of the cloud host is not affected.
With reference to the third implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the determining, based on the storage status of the respective storage pools, whether a target migration storage pool exists in the respective storage pools includes: based on the storage states of the storage pools, sequencing the storage pools to obtain sequencing results of the storage pools; and determining whether a target migration storage pool exists in the storage pools based on the sorting result.
According to the scheduling method of the storage resources, which is provided by the embodiment of the invention, the storage pools are ordered according to the storage states of the storage pools, so that the ordering results corresponding to the storage pools are obtained, and whether the target migration storage pool exists in the storage pools is determined according to the ordering results. According to the method, the target migration storage pool for migrating the log storage resources is determined from the storage pools, so that the utilization rate of the storage resources is improved while the log storage resources are prevented from being exhausted.
With reference to the first aspect or any implementation manner of the first to fifth implementation manners of the first aspect, in a sixth implementation manner of the first aspect, the calculating, based on the log writing rate and the log storage resource, a depletion time of the log storage resource includes: obtaining the residual space capacity corresponding to the log storage resource; and calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
According to the scheduling method of the storage resources, provided by the embodiment of the invention, the quotient between the residual space capacity and the log writing rate is calculated by acquiring the residual space capacity corresponding to the log storage resources, and the quotient between the residual space capacity and the log writing rate is used as the exhaustion time of the log storage resources. The time when the log storage resources are about to be exhausted can be determined by calculating the exhaustion time of the log storage resources, so that the log storage resources can be conveniently and timely scheduled.
According to a second aspect, an embodiment of the present invention provides a scheduling apparatus for a storage resource, including: the acquisition module is used for acquiring log storage resources and log writing rate; the calculation module is used for calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource; the judging module is used for judging whether the depletion time is smaller than a preset time or not; and the scheduling module is used for scheduling the log storage resources when the exhaustion time is smaller than the preset time.
According to a third aspect, an embodiment of the present invention provides an electronic device, including: the system comprises a memory and a processor, wherein the memory and the processor are in communication connection, the memory stores computer instructions, and the processor executes the computer instructions, so as to execute the scheduling method of the storage resource according to the first aspect or any implementation manner of the first aspect.
According to a fourth aspect, an embodiment of the present invention provides a computer readable storage medium, where the computer readable storage medium stores computer instructions for causing a computer to execute the method for scheduling storage resources according to the first aspect or any implementation manner of the first aspect.
It should be noted that, the scheduling device for storage resources, the electronic device, and the computer readable storage medium provided in the embodiments of the present invention have the corresponding beneficial effects, please refer to the description of the corresponding content in the scheduling method for storage resources, and are not repeated here.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a method of scheduling storage resources according to an embodiment of the invention;
FIG. 2 is another flow chart of a method of scheduling storage resources according to an embodiment of the invention;
FIG. 3 is another flow chart of a method of scheduling storage resources according to an embodiment of the invention;
FIG. 4 is a block diagram of a journal storage system in accordance with an embodiment of the present invention;
FIG. 5 is a block diagram of a memory resource scheduler according to an embodiment of the present invention;
Fig. 6 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments of the present invention. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
According to the technical scheme, when the log writing rate is higher, the log storage resources are scheduled based on the exhaustion time of the log storage resources, so that log storage resources are prevented from being exhausted in high-frequency log writing, the problem that a cloud host is down or logs which cannot be written are lost due to the fact that the log storage resources are exhausted is solved, and the log writing capacity of the cloud host is improved.
According to an embodiment of the present invention, there is provided an embodiment of a scheduling method of storage resources, it should be noted that the steps illustrated in the flowchart of the drawings may be performed in a computer system such as a set of computer executable instructions, and that although a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different from that herein.
In this embodiment, a method for scheduling storage resources is provided, which may be used for an electronic device that adopts a log storage system, such as a mobile phone, a tablet computer, a server, etc., fig. 1 is a flowchart of a method for scheduling storage resources according to an embodiment of the present invention, and as shown in fig. 1, the flowchart includes the following steps:
S11, acquiring log storage resources and log writing rate.
The log storage resource is a storage space for storing logs, specifically, a log storage system of the electronic device may include a plurality of storage pools, each storage pool includes one or more storage volumes, each log storage volume may store log information of all cloud hosts corresponding to one node, and log information corresponding to different cloud hosts is stored under different catalogues of the log storage volume.
The log write rate is the rate at which log information is written to the log storage resource, which can be characterized using the number of bytes written to the log storage resource per second. The log writing rate can be an average value of the writing rate of the log information to the log storage resource in a certain period of time, for example, a log storage system in the electronic equipment can monitor the writing rate of the log information to the log storage resource at each moment in real time, and the log writing rate is calculated according to the average value; the log writing rate may also be a real-time rate of writing the log information monitored at the current moment into the log storage resource, and of course, the log writing rate may also be obtained by other manners, which is not limited herein specifically.
S12, calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource.
The exhaustion time is the exhaustion time of the log storage resources, and the log storage system in the electronic equipment can determine the current log storage space according to the log storage resources and estimate the exhaustion time of the log storage resources by combining the current log writing rate. For example, the current log storage space corresponding to the log storage resource is C 0, and the log writing rate is V 0, so that the exhaustion time t=c 0/V0 can be estimated.
S13, judging whether the depletion time is smaller than the preset time.
The preset time is the shortest time for the preset log storage resource to run out, after the estimated depletion time is obtained, the depletion time is compared with the preset time to determine the relation between the depletion time and the preset time, when the depletion time is smaller than the preset time, the step S14 is executed, and otherwise, no operation is executed.
S14, carrying out resource scheduling on the log storage resources.
When the exhaustion time is smaller than the preset time, the log information is indicated to be written continuously at high frequency, at this time, the log storage system of the electronic device can schedule log storage resources, for example, expand the capacity of the log storage resources, or migrate the log storage resources to other storage pools, etc., where the resource scheduling mode is not particularly limited, so long as the scheduling mode can be implemented to expand the log storage space corresponding to the log storage resources and ensure that the log can be written normally, and those skilled in the art can determine according to actual needs.
According to the scheduling method of the storage resources, the log storage resources and the log writing rate are obtained, the consumption time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the consumption time is smaller than the preset time, the log storage resources are about to be consumed, at the moment, the log storage resources are subjected to resource scheduling, the risk of log storage resource consumption is eliminated, the problem that a cloud host is down or logs which cannot be written are lost due to log storage resource consumption is avoided, and the log writing capacity of the cloud host is improved.
In this embodiment, a method for scheduling storage resources is provided, which may be used for an electronic device that adopts a log storage system, such as a mobile phone, a tablet computer, a server, etc., fig. 2 is a flowchart of a method for scheduling storage resources according to an embodiment of the present invention, and as shown in fig. 2, the flowchart includes the following steps:
S21, acquiring a log storage resource and a log writing rate. The detailed description refers to the corresponding related descriptions of the above embodiments, and will not be repeated here.
S22, calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource.
Specifically, the step S22 may include:
s221, obtaining the residual space capacity corresponding to the log storage resource.
The remaining space capacity corresponding to the log storage resource is the log storage space remaining by the current log storage resource. The log storage system of the electronic equipment can count the occupied space of the existing log information in the current log storage resource, and calculate the difference between the occupied space of the existing log information and the total storage space of the log storage resource by combining the total storage space of the log storage resource, so that the residual space capacity corresponding to the log storage resource can be obtained.
S222, calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
The log writing rate is the number of bytes written to the log storage resource per second, the remaining space capacity is divided by the log writing rate, the quotient between the remaining space capacity and the log writing rate can be calculated, and the quotient between the remaining space capacity and the log writing rate is determined as the exhaustion time of the log storage resource.
S23, judging whether the depletion time is smaller than the preset time. The detailed description refers to the corresponding related descriptions of the above embodiments, and will not be repeated here.
And S24, when the exhaustion time is smaller than the preset time, carrying out resource scheduling on the log storage resources.
Specifically, the step S24 may include:
s241, obtaining capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource.
The capacity expansion information is used to represent the expansion multiple of the log storage resource, for example, the capacity expansion information may be 1.5 times of capacity expansion, may be 2 times of capacity expansion, and may be other expansion multiple of capacity, which is not limited herein specifically. The remaining space of the storage pool corresponding to the log storage resource is the current remaining storage space of the storage pool, specifically, a log storage system in the electronic device can acquire the total space of the storage pool where the log storage resource is located, count the space occupied by the log storage resource, and then determine the remaining space of the storage pool according to the total space of the storage pool and the space occupied by the log storage resource.
And S242, scheduling the log storage resources based on the capacity expansion information and the residual space of the storage pool.
The log storage system of the electronic device may determine, according to the remaining space of the storage pool, whether the log storage resource meets the capacity expansion requirement. When the residual space of the storage pool meets the capacity expansion requirement, expanding the log storage resources based on the capacity expansion information so as to realize the resource scheduling of the log storage resources; when the residual space of the storage pool does not meet the capacity expansion requirement, the log storage resources can be subjected to hot migration so as to realize resource scheduling of the log storage resources.
Specifically, as shown in fig. 3, the step S242 may include:
s1, acquiring the current residual space of the storage pool after the capacity expansion of the log storage resources.
The current residual space of the storage pool is the log storage space remained in the storage pool, the log storage system of the electronic equipment expands the capacity of the log storage resource in the storage pool where the log storage resource is located according to the capacity expansion information, and the current residual space of the storage pool after the capacity expansion of the log storage resource is calculated.
S2, judging whether the current residual space is larger than a preset space or not.
The preset space is the lowest remaining space threshold of the preset storage pool. The log storage system of the electronic device may compare the determined current remaining space of the storage pool with a preset space to determine a relationship between the current remaining space and the preset space. And when the current residual space is larger than the preset space, executing the step S3, otherwise executing the step S4.
And S3, expanding the capacity of the log storage resources based on the capacity expansion information.
If the current residual space is larger than the preset space, the residual space of the storage pool where the log storage resource is located is indicated to meet the capacity expansion requirement of the log storage resource, and at the moment, the log storage system can expand the capacity of the log storage resource according to the capacity expansion information. For example, the capacity expansion information is 1.5 times the capacity expansion, and the log storage system can expand the log storage resources in the current storage pool according to 1.5 times of the log storage resources.
S4, obtaining the storage states of the storage pools corresponding to the log storage resources and migration information of the log storage resources.
The storage status of each storage pool is used to characterize storage information of each storage pool, such as storage space remaining in each storage pool, and the estimated exhaustion time of log storage resources corresponding to each storage pool at the current log writing rate. The migration information of the log storage resource is used to thermally migrate the log storage resource to other storage pools.
If the current remaining space is smaller than the preset space, the remaining space of the storage pool where the log storage resource is located does not meet the capacity expansion requirement of the log storage resource, and at this time, the log storage resource can be thermally migrated to other storage pools to avoid the log incapable of being written when the log storage resource is exhausted.
S5, judging whether the target migration storage pool exists in each storage pool or not based on the storage state of each storage pool.
The target migration storage pool is a migration target corresponding to the log storage resource, and the log storage system of the electronic device can determine whether the target migration storage pool exists in each storage pool according to the acquired storage state of each storage pool. When the target migration storage pool exists, executing step S6, otherwise executing step S7.
For example, the target migration storage pool may be the largest remaining space, or may be the longest storage resource exhaustion time estimated by the current log writing rate, or may be determined according to other information, which is not limited herein, and may be determined by a person skilled in the art according to actual needs.
Specifically, the step S5 may include:
(1) And sequencing the storage pools based on the storage states of the storage pools to obtain sequencing results of the storage pools.
And sequencing all the storage pools according to the storage states of the storage pools to obtain sequencing results of the storage pools. Taking the remaining space of the storage pool as an example, a scheduling module is arranged in a log storage system of the electronic equipment, and the scheduling module can order the remaining space of each storage pool from large to small to obtain an ordering result of each storage pool.
(2) Based on the ranking results, it is determined whether a target migration storage pool exists in each storage pool.
And determining a target migration storage pool meeting the requirements from the storage pools according to the sequencing result of the storage pools. Taking the remaining space of the storage pool as an example, the scheduling module may find the storage pool with the largest remaining space from each storage pool according to the sequencing result of each storage pool, and use the storage pool with the largest remaining space as the target migration storage pool.
And sequencing the storage pools according to the storage states of the storage pools to obtain sequencing results corresponding to the storage pools, and determining whether target migration storage pools exist in the storage pools according to the sequencing results. According to the method, the target migration storage pool for migrating the log storage resources is determined from the storage pools, so that the utilization rate of the storage resources is improved while the log storage resources are prevented from being exhausted.
And S6, migrating the log storage resources to a target migration storage pool based on migration information.
When the target migration storage pool exists in each storage pool, the storage pool meeting the migration requirement exists, and at this time, the log storage resources can be thermally migrated to the target migration storage pool according to the migration information.
S7, obtaining new information of the storage pool and migration information of the log storage resource.
When the target migration storage pool does not exist in each storage pool, the storage pools which meet the migration requirement do not exist in all the storage pools, and at the moment, newly-built information of the storage pools can be acquired to newly-built one storage pool to migrate the log storage resources.
And S8, generating a target storage pool corresponding to the storage pool newly-built information based on the storage pool newly-built information.
The target storage pool is a storage pool for migration log storage resources, and when the target migration storage pool does not exist in each storage pool, a new storage pool can be created according to the storage pool newly created information, and the new storage pool is determined to be the target storage pool.
S9, based on migration information of the log storage resources, migrating the log storage resources to a target storage pool.
After the creation of the target storage pool is completed, the log storage resources can be thermally migrated to the target storage pool according to migration information of the log storage resources, so that the expansion of the log storage resources is realized, the target storage pool continues to store the logs written by the cloud host, and the loss of the logs or downtime of the cloud host caused by insufficient storage resources is avoided.
According to the scheduling method of the storage resources, the quotient between the residual space capacity and the writing rate is calculated by acquiring the residual space capacity corresponding to the log storage resources, and the quotient between the residual space capacity and the writing rate is used as the exhaustion time of the log storage resources. The time when the log storage resources are about to be exhausted can be determined by calculating the exhaustion time of the log storage resources, so that the log storage resources can be conveniently and timely scheduled.
The capacity expansion information of the log storage resources and the residual space of the storage pool corresponding to the log storage resources are obtained, and the log storage resources are subjected to resource scheduling based on the capacity expansion information and the residual space of the storage pool, so that the log storage resources can be subjected to resource scheduling in an actual use state, the problem that the log storage resources are exhausted during high-frequency operation is avoided, and the log writing capability during high-frequency operation is ensured.
When the current residual space of the storage pool is larger than the preset space, expanding the capacity of the log storage resources based on the capacity expansion information; and when the current residual space of the storage pool is smaller than the preset space, carrying out resource scheduling on the log storage resources based on the storage state of each storage pool and migration information of the log storage resources so as to ensure that the log of the high-frequency operation can be continuously written into the log storage resources. The risk of log storage resource exhaustion is eliminated through the capacity expansion mode and the migration mode, and reliable storage of the log is ensured while normal use of the cloud host is not affected.
The embodiment of the invention takes a specific log storage system as an example to describe the scheduling method of the storage resources, and the log storage system comprises: ceph-based distributed log storage system, log collection module, monitoring module, scheduling module and elastic telescopic module, the structure of which is shown in figure 4. The distributed log storage scheme divides log storage resources into different log storage resource pools, and each log storage resource pool is divided into different log storage volumes. Each log storage volume stores log information of all cloud hosts on one node, and different catalogues of the log storage volumes are stored according to different cloud hosts.
The log collection module is used for collecting all log information generated in the operation process of the cloud host, distinguishing the collected logs according to the nodes where the cloud host is located, storing the logs into corresponding log storage volumes, and storing the logs into different log catalogues according to the differences of the cloud host.
The monitoring module is used for monitoring the resource surplus condition of each log storage volume in each log storage resource pool in real time, recording the log writing rate in the current period of time and estimating the depletion time T of the log storage volume. Further, the monitoring module may compare the exhaustion time T with a minimum threshold time Tmin specified by the cloud platform operation and maintenance personnel, and if T < Tmin, notify the scheduling module to perform a corresponding operation.
The scheduling module receives the information about to be used up of the log storage volume transmitted by the monitoring module and then performs corresponding operations: calculating the residual capacity C of the log storage resource pool after the capacity expansion of the log storage volume, and if the residual capacity C is larger than the minimum residual capacity Cmin of the log storage resource pool, initiating a capacity expansion instruction of the log storage volume to an elastic expansion module; if the residual capacity C is smaller than the minimum residual capacity Cmin of the log storage resource pools, sequencing all the log storage resource pools according to a preset rule, selecting one log storage resource pool as a target migration storage pool, and initiating a log storage volume cross-storage resource pool hot migration instruction by an elastic expansion module; if all the log storage resource pools can not be used as the target migration storage pools of the log storage volumes, the elastic expansion module is informed of newly building a log storage resource pool in the distributed log storage system. The scheduling module transmits specific operation instructions to the elastic expansion module on the cloud host generating the log no matter what operation is performed on the log storage volume.
After receiving the instruction sent by the scheduling module, the elastic expansion module can identify the received instruction so as to perform corresponding operations of expanding the volume of the log storage volume, thermally migrating the log storage resource pool or newly creating the log storage resource pool. And for the capacity expansion instruction of the log storage volume, the elastic expansion module directly applies for storage resources in a resource pool where the storage volume is located, expands the capacity of the storage volume, feeds back stock information of the storage volume and the storage resource pool to the monitoring module after the capacity expansion is finished, and feeds back operation completion information to the scheduling module. And for a thermal migration instruction of the log storage volume, the elastic expansion module performs thermal migration operation on the log storage volume according to target log storage resource pool information in the thermal migration instruction, and can perform capacity expansion operation on the log storage volume according to the requirements of operation and maintenance personnel during thermal migration, and after the thermal migration of the log storage volume across the resource pools is completed, relevant information is fed back to the monitoring module and the scheduling module. And for the newly-built log storage resource pool, the elastic expansion module is created according to the size of the resource pool specified by the operation and maintenance personnel. And feeding back information to the monitoring module and the scheduling module after the creation is completed.
According to the log storage system provided by the embodiment, when the log storage resources are about to be exhausted, the log storage resources can be subjected to resource scheduling, the risk of exhausting the log storage resources is eliminated, the problem that a cloud host is down or logs which cannot be written in are lost due to the fact that the log storage resources are exhausted is avoided, and the log writing capacity of the cloud host is improved. By monitoring and scheduling the log storage volume, the log is written without upper limit to the greatest extent, and the log of the cloud host is ensured to be reliably stored.
The embodiment also provides a scheduling device for storage resources, which is used for implementing the foregoing embodiments and preferred embodiments, and is not described in detail. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. While the means described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.
The present embodiment provides a scheduling apparatus for storage resources, as shown in fig. 5, including:
The obtaining module 31 is configured to obtain a log storage resource and a log writing rate. The detailed description refers to the corresponding related description of the above method embodiments, and will not be repeated here.
A calculation module 32 for calculating a depletion time of the log storage resource based on the log writing rate and the log storage resource. The detailed description refers to the corresponding related description of the above method embodiments, and will not be repeated here.
A judging module 33, configured to judge whether the exhaustion time is less than a preset time. The detailed description refers to the corresponding related description of the above method embodiments, and will not be repeated here.
And the scheduling module 34 is configured to schedule the log storage resource when the exhaustion time is less than the preset time. The detailed description refers to the corresponding related description of the above method embodiments, and will not be repeated here.
According to the scheduling device for the storage resources, the log storage resources and the log writing rate are obtained, the consumption time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the consumption time is smaller than the preset time, the log storage resources are about to be consumed, and at the moment, the log storage resources are subjected to resource scheduling, so that the risk of log storage resource consumption is eliminated, the problem that a cloud host is down or logs which cannot be written due to log storage resource consumption are lost is avoided, and the log writing capacity of the cloud host is improved.
The scheduling means of the memory resources in this embodiment are presented in the form of functional units, where a unit refers to an ASIC circuit, a processor and a memory executing one or more software or fixed programs, and/or other devices that can provide the above described functionality.
Further functional descriptions of the above modules are the same as those of the above corresponding embodiments, and are not repeated here.
The embodiment of the invention also provides the electronic equipment, which is provided with the scheduling device of the storage resources shown in the figure 5.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, as shown in fig. 6, the electronic device may include: at least one processor 401, such as a CPU (Central Processing Unit ), at least one communication interface 403, a memory 404, at least one communication bus 402. Wherein communication bus 402 is used to enable connected communications between these components. The communication interface 403 may include a Display screen (Display) and a Keyboard (Keyboard), and the optional communication interface 403 may further include a standard wired interface and a wireless interface. The memory 404 may be a high-speed RAM memory (Random Access Memory, volatile random access memory) or a non-volatile memory (non-volatile memory), such as at least one disk memory. The memory 404 may also optionally be at least one storage device located remotely from the aforementioned processor 401. Wherein the processor 401 may be described in connection with fig. 5, the application program is stored in the memory 404, and the processor 401 invokes the program code stored in the memory 404 for performing any of the above-mentioned method steps.
The communication bus 402 may be a peripheral component interconnect standard (PERIPHERAL COMPONENT INTERCONNECT, PCI) bus or an extended industry standard architecture (extended industry standard architecture, EISA) bus, among others. Communication bus 402 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in fig. 6, but not only one bus or one type of bus.
Wherein the memory 404 may include volatile memory (English) such as random-access memory (RAM); the memory may also include a nonvolatile memory (English: non-volatile memory), such as a flash memory (English: flash memory), a hard disk (English: HARD DISK DRIVE, abbreviation: HDD) or a solid state disk (English: solid-STATE DRIVE, abbreviation: SSD); memory 404 may also include a combination of the above types of memory.
The processor 401 may be a central processor (english: central processing unit, abbreviated: CPU), a network processor (english: network processor, abbreviated: NP) or a combination of CPU and NP.
Wherein the processor 401 may further comprise a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof (English: programmable logic device). The PLD may be a complex programmable logic device (English: complex programmable logic device, abbreviated: CPLD), a field-programmable gate array (English: field-programmable GATE ARRAY, abbreviated: FPGA), a general-purpose array logic (English: GENERIC ARRAY logic, abbreviated: GAL), or any combination thereof.
Optionally, the memory 404 is also used for storing program instructions. The processor 401 may invoke program instructions to implement the scheduling method of storage resources as shown in the embodiments of fig. 1 to 3 of the present application.
The embodiment of the invention also provides a non-transitory computer storage medium, which stores computer executable instructions, and the computer executable instructions can execute the processing method of the scheduling method of the storage resource in any method embodiment. Wherein the storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a random access Memory (Random Access Memory, RAM), a Flash Memory (Flash Memory), a hard disk (HARD DISK DRIVE, abbreviated as HDD), a Solid state disk (Solid-state-STATE DRIVE, SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although embodiments of the present invention have been described in connection with the accompanying drawings, various modifications and variations may be made by those skilled in the art without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope of the invention as defined by the appended claims.
Claims (6)
1. A method for scheduling storage resources, comprising:
acquiring a log storage resource and a log writing rate;
calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource;
Judging whether the depletion time is smaller than a preset time or not;
When the exhaustion time is smaller than a preset time, carrying out resource scheduling on the log storage resources; the resource scheduling of the log storage resource includes: acquiring capacity expansion information of the log storage resource and the residual space of a storage pool corresponding to the log storage resource; scheduling the log storage resources based on the capacity expansion information and the residual space of the storage pool; the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool includes: acquiring the current residual space of the storage pool after the capacity expansion of the log storage resource; judging whether the current residual space is larger than a preset space or not; when the current residual space is smaller than the preset space, acquiring the storage state of each storage pool corresponding to the log storage resource and migration information of the log storage resource; judging whether a target migration storage pool exists in each storage pool or not based on the storage state of each storage pool; when a target migration storage pool exists in each storage pool, migrating the log storage resource to the target migration storage pool based on the migration information; the determining, based on the storage status of the storage pools, whether the target migration storage pool exists in the storage pools includes: based on the storage states of the storage pools, sequencing the storage pools to obtain sequencing results of the storage pools; based on the sorting result, taking the storage pool with the largest residual space in the storage pools as the target migration storage pool;
The resource scheduling for the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: and when the current residual space is larger than the preset space, expanding the capacity of the log storage resource based on the capacity expansion information.
2. The method as recited in claim 1, further comprising:
When the target migration storage pool does not exist in each storage pool, acquiring storage pool newly-built information and migration information of the log storage resource;
generating a target storage pool corresponding to the storage pool newly-built information based on the storage pool newly-built information;
And migrating the log storage resource to the target storage pool based on the migration information of the log storage resource.
3. The method of any of claims 1-2, wherein the calculating the exhaustion time of the log storage resource based on the log write rate and the log storage resource comprises:
obtaining the residual space capacity corresponding to the log storage resource;
And calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
4. A scheduling apparatus for storage resources, comprising:
the acquisition module is used for acquiring log storage resources and log writing rate;
The calculation module is used for calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource;
The judging module is used for judging whether the depletion time is smaller than a preset time or not;
The scheduling module is used for scheduling the log storage resources when the exhaustion time is smaller than a preset time; the resource scheduling of the log storage resource includes: acquiring capacity expansion information of the log storage resource and the residual space of a storage pool corresponding to the log storage resource; scheduling the log storage resources based on the capacity expansion information and the residual space of the storage pool; the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool includes: acquiring the current residual space of the storage pool after the capacity expansion of the log storage resource; judging whether the current residual space is larger than a preset space or not; when the current residual space is smaller than the preset space, acquiring the storage state of each storage pool corresponding to the log storage resource and migration information of the log storage resource; judging whether a target migration storage pool exists in each storage pool or not based on the storage state of each storage pool; when a target migration storage pool exists in each storage pool, migrating the log storage resource to the target migration storage pool based on the migration information; the determining, based on the storage status of the storage pools, whether the target migration storage pool exists in the storage pools includes: based on the storage states of the storage pools, sequencing the storage pools to obtain sequencing results of the storage pools; based on the sorting result, taking the storage pool with the largest residual space in the storage pools as the target migration storage pool; the resource scheduling for the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: and when the current residual space is larger than the preset space, expanding the capacity of the log storage resource based on the capacity expansion information.
5. An electronic device, comprising:
a memory and a processor, said memory and said processor being communicatively coupled to each other, said memory having stored therein computer instructions, said processor executing the method for scheduling storage resources of any of claims 1-4 by executing said computer instructions.
6. A computer readable storage medium storing computer instructions for causing a computer to perform the scheduling method of storage resources of any one of claims 1-4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600523.6A CN114461138B (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600523.6A CN114461138B (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114461138A CN114461138A (en) | 2022-05-10 |
| CN114461138B true CN114461138B (en) | 2024-06-07 |
Family
ID=81407612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111600523.6A Active CN114461138B (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114461138B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101226457A (en) * | 2008-01-25 | 2008-07-23 | 中兴通讯股份有限公司 | On-line capacity-enlarging system and method for magnetic disc array |
| JP2008217575A (en) * | 2007-03-06 | 2008-09-18 | Nec Corp | Storage device and configuration optimization method thereof |
| CN102096557A (en) * | 2010-12-31 | 2011-06-15 | 成都市华为赛门铁克科技有限公司 | Capacity expansion method, device and system for independent redundant array of inexpensive disc (RAID) |
| CN103247347A (en) * | 2012-02-11 | 2013-08-14 | 三星电子株式会社 | Method and system for providing smart memory architecture |
| WO2016107167A1 (en) * | 2014-12-31 | 2016-07-07 | 中兴通讯股份有限公司 | Storage architecture creation method, storage access method and storage system |
| CN107340972A (en) * | 2016-11-22 | 2017-11-10 | 青海师范大学 | A kind of RAID0 capacity expansion method |
| CA3104353A1 (en) * | 2018-07-25 | 2020-01-30 | Transwarp Technology (Shanghai) Co., Ltd. | Storage volume creation method and apparatus, server, and storage medium |
| CN111414136A (en) * | 2020-03-13 | 2020-07-14 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for creating storage pool |
| CN112463043A (en) * | 2020-11-20 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Storage cluster capacity expansion method, system and related device |
| CN112799596A (en) * | 2021-02-03 | 2021-05-14 | 联想(北京)有限公司 | Capacity expansion control method and device for storage resources and electronic equipment |
| CN113098946A (en) * | 2021-03-26 | 2021-07-09 | 西安万像电子科技有限公司 | Cloud desktop scene identification method and device, storage medium and electronic device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8918661B2 (en) * | 2010-05-11 | 2014-12-23 | Hitachi, Ltd. | Method and apparatus for assigning storage resources to a power saving target storage pool based on either access frequency or power consumption |
| CN103365781B (en) * | 2012-03-29 | 2016-05-04 | 国际商业机器公司 | For dynamically reconfiguring the method and apparatus of storage system |
| US10678456B2 (en) * | 2018-09-28 | 2020-06-09 | International Business Machines Corporation | Storage capacity usage management |
| US10949116B2 (en) * | 2019-07-30 | 2021-03-16 | EMC IP Holding Company LLC | Storage resource capacity prediction utilizing a plurality of time series forecasting models |
| US11163476B2 (en) * | 2019-10-04 | 2021-11-02 | International Business Machines Corporation | Dynamic rebalancing of free space between storage pools |
-
2021
- 2021-12-24 CN CN202111600523.6A patent/CN114461138B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2008217575A (en) * | 2007-03-06 | 2008-09-18 | Nec Corp | Storage device and configuration optimization method thereof |
| CN101226457A (en) * | 2008-01-25 | 2008-07-23 | 中兴通讯股份有限公司 | On-line capacity-enlarging system and method for magnetic disc array |
| CN102096557A (en) * | 2010-12-31 | 2011-06-15 | 成都市华为赛门铁克科技有限公司 | Capacity expansion method, device and system for independent redundant array of inexpensive disc (RAID) |
| CN103247347A (en) * | 2012-02-11 | 2013-08-14 | 三星电子株式会社 | Method and system for providing smart memory architecture |
| WO2016107167A1 (en) * | 2014-12-31 | 2016-07-07 | 中兴通讯股份有限公司 | Storage architecture creation method, storage access method and storage system |
| CN107340972A (en) * | 2016-11-22 | 2017-11-10 | 青海师范大学 | A kind of RAID0 capacity expansion method |
| CA3104353A1 (en) * | 2018-07-25 | 2020-01-30 | Transwarp Technology (Shanghai) Co., Ltd. | Storage volume creation method and apparatus, server, and storage medium |
| CN111414136A (en) * | 2020-03-13 | 2020-07-14 | 苏州浪潮智能科技有限公司 | Method, system, device and medium for creating storage pool |
| CN112463043A (en) * | 2020-11-20 | 2021-03-09 | 苏州浪潮智能科技有限公司 | Storage cluster capacity expansion method, system and related device |
| CN112799596A (en) * | 2021-02-03 | 2021-05-14 | 联想(北京)有限公司 | Capacity expansion control method and device for storage resources and electronic equipment |
| CN113098946A (en) * | 2021-03-26 | 2021-07-09 | 西安万像电子科技有限公司 | Cloud desktop scene identification method and device, storage medium and electronic device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114461138A (en) | 2022-05-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| EP3070591A1 (en) | Cold and hot data identification threshold calculation method, apparatus and system | |
| CN109343965A (en) | Resource adjusting method, device, cloud platform and server | |
| CN108205541B (en) | Method and device for scheduling distributed web crawler tasks | |
| CN110311831B (en) | Container cloud-based system resource monitoring method and related equipment | |
| CN106452818B (en) | Resource scheduling method and system | |
| CN110442498B (en) | Abnormal data node positioning method and device, storage medium and computer equipment | |
| US20180011742A1 (en) | Job scheduling management | |
| JP6191691B2 (en) | Abnormality detection apparatus, control method, and program | |
| US20150370587A1 (en) | Computer-readable recording medium having stored therein outputting program, output apparatus and outputting method | |
| CN106027595A (en) | Access log processing method and system for CDN node | |
| KR102406580B1 (en) | Online Flash Resource Allocation Method and System Based on a TCO Model | |
| CN110147470B (en) | Cross-machine-room data comparison system and method | |
| CN110213128B (en) | Service port detection method, electronic device and computer storage medium | |
| CN107977167A (en) | Optimization method is read in a kind of degeneration of distributed memory system based on correcting and eleting codes | |
| CN102916831A (en) | Method and system for acquiring health degree of business system | |
| CN111767199A (en) | Resource management method, device, equipment and system based on batch processing operation | |
| CN103309843B (en) | The collocation method of server and system | |
| CN109597764A (en) | A kind of test method and relevant apparatus of catalogue quota | |
| CN115291806A (en) | Processing method, processing device, electronic equipment and storage medium | |
| EP2940600A1 (en) | Data scanning method and device | |
| CN116483546B (en) | Distributed training task scheduling method, device, equipment and storage medium | |
| CN109992408B (en) | Resource allocation method, device, electronic equipment and storage medium | |
| CN114461138B (en) | Scheduling method and device of storage resources, electronic equipment and readable storage medium | |
| CN110347546B (en) | Dynamic adjustment method, device, medium and electronic equipment for monitoring task | |
| CN110188258B (en) | Method and device for acquiring external data by using crawler |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |