CN114461138A - Scheduling method and device of storage resources, electronic equipment and readable storage medium - Google Patents
Scheduling method and device of storage resources, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN114461138A CN114461138A CN202111600523.6A CN202111600523A CN114461138A CN 114461138 A CN114461138 A CN 114461138A CN 202111600523 A CN202111600523 A CN 202111600523A CN 114461138 A CN114461138 A CN 114461138A
- Authority
- CN
- China
- Prior art keywords
- storage
- log
- resource
- pool
- log storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
- G06F3/0644—Management of space entities, e.g. partitions, extents, pools
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/0647—Migration mechanisms
Abstract
The invention relates to the technical field of data storage, and discloses a method and a device for scheduling storage resources, electronic equipment and a readable storage medium. Wherein, the method comprises the following steps: acquiring log storage resources and a log writing rate; calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource; judging whether the depletion time is less than the preset time or not; and when the exhaustion time is less than the preset time, performing resource scheduling on the log storage resource. By implementing the method and the device, the risk of log storage resource exhaustion is eliminated, the problem that the cloud host is down or the log which cannot be written is lost due to the log storage resource exhaustion is avoided, and the log writing capability of the cloud host is improved.
Description
Technical Field
The invention relates to the technical field of data storage, in particular to a method and a device for scheduling storage resources, electronic equipment and a readable storage medium.
Background
The log plays an important role in the operation management of the information system, particularly in the field of security, records the detailed information of all activities of a user so as to effectively track the time, the place and the process of an event which violates a security rule after the event occurs, and is an important basis for the tracing, forensics and analysis of the security event. In the field of cloud computing, the internal operation details of the cloud service provider should also be logged and monitored in real time.
With the continuous development of cloud computing service scenes and service scales, more and more large enterprises select to use a cloud computing platform as a main infrastructure for providing services to the outside, and in order to ensure the data security of cloud platform users, a log auditing system is widely adopted by cloud service providers as an effective security means. In the current stage, log storage is usually performed on the log auditing system based on the rear end of the distributed storage system Ceph, in a general case, the distributed storage system Ceph can allocate a space with a specific size to each node according to the log writing amount of a user to perform log writing operation on all cloud hosts of the node, new logs are continuously written, expired logs are continuously deleted, and therefore the remaining resources of the storage space are maintained in a relatively balanced state. However, if the user continues to perform the high-frequency operation in a certain specific time period, that is, new logs are continuously written, and old logs are not expired and cannot be deleted, the log storage space allocated to the node is continuously reduced until the cloud host stops working or logs stop recording after resources are exhausted, so that the cloud host goes down or logs which cannot be written are lost.
Disclosure of Invention
In view of this, embodiments of the present invention provide a method and an apparatus for scheduling storage resources, an electronic device, and a readable storage medium, so as to solve the problem that log storage resources are easily exhausted when a user continuously performs high-frequency operations, which causes a cloud host to be down or a log that cannot be written to is lost.
According to a first aspect, an embodiment of the present invention provides a method for scheduling storage resources, including: acquiring log storage resources and a log writing rate; calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource; judging whether the depletion time is less than a preset time or not; and when the exhaustion time is less than the preset time, performing resource scheduling on the log storage resource.
According to the scheduling method of the storage resources, the log storage resources and the log writing rate are obtained, the exhaustion time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the exhaustion time is smaller than the preset time, the log storage resources are indicated to be exhausted, at the moment, the resource scheduling is performed on the log storage resources, the risk of exhaustion of the log storage resources is eliminated, the problem that a cloud host is down or logs which cannot be written into the cloud host are lost due to exhaustion of the log storage resources is solved, and the log writing capacity of the cloud host is improved.
With reference to the first aspect, in a first implementation manner of the first aspect, the performing resource scheduling on the log storage resource includes: acquiring capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource; and performing resource scheduling on the log storage resource based on the capacity expansion information and the residual space of the storage pool.
According to the scheduling method of the storage resource provided by the embodiment of the invention, the capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource are obtained, and the log storage resource is scheduled based on the capacity expansion information and the residual space of the storage pool, so that the resource scheduling of the log storage resource can be carried out in an actual use state, the problem of log storage resource exhaustion in high-frequency operation is avoided, and the log writing capability in high-frequency operation is ensured.
With reference to the first implementation manner of the first aspect, in a second implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool includes: acquiring the current residual space of the storage pool after the capacity expansion of the log storage resources; judging whether the current residual space is larger than a preset space or not; and when the current residual space is larger than the preset space, carrying out capacity expansion on the log storage resource based on the capacity expansion information.
With reference to the second implementation manner of the first aspect, in a third implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: when the current residual space is smaller than the preset space, acquiring the storage state of each storage pool corresponding to the log storage resource and the migration information of the log storage resource; judging whether a target migration storage pool exists in each storage pool or not based on the storage state of each storage pool; and when a target migration storage pool exists in the storage pools, migrating the log storage resources to the target migration storage pool based on the migration information.
With reference to the third implementation manner of the first aspect, in a fourth implementation manner of the first aspect, the performing resource scheduling on the log storage resource based on the capacity expansion information and the remaining space of the storage pool further includes: when the target migration storage pool does not exist in each storage pool, acquiring new storage pool creation information and migration information of the log storage resource; generating a target storage pool corresponding to the storage pool new construction information based on the storage pool new construction information; migrating the journal storage resource to the target storage pool based on the migration information of the journal storage resource.
According to the scheduling method of the storage resources provided by the embodiment of the invention, when the current remaining space of the storage pool is larger than the preset space, capacity expansion is carried out on the log storage resources based on capacity expansion information; and when the current residual space of the storage pools is smaller than the preset space, performing resource scheduling on the log storage resources based on the storage state of each storage pool and the migration information of the log storage resources so as to ensure that the logs operated at high frequency can be continuously written into the log storage resources. The risk of log storage resource exhaustion is eliminated through a capacity expansion mode and a migration mode, and the reliable storage of the logs is guaranteed while the normal use of the cloud host is not influenced.
With reference to the third implementation manner of the first aspect, in a fifth implementation manner of the first aspect, the determining whether a target migration storage pool exists in the storage pools based on the storage statuses of the storage pools includes: sorting the storage pools based on the storage states of the storage pools to obtain sorting results of the storage pools; determining whether a target migration storage pool exists in the respective storage pools based on the ranking result.
According to the scheduling method of the storage resources provided by the embodiment of the invention, the storage pools are sequenced according to the storage states of the storage pools to obtain sequencing results corresponding to the storage pools, and whether the target migration storage pool exists in each storage pool is determined according to the sequencing results. According to the method, the target migration storage pool for migrating the log storage resource is determined from the storage pools, so that the log storage resource is prevented from being exhausted, and the utilization rate of the storage resource is improved.
With reference to the first aspect or any one of the first to fifth embodiments of the first aspect, in a sixth embodiment of the first aspect, the calculating an exhaustion time of the log storage resource based on the log writing rate and the log storage resource includes: acquiring the residual space capacity corresponding to the log storage resource; and calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
According to the scheduling method of the storage resources, the quotient between the residual space capacity and the log writing rate is calculated by acquiring the residual space capacity corresponding to the log storage resources, and the quotient between the residual space capacity and the log writing rate is used as the exhaustion time of the log storage resources. By calculating the exhaustion time of the log storage resource, the time that the log storage resource is about to be exhausted can be determined, and the log storage resource is convenient to be scheduled in time.
According to a second aspect, an embodiment of the present invention provides an apparatus for scheduling storage resources, including: the acquisition module is used for acquiring log storage resources and log writing rate; the calculation module is used for calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource; the judging module is used for judging whether the depletion time is less than the preset time or not; and the scheduling module is used for scheduling the resources of the log storage resources when the exhaustion time is less than the preset time.
According to a third aspect, an embodiment of the present invention provides an electronic device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the method for scheduling storage resources according to the first aspect or any embodiment of the first aspect.
According to a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions for causing a computer to execute the method for scheduling storage resources according to the first aspect or any implementation manner of the first aspect.
It should be noted that, for corresponding beneficial effects of the scheduling apparatus for storage resources, the electronic device and the computer-readable storage medium provided in the embodiment of the present invention, please refer to descriptions of corresponding contents in the scheduling method for storage resources, which are not described herein again.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flow chart of a method of scheduling storage resources according to an embodiment of the invention;
FIG. 2 is another flow chart of a method of scheduling storage resources according to an embodiment of the present invention;
FIG. 3 is another flow chart of a method of scheduling storage resources according to an embodiment of the present invention;
FIG. 4 is a block diagram illustrating a log storage system in an embodiment of the invention;
fig. 5 is a block diagram of a scheduling apparatus of storage resources according to an embodiment of the present invention;
fig. 6 is a schematic diagram of a hardware structure of an electronic device according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
According to the technical scheme, when the log writing rate is high, the log storage resources are scheduled based on the exhaustion time of the log storage resources, so that the log storage resources are prevented from being exhausted due to high-frequency log writing, the problem that a cloud host is down or logs which cannot be written are lost due to the exhaustion of the log storage resources is solved, and the log writing capacity of the cloud host is improved.
In accordance with an embodiment of the present invention, there is provided an embodiment of a method for scheduling storage resources, it should be noted that the steps illustrated in the flowchart of the accompanying drawings may be performed in a computer system such as a set of computer executable instructions, and that while a logical order is illustrated in the flowchart, in some cases the steps illustrated or described may be performed in an order different than that herein.
In this embodiment, a method for scheduling a storage resource is provided, which may be used in an electronic device adopting a log storage system, such as a mobile phone, a tablet computer, a server, and the like, fig. 1 is a flowchart of a method for scheduling a storage resource according to an embodiment of the present invention, and as shown in fig. 1, the flowchart includes the following steps:
and S11, acquiring the log storage resource and the log writing rate.
The log storage resource is a storage space used for storing logs, and specifically, a log storage system of the electronic device may include a plurality of storage pools, each storage pool includes one or more storage volumes, each log storage volume may store log information of all cloud hosts corresponding to one node, and log information corresponding to different cloud hosts is stored in different directories of the log storage volume.
The log write rate is the rate at which log information is written to the log storage resource, which can be characterized using the number of bytes written to the log storage resource per second. The log writing rate may be an average value of rates at which log information is written to the log storage resource within a certain period of time, for example, a log storage system in the electronic device may monitor rates at which log information is written to the log storage resource at various times in real time, and calculate the log writing rate accordingly; the log writing rate may also be a real-time rate at which the log information monitored at the current time is written to the log storage resource, and the log writing rate may also be obtained by other methods, which is not specifically limited herein.
S12, based on the log writing rate and the log storage resource, the exhaustion time of the log storage resource is calculated.
The exhaustion time is the time when the log storage resource is exhausted, and the log storage system in the electronic device can determine the current log storage space according to the log storage resource and estimate the exhaustion time of the log storage resource by combining the current log writing rate. For example, the current log storage space corresponding to the log storage resource is C0The log write rate is V0From this, the exhaustion time T ═ C can be estimated0/V0。
And S13, judging whether the depletion time is less than the preset time.
The preset time is the preset shortest time for the log storage resource to be exhausted, after the exhausted time is estimated, the exhausted time is compared with the preset time to determine the relationship between the exhausted time and the preset time, when the exhausted time is less than the preset time, the step S14 is executed, otherwise, no operation is executed.
S14, resource scheduling is carried out on the log storage resource.
When the exhaustion time is less than the preset time, it indicates that the log information is continuously written in high frequency, and at this time, the log storage system of the electronic device may schedule the log storage resource, for example, expand the log storage resource, or migrate the log storage resource to another storage pool, and the like.
According to the scheduling method of the storage resources, the log storage resources and the log writing rate are obtained, the exhaustion time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the exhaustion time is less than the preset time, the log storage resources are indicated to be exhausted, at this time, the resource scheduling is performed on the log storage resources, the risk of exhaustion of the log storage resources is eliminated, the problem that a cloud host is down or logs which cannot be written into the cloud host are lost due to exhaustion of the log storage resources is avoided, and the log writing capability of the cloud host is improved.
In this embodiment, a method for scheduling a storage resource is provided, which may be used in an electronic device adopting a log storage system, such as a mobile phone, a tablet computer, a server, and the like, and fig. 2 is a flowchart of a method for scheduling a storage resource according to an embodiment of the present invention, and as shown in fig. 2, the flowchart includes the following steps:
and S21, acquiring the log storage resource and the log writing rate. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
S22, based on the log writing rate and the log storage resource, calculating the exhaustion time of the log storage resource.
Specifically, the step S22 may include:
s221, obtaining the residual space capacity corresponding to the log storage resource.
And the residual space capacity corresponding to the log storage resource is the log storage space remained by the current log storage resource. The log storage system of the electronic device can count the space occupied by the existing log information in the current log storage resource, and calculate the difference between the total storage space and the space occupied by the existing log information by combining the total storage space of the log storage resource, so that the residual space capacity corresponding to the log storage resource can be obtained.
S222, calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
The log writing rate is the number of bytes written to the log storage resource per second, the remaining space capacity is divided by the log writing rate, the quotient between the remaining space capacity and the log writing rate can be calculated, and the quotient between the remaining space capacity and the log writing rate is determined as the exhaustion time of the log storage resource.
And S23, judging whether the depletion time is less than the preset time. For a detailed description, refer to the corresponding related description of the above embodiments, which is not repeated herein.
S24, when the exhausted time is less than the preset time, the resource scheduling is carried out to the log storage resource.
Specifically, the step S24 may include:
s241, obtaining the capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource.
The capacity expansion information is used to represent capacity expansion multiples of the log storage resource, for example, the capacity expansion information may be 1.5 times of capacity expansion, may be 2 times of capacity expansion, and may also be other capacity expansion multiples, which is not specifically limited here. The remaining space of the storage pool corresponding to the log storage resource is the current remaining storage space of the storage pool, and specifically, a log storage system in the electronic device may obtain the total space of the storage pool where the log storage resource is located, count the space occupied by the log storage resource, and then determine the remaining space of the storage pool according to the total space of the storage pool and the space occupied by the log storage resource.
And S242, performing resource scheduling on the log storage resource based on the capacity expansion information and the residual space of the storage pool.
The log storage system of the electronic device may determine whether the log storage resource meets the capacity expansion requirement according to the remaining space of the storage pool. When the residual space of the storage pool meets the capacity expansion requirement, capacity expansion is carried out on the log storage resource based on the capacity expansion information so as to realize resource scheduling of the log storage resource; when the residual space of the storage pool does not meet the capacity expansion requirement, the log storage resource can be subjected to thermal migration so as to realize resource scheduling of the log storage resource.
Specifically, as shown in fig. 3, the step S242 may include:
and S1, acquiring the current residual space of the storage pool after the capacity expansion of the log storage resources.
And the current residual space of the storage pool is the residual log storage space of the storage pool, and the log storage system of the electronic equipment expands the capacity of the log storage resource in the storage pool in which the log storage resource is located according to the capacity expansion information and calculates the current residual space of the storage pool after the capacity expansion of the log storage resource.
And S2, judging whether the current remaining space is larger than the preset space.
The preset space is a preset lowest remaining space threshold of the storage pool. The log storage system of the electronic device may compare the current remaining space of the storage pool determined by the log storage system with the preset space to determine a relationship between the current remaining space and the preset space. When the current remaining space is greater than the preset space, the step S3 is performed, otherwise, the step S4 is performed.
S3, capacity expansion is carried out on the log storage resource based on the capacity expansion information.
If the current remaining space is larger than the preset space, it indicates that the remaining space of the storage pool where the log storage resource is located meets the capacity expansion requirement of the log storage resource, and at this time, the log storage system can perform capacity expansion on the log storage resource according to the capacity expansion information. For example, the capacity expansion information is 1.5 times capacity expansion, and at this time, the journal storage system may expand the capacity of the journal storage resource in the current storage pool by 1.5 times of the journal storage resource.
S4, obtaining the storage state of each storage pool corresponding to the journal storage resource and the migration information of the journal storage resource.
The storage status of each storage pool is used to represent the storage information of each storage pool, such as the storage space left in each storage pool, and the estimated exhaustion time of the log storage resource corresponding to each storage pool at the current log writing rate. The migration information of the journal storage resource is used to hot-migrate the journal storage resource to other storage pools.
If the current remaining space is smaller than the preset space, the remaining space of the storage pool where the log storage resource is located does not meet the capacity expansion requirement of the log storage resource, and at the moment, the log storage resource can be thermally migrated to other storage pools to avoid that the log cannot be written when the log storage resource is exhausted.
S5, based on the storage status of each storage pool, judging whether the target migration storage pool exists in each storage pool.
The target migration storage pool is a migration target corresponding to the log storage resource, and the log storage system of the electronic device may determine whether the target migration storage pool exists in each storage pool according to the acquired storage state of each storage pool. When the target migration storage pool exists, step S6 is performed, otherwise step S7 is performed.
For example, the target migration storage pool may be the largest remaining space, or the longest storage resource exhaustion time estimated at the current log writing rate, and of course, the target migration storage pool may also be determined according to other information, which is not specifically limited herein and may be determined by a person skilled in the art according to actual needs.
Specifically, the step S5 may include:
(1) and sequencing the storage pools based on the storage states of the storage pools to obtain the sequencing result of the storage pools.
And sequencing all the storage pools according to the storage states of the storage pools to obtain the sequencing result of the storage pools. Taking the remaining space of the storage pools as an example, the log storage system of the electronic device is provided with a scheduling module, and the scheduling module can perform sequencing according to the remaining space of each storage pool from large to small to obtain a sequencing result of each storage pool.
(2) Whether a target migration storage pool exists in each storage pool is determined based on the sorting result.
And determining a target migration storage pool meeting the requirement from each storage pool according to the sequencing result of each storage pool. Taking the remaining space of the storage pools as an example, the scheduling module may find the storage pool with the largest remaining space from each storage pool according to the sorting result of each storage pool, and use the storage pool with the largest remaining space as the target migration storage pool.
And sequencing the storage pools according to the storage states of the storage pools to obtain sequencing results corresponding to the storage pools, and determining whether a target migration storage pool exists in each storage pool according to the sequencing results. According to the method, the target migration storage pool for migrating the log storage resource is determined from the storage pools, so that the log storage resource is prevented from being exhausted, and the utilization rate of the storage resource is improved.
And S6, migrating the log storage resources to the target migration storage pool based on the migration information.
When the target migration storage pool exists in each storage pool, the storage pool meeting the migration requirement exists, and at this time, the log storage resource can be thermally migrated to the target migration storage pool according to the migration information.
S7, obtaining new information of the storage pool and the migration information of the log storage resource.
When the target migration storage pool does not exist in each storage pool, it indicates that all storage pools do not exist in storage pools meeting the migration requirement, and at this time, new information of the storage pools can be acquired to newly create one storage pool to perform the migration of the log storage resource.
S8, generating a target storage pool corresponding to the storage pool new creation information based on the storage pool new creation information.
The target storage pool is a storage pool used for migrating the log storage resources, and when the target migration storage pool does not exist in each storage pool, a new storage pool can be created according to the storage pool new creation information, and the new storage pool is determined as the target storage pool.
And S9, migrating the journal storage resource to the target storage pool based on the migration information of the journal storage resource.
After the target storage pool is created, the log storage resources can be transferred to the target storage pool in a hot mode according to the transfer information of the log storage resources, so that the log storage resources are expanded, logs written by the cloud host are continuously stored in the target storage pool, and log loss or cloud host downtime caused by insufficient storage resources is avoided.
In the method for scheduling storage resources provided by this embodiment, the remaining space capacity corresponding to the log storage resource is obtained, a quotient between the remaining space capacity and the write rate is calculated, and the quotient between the remaining space capacity and the write rate is used as the exhaustion time of the log storage resource. By calculating the exhaustion time of the log storage resource, the time that the log storage resource is about to be exhausted can be determined, and the log storage resource is convenient to be scheduled in time.
By acquiring the capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource and scheduling the resource of the log storage resource based on the capacity expansion information and the residual space of the storage pool, the resource scheduling of the log storage resource can be performed in an actual use state, the problem of log storage resource exhaustion in high-frequency operation is avoided, and the log writing capability in high-frequency operation is ensured.
When the current residual space of the storage pool is larger than the preset space, carrying out capacity expansion on the log storage resource based on capacity expansion information; and when the current residual space of the storage pools is smaller than the preset space, performing resource scheduling on the log storage resources based on the storage state of each storage pool and the migration information of the log storage resources so as to ensure that the logs operated at high frequency can be continuously written into the log storage resources. The risk of log storage resource exhaustion is eliminated through a capacity expansion mode and a migration mode, and the reliable storage of the logs is guaranteed while the normal use of the cloud host is not influenced.
The embodiment of the present invention describes a method for scheduling storage resources by taking a specific log storage system as an example, where the log storage system includes: the distributed log storage system based on the Ceph, the log collection module, the monitoring module, the scheduling module and the elastic expansion module are structurally shown in fig. 4. The distributed log storage scheme divides log storage resources into different log storage resource pools, and each log storage resource pool is divided into different log storage volumes. Each log storage volume stores log information of all cloud hosts on one node, and different directories of the log storage volumes are stored according to different cloud hosts.
The log collection module is used for collecting all log information generated in the operation process of the cloud host, distinguishing the collected logs according to the nodes where the cloud host is located, then storing the logs into corresponding log storage volumes, and then storing the logs into different log catalogs according to different cloud hosts.
The monitoring module is used for monitoring the resource surplus condition of each log storage volume in each log storage resource pool in real time, recording the log writing rate in a current period of time, and estimating the depletion time T of the log storage volume. Further, the monitoring module may compare the exhaustion time T with a minimum threshold time Tmin specified by the cloud platform operation and maintenance personnel, and if T < Tmin, notify the scheduling module to perform a corresponding operation.
The scheduling module receives the information that the log storage volume is about to be used up, which is transmitted by the monitoring module, and then performs corresponding operations: calculating the residual capacity C of the log storage resource pool after the log storage volume is expanded, and if the residual capacity C is larger than the minimum residual capacity Cmin of the log storage resource pool, initiating an expansion instruction of the log storage volume to the elastic expansion module; if the residual capacity C is smaller than the minimum residual capacity Cmin of the log storage resource pool, sequencing all the log storage resource pools according to a preset rule, selecting one log storage resource pool as a target migration storage pool, and then initiating a log storage volume cross-storage resource pool hot migration instruction by the elastic expansion module; if all the log storage resource pools cannot be used as target migration storage pools of the log storage volumes, the elastic expansion module is informed to newly establish a log storage resource pool in the distributed log storage system. No matter what kind of operation is performed on the log storage volume, the scheduling module transmits the specific operation instruction to the elastic expansion module on the cloud host that generates the log.
After receiving the instruction sent by the scheduling module, the elastic expansion module can identify the received instruction so as to perform capacity expansion of a corresponding log storage volume, thermal migration across storage resource pools or operation of newly building a log storage resource pool. For a capacity expansion instruction of a log storage volume, the elastic expansion module directly applies for storage resources in a resource pool where the storage volume is located, expands the capacity of the storage volume, feeds back stock information of the storage volume and the storage resource pool to the monitoring module after capacity expansion is finished, and feeds back operation completion information to the scheduling module. For a thermal migration instruction of the log storage volume, the elastic expansion module performs thermal migration operation on the log storage volume according to target log storage resource pool information in the thermal migration instruction, capacity expansion operation can be performed on the log storage volume according to requirements of operation and maintenance personnel during thermal migration, and relevant information is fed back to the monitoring module and the scheduling module after cross-resource pool thermal migration of the log storage volume is completed. And for the newly-built log storage resource pool, the elastic expansion module is established according to the size of the resource pool specified by operation and maintenance personnel. And after the creation is finished, the information is fed back to the monitoring module and the scheduling module.
According to the log storage system provided by the embodiment, when the log storage resources are about to be exhausted, the log storage resources can be scheduled, the risk of exhausting the log storage resources is eliminated, the problem that the cloud host is down or logs which cannot be written into the cloud host are lost due to exhausting of the log storage resources is avoided, and the log writing capacity of the cloud host is improved. By monitoring and scheduling the log storage volume, the log is written in without upper limit to the maximum extent, and the log of the cloud host can be reliably stored.
In this embodiment, a scheduling apparatus for a storage resource is further provided, and the apparatus is used to implement the foregoing embodiments and preferred embodiments, and details of which have been already described are omitted. As used below, the term "module" may be a combination of software and/or hardware that implements a predetermined function. Although the means described in the embodiments below are preferably implemented in software, an implementation in hardware, or a combination of software and hardware is also possible and contemplated.
The present embodiment provides a scheduling apparatus for storage resources, as shown in fig. 5, including:
the obtaining module 31 is configured to obtain a log storage resource and a log writing rate. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
And the calculating module 32 is used for calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
And a judging module 33, configured to judge whether the depletion time is less than a preset time. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
And the scheduling module 34 is configured to perform resource scheduling on the log storage resource when the exhaustion time is less than the preset time. For a detailed description, reference is made to the corresponding related description of the above method embodiments, which is not repeated herein.
According to the scheduling device for the storage resources, the log storage resources and the log writing rate are obtained, the exhaustion time of the log storage resources is calculated based on the log writing rate and the log storage resources, if the exhaustion time is less than the preset time, the log storage resources are indicated to be exhausted, at this time, the resource scheduling is performed on the log storage resources, the risk of exhaustion of the log storage resources is eliminated, the problem that a cloud host is down or a log which cannot be written into the cloud host is lost due to exhaustion of the log storage resources is avoided, and the log writing capability of the cloud host is improved.
The memory resource scheduling means in this embodiment is in the form of a functional unit, where the unit refers to an ASIC circuit, a processor and a memory executing one or more software or fixed programs, and/or other devices capable of providing the above functions.
Further functional descriptions of the modules are the same as those of the corresponding embodiments, and are not repeated herein.
An embodiment of the present invention further provides an electronic device, which has the scheduling apparatus for storage resources shown in fig. 5.
Referring to fig. 6, fig. 6 is a schematic structural diagram of an electronic device according to an alternative embodiment of the present invention, and as shown in fig. 6, the electronic device may include: at least one processor 401, such as a CPU (Central Processing Unit), at least one communication interface 403, memory 404, and at least one communication bus 402. Wherein a communication bus 402 is used to enable connective communication between these components. The communication interface 403 may include a Display (Display) and a Keyboard (Keyboard), and the optional communication interface 403 may also include a standard wired interface and a standard wireless interface. The Memory 404 may be a high-speed RAM Memory (volatile Random Access Memory) or a non-volatile Memory (non-volatile Memory), such as at least one disk Memory. The memory 404 may optionally be at least one memory device located remotely from the processor 401. Wherein the processor 401 may be combined with the apparatus as described in fig. 5, the memory 404 stores an application program, and the processor 401 calls the program code stored in the memory 404 for performing any of the above-mentioned method steps.
The communication bus 402 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 402 may be divided into an address bus, a data bus, a control bus, and the like. For ease of illustration, only one thick line is shown in FIG. 6, but this is not intended to represent only one bus or type of bus.
The memory 404 may include a volatile memory (RAM), such as a random-access memory (RAM); the memory may also include a non-volatile memory (english: non-volatile memory), such as a flash memory (english: flash memory), a hard disk (english: hard disk drive, abbreviated: HDD) or a solid-state drive (english: SSD); the memory 404 may also comprise a combination of the above types of memory.
The processor 401 may be a Central Processing Unit (CPU), a Network Processor (NP), or a combination of a CPU and an NP.
The processor 401 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a Programmable Logic Device (PLD), or a combination thereof. The PLD may be a Complex Programmable Logic Device (CPLD), a field-programmable gate array (FPGA), a General Array Logic (GAL), or any combination thereof.
Optionally, the memory 404 is also used to store program instructions. Processor 401 may invoke program instructions to implement the method for scheduling storage resources as illustrated in the embodiments of fig. 1-3 of the present application.
Embodiments of the present invention further provide a non-transitory computer storage medium, where the computer storage medium stores computer-executable instructions, and the computer-executable instructions may execute the processing method of the scheduling method for storage resources in any method embodiment described above. The storage medium may be a magnetic Disk, an optical Disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a Flash Memory (Flash Memory), a Hard Disk (Hard Disk Drive, abbreviated as HDD), a Solid State Drive (SSD), or the like; the storage medium may also comprise a combination of memories of the kind described above.
Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations fall within the scope defined by the appended claims.
Claims (10)
1. A method for scheduling storage resources, comprising:
acquiring log storage resources and a log writing rate;
calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource;
judging whether the depletion time is less than a preset time or not;
and when the exhaustion time is less than the preset time, performing resource scheduling on the log storage resource.
2. The method of claim 1, wherein the resource scheduling the log storage resource comprises:
acquiring capacity expansion information of the log storage resource and the residual space of the storage pool corresponding to the log storage resource;
and performing resource scheduling on the log storage resource based on the capacity expansion information and the residual space of the storage pool.
3. The method of claim 2, wherein the resource scheduling the journal storage resource based on the capacity expansion information and the remaining space of the storage pool comprises:
acquiring the current residual space of the storage pool after the capacity expansion of the log storage resources;
judging whether the current residual space is larger than a preset space or not;
and when the current residual space is larger than the preset space, carrying out capacity expansion on the log storage resource based on the capacity expansion information.
4. The method of claim 3, wherein the resource scheduling for the journal storage resource based on the capacity expansion information and the remaining space of the storage pool further comprises:
when the current residual space is smaller than the preset space, acquiring the storage state of each storage pool corresponding to the log storage resource and the migration information of the log storage resource;
judging whether a target migration storage pool exists in each storage pool or not based on the storage state of each storage pool;
and when a target migration storage pool exists in the storage pools, migrating the log storage resources to the target migration storage pool based on the migration information.
5. The method of claim 4, further comprising:
when the target migration storage pool does not exist in each storage pool, acquiring new storage pool creation information and migration information of the log storage resource;
generating a target storage pool corresponding to the storage pool new construction information based on the storage pool new construction information;
migrating the journal storage resource to the target storage pool based on the migration information of the journal storage resource.
6. The method of claim 4, wherein said determining whether a target migration storage pool exists in said respective storage pools based on the storage status of said respective storage pools comprises:
sorting the storage pools based on the storage states of the storage pools to obtain sorting results of the storage pools;
determining whether a target migration storage pool exists in the respective storage pools based on the ranking result.
7. The method of any of claims 1-6, wherein calculating the exhaustion time of the log storage resource based on the log write rate and the log storage resource comprises:
acquiring the residual space capacity corresponding to the log storage resource;
and calculating the quotient between the residual space capacity and the log writing rate to obtain the exhaustion time of the log storage resource.
8. A scheduling apparatus for a storage resource, comprising:
the acquisition module is used for acquiring log storage resources and log writing rate;
the calculation module is used for calculating the exhaustion time of the log storage resource based on the log writing rate and the log storage resource;
the judging module is used for judging whether the depletion time is less than the preset time or not;
and the scheduling module is used for scheduling the resources of the log storage resources when the exhaustion time is less than the preset time.
9. An electronic device, comprising:
a memory and a processor, the memory and the processor being communicatively connected to each other, the memory having stored therein computer instructions, the processor executing the computer instructions to perform the method of scheduling of storage resources of any of claims 1-7.
10. A computer-readable storage medium storing computer instructions for causing a computer to perform the method of scheduling storage resources of any one of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600523.6A CN114461138A (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111600523.6A CN114461138A (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114461138A true CN114461138A (en) | 2022-05-10 |
Family
ID=81407612
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111600523.6A Pending CN114461138A (en) | 2021-12-24 | 2021-12-24 | Scheduling method and device of storage resources, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114461138A (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247347A (en) * | 2012-02-11 | 2013-08-14 | 三星电子株式会社 | Method and system for providing smart memory architecture |
| US20130297905A1 (en) * | 2012-03-29 | 2013-11-07 | International Business Machines Corporation | Dynamic reconfiguration of storage system |
| WO2016107167A1 (en) * | 2014-12-31 | 2016-07-07 | 中兴通讯股份有限公司 | Storage architecture creation method, storage access method and storage system |
| US20200104065A1 (en) * | 2018-09-28 | 2020-04-02 | International Business Machines Corporation | Storage capacity usage management |
| US20210034278A1 (en) * | 2019-07-30 | 2021-02-04 | Emc Ip Holding Company Llc. | Storage resource capacity prediction utilizing a plurality of time series forecasting models |
| US20210103399A1 (en) * | 2019-10-04 | 2021-04-08 | International Business Machines Corporation | Dynamic rebalancing of free space between storage pools |
-
2021
- 2021-12-24 CN CN202111600523.6A patent/CN114461138A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103247347A (en) * | 2012-02-11 | 2013-08-14 | 三星电子株式会社 | Method and system for providing smart memory architecture |
| US20130297905A1 (en) * | 2012-03-29 | 2013-11-07 | International Business Machines Corporation | Dynamic reconfiguration of storage system |
| WO2016107167A1 (en) * | 2014-12-31 | 2016-07-07 | 中兴通讯股份有限公司 | Storage architecture creation method, storage access method and storage system |
| US20200104065A1 (en) * | 2018-09-28 | 2020-04-02 | International Business Machines Corporation | Storage capacity usage management |
| US20210034278A1 (en) * | 2019-07-30 | 2021-02-04 | Emc Ip Holding Company Llc. | Storage resource capacity prediction utilizing a plurality of time series forecasting models |
| US20210103399A1 (en) * | 2019-10-04 | 2021-04-08 | International Business Machines Corporation | Dynamic rebalancing of free space between storage pools |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6949045B2 (en) | Distributed cluster type training method and equipment | |
| JP6248560B2 (en) | Management program, management method, and management apparatus | |
| CN106452818B (en) | Resource scheduling method and system | |
| JPWO2008102739A1 (en) | Virtual server system and physical server selection method | |
| JP6191691B2 (en) | Abnormality detection apparatus, control method, and program | |
| US20120221730A1 (en) | Resource control system and resource control method | |
| JP2013509658A (en) | Allocation of storage memory based on future usage estimates | |
| US20150370587A1 (en) | Computer-readable recording medium having stored therein outputting program, output apparatus and outputting method | |
| CN105991458B (en) | Load balancing method and load balancing device | |
| CN106027595A (en) | Access log processing method and system for CDN node | |
| KR102406580B1 (en) | Online Flash Resource Allocation Method and System Based on a TCO Model | |
| CN111966449B (en) | Virtual machine backup management method, system, terminal and storage medium | |
| CN107656807A (en) | The automatic elastic telescopic method and device of a kind of virtual resource | |
| US20200272526A1 (en) | Methods and systems for automated scaling of computing clusters | |
| JP2019008454A (en) | Information processing system and resource allocation method | |
| KR101152315B1 (en) | Method for virtuallizing a server for network video recorder | |
| JP6940761B2 (en) | Information processing equipment, virtual machine monitoring programs, and information processing systems | |
| KR101608859B1 (en) | Cloud based intelligent power control system | |
| CN107426012B (en) | Fault recovery method and device based on super-fusion architecture | |
| CN111400241B (en) | Data reconstruction method and device | |
| US20150195174A1 (en) | Traffic data collection apparatus, traffic data collection method and program | |
| CN114461138A (en) | Scheduling method and device of storage resources, electronic equipment and readable storage medium | |
| CN108604231A (en) | Mirror processing method and computing device | |
| CN110493071B (en) | Message system resource balancing device, method and equipment | |
| JP5867499B2 (en) | Virtual server system, management server device, and system management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |