CN113364878A - Data scheduling method and device, electronic device and storage medium - Google Patents
Data scheduling method and device, electronic device and storage medium Download PDFInfo
- Publication number
- CN113364878A CN113364878A CN202110675130.5A CN202110675130A CN113364878A CN 113364878 A CN113364878 A CN 113364878A CN 202110675130 A CN202110675130 A CN 202110675130A CN 113364878 A CN113364878 A CN 113364878A
- Authority
- CN
- China
- Prior art keywords
- data
- cluster
- prediction
- expected
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1001—Protocols in which an application is distributed across nodes in the network for accessing one among a plurality of replicated servers
Abstract
The disclosure provides a data scheduling method, relates to the technical field of cloud computing, and particularly relates to the technical field of distributed storage. The specific implementation scheme is as follows: obtaining an expected traffic distribution value of the target data among a plurality of storage devices included in the cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period; calculating a cluster deviation degree according to the expected flow distribution value and the actual flow distribution values of the plurality of storage devices in the current time period; comparing the cluster deviation degree with a preset deviation threshold value; and scheduling the actual flow distribution of the plurality of storage devices in the current time period so that the cluster deviation degree is less than or equal to the predetermined deviation threshold value when the cluster deviation degree is greater than the predetermined deviation threshold value. The present disclosure also provides a data scheduling apparatus, an electronic device, a storage medium storing computer instructions, and a computer program product.
Description
Technical Field
The present disclosure relates to the field of cloud computing, and further relates to the field of distributed storage technologies. And more particularly, to a data scheduling method, a data scheduling apparatus, an electronic device, a non-transitory computer readable storage medium having computer instructions stored thereon, and a computer program product.
Background
The fully hosted storage service refers to a storage cluster hosted by a third party, where a user stores data, rather than a private service cluster maintained by the user. The full-managed storage service has the advantages of flexible access, strong data disaster tolerance capability, convenient service expansion and the like.
Disclosure of Invention
The disclosure provides a data scheduling method, device, equipment, storage medium and computer program product.
According to a first aspect, there is provided a data scheduling method, comprising: obtaining an expected traffic distribution value of the target data among a plurality of storage devices included in the cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period; calculating a cluster deviation degree according to the expected flow distribution value and the actual flow distribution values of the plurality of storage devices in the current time period; comparing the cluster deviation degree with a preset deviation threshold value; and scheduling the actual flow distribution of the plurality of storage devices in the current time period so that the cluster deviation degree is less than or equal to the predetermined deviation threshold value when the cluster deviation degree is greater than the predetermined deviation threshold value.
According to a second aspect, there is provided a data scheduling apparatus comprising: the prediction module is used for obtaining an expected flow distribution value of the target data among a plurality of storage devices included in the cluster during a prediction time period according to a flow distribution prediction value of the target data during the prediction time period; a calculating module, configured to calculate a cluster deviation degree according to the expected flow distribution value and actual flow distribution values of the plurality of storage devices in a current time period; the comparison module is used for comparing the cluster deviation degree with a preset deviation threshold value; and a scheduling module, configured to schedule actual traffic distribution of the plurality of storage devices in a current time period so that the cluster deviation degree is less than or equal to the predetermined deviation threshold value when the cluster deviation degree is greater than the predetermined deviation threshold value.
According to a third aspect, there is provided an electronic device comprising: at least one processor; and a memory communicatively coupled to the at least one processor; the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method provided by the embodiment of the disclosure.
According to a fourth aspect, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform a method provided by an embodiment of the disclosure.
According to a fifth aspect, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method provided by embodiments of the present disclosure.
It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.
Drawings
The drawings are included to provide a better understanding of the present solution and are not to be construed as limiting the present disclosure. Wherein:
fig. 1 is a schematic diagram of an exemplary system architecture to which a data scheduling method or apparatus may be applied, according to one embodiment of the present disclosure;
FIG. 2 is a flow diagram of a method of data scheduling according to one embodiment of the present disclosure;
FIG. 3 is a flow chart diagram of a method of data scheduling according to another embodiment of the present disclosure;
FIG. 4 is a flow chart diagram of a method of data scheduling according to another embodiment of the present disclosure;
FIG. 5A is a schematic diagram of storage devices of a data scheduling method according to one embodiment of the present disclosure before scheduling;
FIG. 5B is a schematic diagram of scheduled storage devices according to an embodiment of the present disclosure;
fig. 6 is a block diagram of a data scheduling apparatus according to an embodiment of the present disclosure;
fig. 7 is a block diagram of an electronic device of a data scheduling method according to one embodiment of the present disclosure.
Detailed Description
Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
After a large amount of services access a storage cluster, the daily traffic of the services, the peak traffic during peak periods, the cross-cluster backup traffic, the traffic required for recovering the fault node of the cluster, and the scheduling traffic caused by the cluster load balancing may cause a complex traffic management problem.
The current approach to data allocation and traffic management in a fully hosted storage service is as follows.
The corresponding scheduling policy may be arranged by cluster type. For example, the clusters are classified according to the service mode, and the clusters are split according to the service characteristics of each classification, for example, the clusters are split into low-throughput high-storage clusters, high-throughput clusters, and the like. This type of approach is common in small-scale clustering and various types of customized clustering. This approach is mainly faced with a fixed traffic scenario. However, in actual use, service usage scenarios are often changed, for example, when the service is just on line, the service is a low throughput scenario, but as the service user volume increases, the service is changed into a high throughput scenario; similarly, the service can be adjusted along the product line from a high-throughput scene to a low-throughput scene. When the service flow changes, the original cluster scheduling strategy is used, which may cause stability problems or resource waste.
Storage resources may be reserved. For example, in a private cloud-based fully-managed storage service scenario, cluster peak traffic may be recorded, redundant resources may be reserved in proportion, and peak situations may be dealt with. Or, in a public cloud-based fully managed storage service scenario, storage resources are normalized to Capacity Units (CUs) according to storage engine capabilities, for example, 1CU ═ 1GB storage, write requests of not more than 4KB once per second and read requests of not more than 4KB 2 times per second, and resources are reserved for services with CUs as a basic unit. However, the above solution only focuses on solving the stability problem caused by the short resource board, and does not consider the uneven use of the storage resource by the cluster. Therefore, the above scheme not only can cause a large amount of resource fragments in the cluster, but also can cause serious deviation of hardware service life in the cluster, for example, the disk wear degree of the node bearing high-frequency reading and writing is far greater than that of the node bearing a large amount of cold data storage, thereby affecting the overall stability of the cluster.
Fig. 1 is a schematic diagram of an exemplary system architecture to which a data scheduling method or apparatus may be applied, according to one embodiment of the present disclosure.
As shown in fig. 1, an exemplary system architecture 100 of the application data scheduling method or apparatus includes a scheduling task management module 101, a cluster control module 102, a metadata management module 103, and a storage cluster formed by a plurality of storage devices (storage device 1104 to storage device N106).
The scheduling task management module 101 triggers adjustment of the binding relationship between the data and the storage device according to the cluster load condition, and tracks the execution status of the adjustment task to ensure that the adjustment task is completed finally.
The cluster control module 102 is responsible for managing the storage devices of the entire storage cluster, and when a single storage device fails, the cluster control module 102 migrates the failed node and joins the non-failed storage device. And the cluster control module is also responsible for responding to requests from the scheduled task management module 101 to adjust the routing relationship between the data and the storage devices.
The metadata management module 103 records global data distribution, such as routing relationships between the business data and the storage devices. The metadata management module 103 may not modify the data distribution autonomously, but the cluster control module actively or passively responds to external changes to adjust the routing relationship between the service data and the storage device in the metadata management module 103.
Storage device 1104, storage device 2105, storage device N106, form a storage cluster for storing data. Wherein N is more than 2 and is a positive integer. In the above example, N is equal to 6.
Fig. 2 is a flow chart of a data scheduling method according to one embodiment of the present disclosure.
As shown in fig. 2, the data scheduling method 200 may include operations S210 to S240.
In operation S210, a desired traffic distribution value of the target data among a plurality of storage devices included in the cluster during the prediction period is obtained according to the traffic distribution prediction value of the target data during the prediction period.
According to the embodiment of the present disclosure, the traffic distribution prediction value includes a predicted write data amount, read data amount, and storage space usage amount per storage device.
For example, the predicted flow distribution value of each storage device may be represented in the form of a triplet (r, w, s), where r is the predicted write data size of the storage device, w is the predicted read data size of the storage device, and s is the predicted storage space usage of the storage device
In one example, the predicted value of the flow distribution of the target data in the prediction period may be obtained, for example, by using a time-series prediction tool Prophet.
For example, a certain time stamp sequence ds and an actual traffic distribution data set may be used as input to the Prophet. For example, the actual traffic distribution data of 300 days of a storage cluster may be collected, each day corresponds to a timestamp, and a timestamp sequence may be obtained by sequentially arranging; the actual traffic distribution data for each day corresponds to a timestamp, forming a traffic distribution data set. And taking the timestamp sequence and the corresponding flow distribution data set as the input of the Prophet, and predicting the flow distribution predicted value in the prediction period. Those skilled in the art can understand that other timing prediction methods can be used to obtain the predicted value of the traffic distribution in the prediction period, and are not described herein again.
According to the embodiment of the present disclosure, a dynamic programming method may be used to obtain an expected traffic distribution value of target data among a plurality of storage devices included in a cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period.
According to an embodiment of the present disclosure, the expected traffic distribution value includes an expected write data amount, an expected read data amount, and an expected storage space usage amount of each storage device in the prediction period.
For example, the flow distribution prediction value of each storage device is represented in a form of a triplet (r, w, s), and the flow distribution prediction value is used as an input of the dynamic programming method, so as to obtain an expected flow distribution value (e (r), e (w), e (s)) of the target data among each storage device included in the cluster during a prediction period represented in the triplet form, where e (r) is an expected write data amount of one storage device during the prediction period, e (w) is an expected read data amount of one storage device during the prediction period, and e(s) is an expected storage space usage amount of one storage device during the prediction period.
In operation S220, a cluster deviation degree is calculated according to the desired flow distribution value and the actual flow distribution values of the plurality of storage devices in the current period.
According to the embodiment of the present disclosure, the degree of deviation of the expected flow distribution value from the actual flow distribution value may be taken as the cluster degree of deviation.
For example, the expected write data amount E (r) and the actual write data amount r can be calculatedrealCan calculate the expected read data amount E (w) and the actual read data amount wrealCan calculate the expected storage space writing quantity E(s) and the actual storage space writing quantity srealThe degree of deviation can be variance, difference, deviation, standard deviation, etc. Further operations, such as summing, averaging, maximizing, etc., may be performed on the plurality of degrees of deviation as the cluster degree of deviation.
According to the disclosed embodiments, the cluster deviation degree may be periodically calculated.
For example, the cluster deviation degree is calculated every 7 or 15 days.
In operation S230, the cluster deviation degree is compared with a predetermined deviation threshold.
For example, comparing the cluster deviation degree with a predetermined deviation threshold value, it can be obtained that the cluster deviation degree is greater than the predetermined deviation threshold value or the cluster deviation degree is less than or equal to the predetermined deviation threshold value. In one example, the deviation threshold may be set at 20%. It will be appreciated by those skilled in the art that other values of the deviation threshold may be set depending on actual requirements.
In operation S240, in a case that the cluster deviation degree is greater than the predetermined deviation threshold, the actual flow distribution of the plurality of storage devices in the current period is scheduled so that the cluster deviation degree is less than or equal to the predetermined deviation threshold.
For example, the actual traffic distribution of the plurality of storage devices in the current period may be scheduled immediately, or the actual traffic distribution of the plurality of storage devices in the current period may be scheduled at a predetermined time.
By the embodiment of the disclosure, the cluster management cost is simplified, the service data use mode can be changed in a self-adaptive manner, the overall resource utilization rate of the storage cluster is improved, and the problem of cluster resource fragmentation is relieved.
Fig. 3 is a flow chart of a data scheduling method according to another embodiment of the present disclosure.
As shown in fig. 3, the data scheduling method 320 may calculate a cluster deviation degree according to the expected traffic distribution value and the actual traffic distribution values of the plurality of storage devices in the current time period. The data scheduling method 320 may include operations S321 to S322.
In operation S321, for each storage device, a mean square error between the expected write data amount, the expected read data amount, and the expected storage space usage amount of the prediction period and the actual write data amount, the actual read data amount, and the actual storage space usage amount of the current period is calculated, respectively, to obtain a write data amount mean square error, a read data amount mean square error, and a storage space usage amount mean square error.
For example, the write data amount mean square error σ (r) can be calculated by the following equation:
for example, the read data amount mean square error σ (w) can be calculated by the following equation:
for example, the mean square error σ(s) of the memory space usage amount can be calculated by the following equation:
in operation S322, the maximum value of the obtained mean square error of the write data amount, the mean square error of the read data amount, and the mean square error of the storage space usage amount is used as the degree of cluster deviation.
For example, the maximum value among a plurality of σ (r), a plurality of σ (w), and a plurality of σ(s) is the cluster deviation degree Dcluster。
By the embodiment of the disclosure, the calculated cluster deviation degree can be used as a reliable basis for establishing the scheduling task.
Fig. 4 is a flowchart of a data scheduling method according to another embodiment of the present disclosure.
As shown in fig. 4, the data scheduling method 440 may schedule the actual traffic distribution of the plurality of storage devices in the current time period so that the cluster deviation degree is equal to or less than the predetermined deviation threshold value, when the cluster deviation degree is greater than the predetermined deviation threshold value. The data scheduling method 440 may include operations S441 through S443.
In operation S441, a scheduling task for adjusting an actual traffic distribution among the plurality of storage devices for the target data is created.
According to an embodiment of the present disclosure, the scheduling task includes: and adjusting the data flow among the plurality of storage devices according to the rated data flow of each storage device so that the data flow of at least one storage device in the plurality of storage devices reaches the rated data flow of the at least one storage device.
For example, in a period t1, first data is stored in the first storage device, and the flow rate of the first data is a MB; the second storage device stores second data, and the flow rate of the second data is b MB. In a period t2, first data are stored in the first storage device, and the flow rate of the first data is c MB; the second storage device stores second data, and the flow rate of the second data is d MB. The rated data flow of the first storage device is e MB, e is more than or equal to a + b, and e is more than or equal to c + d. Then the scheduling task may be: the second data is transferred to the first storage device. After the scheduling task is completed, the first storage device stores first data and second data, and in a time period t1, the flow rate of the first data is a MB, and the flow rate of the second data is b MB; during the period t2, the flow rate of the first data is c MB, and the flow rate of the second data is d MB. The second storage device may be on standby or may be used to store other data.
In operation S442, a time for executing the scheduling task is determined according to the current total data traffic of the cluster.
For example, if the current total data traffic is high, the scheduling task may be executed later to avoid the peak traffic.
For example, if the current total data traffic is low, the scheduling task may be executed immediately or delayed,
in operation S443, in a case where it is determined that the current total data traffic of the cluster is equal to or less than a predetermined data traffic threshold, the scheduling task is executed.
For example, before execution, it is determined whether the current total data traffic is small to ensure that peak traffic is avoided. Normal operation of the storage cluster can be ensured.
The large cluster deviation degree is a predetermined deviation threshold value, which may be caused by the large variation of the service user quantity, the need of data backup and the like in a prediction period, for example, the prediction period is a sales promotion period routine for the shopping platform every year. Scheduling the actual traffic distribution of the plurality of storage devices over the current time period may result in a change in cluster bias. Scheduling traffic itself also generates a large amount of traffic and also consumes memory resources. Before the prediction period actually comes, the stability of the storage cluster can be fully guaranteed by adjusting the actual flow distribution.
Through the embodiment of the disclosure, the scheduling task can be executed in the valley-peak period, the resource utilization rate of the storage device is improved, and the problem of performance degradation caused by the simultaneous occurrence of a large flow of a plurality of services on the same storage device can be avoided.
Fig. 5A is a schematic diagram of storage devices of a data scheduling method according to an embodiment of the present disclosure before scheduling.
As shown in fig. 5A, in an example, a storage cluster includes storage a 501, storage B502, storage C503, storage D504.
At time t1, data a 505 is stored in the storage device a 501, and the actual read data amount of the data a 505 is x1MB (MB). The storage device B502 stores data B506 therein, and the actual read data amount of the data B506 is y1MB (MB). Data C507 is stored in the storage device C503, and the actual read data amount of the data C507 is z1MB (MB). Data D508 is stored in the storage device D504, and the actual read data amount of the data D508 is v1 MB。
The data a 505, the data B506, the data C507, and the data D508 are used as target data, and the predicted value of the flow distribution in the prediction period t2_ prediction according to the target data is predicted. For example, the predicted read data amount of data A505 is x2_predictionPredicted read data amount of MB, data B506 is y2_predictionPredicted read data amount of MB, data C507 is z2_predictionPredicted read data amount of MB, data D508 is v2_predictionMB (MB). Thereby obtaining a desired flow distribution value of the above-mentioned target data among the plurality of storage devices included in the cluster during the prediction period, for example, a desired read data amount of the data a 505 is E (x)2_prediction) MB, data B506 having a desired read data amount E (y)2_prediction) MB, data C507, the expected read data amount is E (z)2_prediction) MB, the expected read data amount of data D508 is E (v)2_prediction)MB。
In the embodiment of the present disclosure, the degree of deviation of the expected flow distribution value from the actual flow distribution value is calculated as the cluster degree of deviation. For example, the mean square error between the expected read data amount and the actual read data amount may be calculated to obtain the degree of deviation σ (x) of the storage device a 501, the degree of deviation σ (y) of the storage device B502, the degree of deviation σ (z) of the storage device C503, and the degree of deviation σ (v) of the storage device D504, and the maximum value thereof may be the cluster deviation degree. If the cluster deviation degree is greater than the predetermined deviation threshold, the data stored in storage apparatus a 501, storage apparatus B502, storage apparatus C503, and storage apparatus D504 may be scheduled so that the cluster deviation degree is equal to or less than the predetermined deviation threshold.
Fig. 5B is a schematic diagram of scheduled storage devices according to an embodiment of the present disclosure.
As shown in FIG. 5B, at time t2 after the scheduling operation occurs, data B506 is transferred to storage device A501, where the actual read data amount of storage device A is (x)2+y2) MB (MB). Data C507 is still stored in the storage device C503, and at time t2, the read data amount of data C is z2MB (MB). Data D508 is still stored in the storage device D504, and at time t2, the read data amount of data D is v2 MB。
It should be appreciated that transferring data B506 to storage device A501 may cause storage device B to be idle, which may help reduce cluster bias.
It should be appreciated that scheduling the actual traffic distribution may be transferring all or part of the data stored in one storage device to another storage device. For example, all or a portion of the data C507 may be transferred to one or more of the storage devices A, B or D.
It should be understood that the storage cluster in the embodiments of the present disclosure contains 4 storage devices. In other embodiments, the storage cluster may set m storage devices as needed, where m > 1, and m is a positive integer.
Fig. 6 is a block diagram of a data scheduling apparatus according to an embodiment of the present disclosure.
As shown in fig. 6, the data scheduling apparatus may include a prediction module 610, a calculation module 620, a comparison module 630, and a scheduling module 640.
The prediction module 610 is configured to obtain an expected traffic distribution value of the target data among a plurality of storage devices included in the cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period.
A calculating module 620, configured to calculate a cluster deviation degree according to the expected flow distribution value and the actual flow distribution values of the plurality of storage devices in the current time period.
A comparing module 630, configured to compare the cluster deviation degree with a predetermined deviation threshold.
A scheduling module 640, configured to schedule actual traffic distribution of the plurality of storage devices in a current time period when the cluster deviation degree is greater than the predetermined deviation threshold, so that the cluster deviation degree is smaller than or equal to the predetermined deviation threshold.
As an alternative embodiment of the present disclosure, the traffic distribution prediction value includes a predicted write data amount, read data amount, and storage space usage amount for each storage device.
As an alternative embodiment of the present disclosure, the expected traffic distribution value includes an expected write data amount, an expected read data amount, and an expected storage space usage amount of each storage device in a prediction period; wherein, the calculation module includes: the mean square error calculation submodule is used for respectively calculating the mean square errors between the expected writing data volume, the expected reading data volume and the expected storage space usage of the prediction time period and the actual writing data volume, the actual reading data volume and the actual storage space usage of the current time period aiming at each storage device to obtain the mean square error of the writing data volume, the mean square error of the reading data volume and the storage space usage; and the cluster deviation degree calculation submodule is used for taking the maximum value of the obtained mean square error of the write-in data quantity, the obtained mean square error of the read data quantity and the obtained mean square error of the use quantity of the storage space as the cluster deviation degree.
As an optional embodiment of the present disclosure, the scheduling module includes: a creating submodule configured to create a scheduling task, where the scheduling task is used to adjust actual traffic distribution among the plurality of storage devices for the target data; and the determining submodule is used for determining the time for executing the scheduling task according to the current total data traffic of the cluster.
As an optional embodiment of the present disclosure, the scheduling module further includes: and the execution submodule is used for executing the scheduling task under the condition that the current total data traffic of the cluster is determined to be less than or equal to a preset data traffic threshold.
As an optional embodiment of the present disclosure, the scheduling task includes: and adjusting the data flow among the plurality of storage devices according to the rated data flow of each storage device so that the data flow of at least one storage device in the plurality of storage devices reaches the rated data flow of the at least one storage device.
As an optional embodiment of the present disclosure, the prediction module includes: and the expected flow prediction sub-module is used for obtaining an expected flow distribution value of the target data among a plurality of storage devices included in the cluster during the prediction period according to the flow distribution prediction value of the target data in the prediction period by using a dynamic programming method.
As an optional embodiment of the present disclosure, the system further includes a flow distribution prediction module, configured to obtain a predicted flow distribution value of the target data in a prediction time period by using a time sequence prediction tool Prophet.
It should be understood that the embodiments of the apparatus part of the present disclosure are the same as or similar to the embodiments of the method part of the present disclosure, and the technical problems to be solved and the technical effects to be achieved are also the same as or similar to each other, and the detailed description of the present disclosure is omitted.
The present disclosure also provides an electronic device and a readable storage medium according to an embodiment of the present disclosure.
Fig. 7 is a block diagram of an electronic device of a data scheduling method according to one embodiment of the present disclosure. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.
As shown in fig. 7, the device 700 comprises a computing unit 701, which may perform various suitable actions and processes according to a computer program stored in a Read Only Memory (ROM)702 or a computer program loaded from a storage unit 708 into a Random Access Memory (RAM) 703. In the RAM703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM703 are connected to each other by a bus 704. An input/output (I/O) interface 705 is also connected to bus 704.
Various components in the device 700 are connected to the I/O interface 705, including: an input unit 706 such as a keyboard, a mouse, or the like; an output unit 707 such as various types of displays, speakers, and the like; a storage unit 708 such as a magnetic disk, optical disk, or the like; and a communication unit 709 such as a network card, modem, wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information/data with other devices via a computer network, such as the internet, and/or various telecommunication networks.
Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.
The computer system may include a client and a server. A user terminal and a server are generally remote from each other and typically interact through a communication network. The relationship of user side and server arises by virtue of computer programs running on the respective computers and having a user side-server relationship to each other.
According to the technical scheme of the embodiment of the disclosure, the cluster management cost is simplified, the service data use mode change can be self-adapted, the overall resource utilization rate of the storage cluster is improved, and the problem of cluster resource fragments is eliminated.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.
The above detailed description should not be construed as limiting the scope of the disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present disclosure should be included in the scope of protection of the present disclosure.
Claims (19)
1. A method of data scheduling, comprising:
obtaining an expected traffic distribution value of target data among a plurality of storage devices included in a cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period;
calculating a cluster deviation degree according to the expected flow distribution value and the actual flow distribution values of the plurality of storage devices in the current time period;
comparing the cluster degree of deviation to a predetermined deviation threshold; and
and in the case that the cluster deviation degree is greater than the preset deviation threshold value, scheduling the actual flow distribution of the plurality of storage devices in the current time period so that the cluster deviation degree is smaller than or equal to the preset deviation threshold value.
2. The method of claim 1, wherein the traffic distribution prediction value comprises a predicted amount of write data, read data, and storage space usage per storage device.
3. The method of claim 1 or 2, wherein the expected traffic profile values comprise an expected amount of write data, an expected amount of read data, and an expected amount of storage space usage for each storage device over a predicted period of time;
wherein said calculating a cluster deviation degree according to the expected distribution value and the actual flow distribution values of the plurality of storage devices in the current time period comprises:
for each storage device, respectively calculating the mean square deviations between the expected write data volume, the expected read data volume and the expected storage space usage of the prediction time period and the actual write data volume, the actual read data volume and the actual storage space usage of the current time period to obtain a write data volume mean square deviation, a read data volume mean square deviation and a storage space usage mean square deviation; and
and taking the maximum value of the obtained mean square error of the write data quantity, the mean square error of the read data quantity and the mean square error of the use quantity of the storage space as the cluster deviation degree.
4. The method of claim 1, wherein scheduling the actual traffic distribution of the plurality of storage devices over the current time period such that the cluster divergence is less than or equal to the predetermined threshold in the event that the cluster divergence is greater than the predetermined threshold comprises:
creating a scheduling task, wherein the scheduling task is used for adjusting actual flow distribution among the target data and the plurality of storage devices; and
and determining the time for executing the scheduling task according to the current total data traffic of the cluster.
5. The method of claim 4, wherein the scheduling task is performed upon determining that a current total data traffic of the cluster is less than or equal to a predetermined data traffic threshold.
6. The method of claim 4 or 5, wherein the scheduling tasks comprises:
and adjusting the data flow among the plurality of storage devices according to the rated data flow of each storage device, so that the data flow of at least one storage device in the plurality of storage devices reaches the rated data flow of the at least one storage device.
7. The method of any of claims 1-6, wherein deriving an expected traffic distribution value for target data among a plurality of storage devices included in a cluster during a prediction period from a traffic distribution prediction value for the target data during the prediction period comprises:
obtaining an expected traffic distribution value of the target data among a plurality of storage devices included in the cluster during a prediction period according to a traffic distribution prediction value of the target data during the prediction period using a dynamic programming method.
8. The method according to any one of claims 1-6, further comprising deriving a predicted value of the flow distribution of the target data over a predicted period of time using a time series prediction tool Prophet.
9. A data scheduling apparatus, comprising:
the prediction module is used for obtaining an expected flow distribution value of the target data among a plurality of storage devices included in the cluster during a prediction time period according to a flow distribution prediction value of the target data during the prediction time period;
the calculation module is used for calculating the cluster deviation degree according to the expected flow distribution value and the actual flow distribution value of the plurality of storage devices in the current time period;
a comparison module for comparing the cluster deviation degree with a predetermined deviation threshold;
and the scheduling module is used for scheduling the actual flow distribution of the plurality of storage devices in the current time period under the condition that the cluster deviation degree is greater than the preset deviation threshold value, so that the cluster deviation degree is smaller than or equal to the preset deviation threshold value.
10. The apparatus of claim 9, wherein the traffic distribution prediction value comprises a predicted amount of write data, read data, and storage space usage per storage device.
11. The apparatus of claim 9 or 10, wherein the expected traffic profile value comprises an expected amount of write data, an expected amount of read data, and an expected amount of storage space usage for each storage device over a prediction period;
wherein, the calculation module includes: the mean square error calculation submodule is used for respectively calculating the mean square errors between the expected writing data volume, the expected reading data volume and the expected storage space usage of the prediction time period and the actual writing data volume, the actual reading data volume and the actual storage space usage of the current time period aiming at each storage device to obtain the mean square error of the writing data volume, the mean square error of the reading data volume and the storage space usage; and
and the cluster deviation degree calculation submodule is used for taking the maximum value of the obtained mean square error of the write-in data quantity, the obtained mean square error of the read data quantity and the obtained mean square error of the use quantity of the storage space as the cluster deviation degree.
12. The apparatus of claim 9, wherein the scheduling module comprises:
a creating submodule, configured to create a scheduling task, where the scheduling task is used to adjust actual traffic distribution among the plurality of storage devices according to the target data; and
and the determining submodule is used for determining the time for executing the scheduling task according to the current total data traffic of the cluster.
13. The apparatus of claim 12, wherein the scheduling module further comprises: and the execution submodule is used for executing the scheduling task under the condition that the current total data traffic of the cluster is determined to be less than or equal to a preset data traffic threshold.
14. The apparatus of claim 12 or 13, the scheduling task comprising: and adjusting the data flow among the plurality of storage devices according to the rated data flow of each storage device, so that the data flow of at least one storage device in the plurality of storage devices reaches the rated data flow of the at least one storage device.
15. The apparatus of any of claims 9-13, wherein the prediction module comprises:
and the expected flow prediction sub-module is used for obtaining an expected flow distribution value of the target data among a plurality of storage devices included in the cluster during the prediction period according to the flow distribution prediction value of the target data during the prediction period by using a dynamic programming method.
16. The apparatus of any of claims 9-13, further comprising:
and the flow distribution prediction module is used for obtaining a flow distribution prediction value of the target data in a prediction time period by using a time sequence prediction tool Prophet.
17. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.
18. A non-transitory computer readable storage medium having stored thereon computer instructions for causing the computer to perform the method of any one of claims 1-8.
19. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110675130.5A CN113364878B (en) | 2021-06-17 | 2021-06-17 | Data scheduling method and device, electronic device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110675130.5A CN113364878B (en) | 2021-06-17 | 2021-06-17 | Data scheduling method and device, electronic device and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113364878A true CN113364878A (en) | 2021-09-07 |
| CN113364878B CN113364878B (en) | 2022-11-29 |
Family
ID=77535101
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110675130.5A Active CN113364878B (en) | 2021-06-17 | 2021-06-17 | Data scheduling method and device, electronic device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113364878B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108259376A (en) * | 2018-04-24 | 2018-07-06 | 北京奇艺世纪科技有限公司 | The control method and relevant device of server cluster service traffics |
| CN109194539A (en) * | 2018-08-13 | 2019-01-11 | 中国平安人寿保险股份有限公司 | Data management-control method, device, computer equipment and storage medium |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| US20190253490A1 (en) * | 2016-10-31 | 2019-08-15 | Huawei Technologies Co., Ltd. | Resource load balancing control method and cluster scheduler |
| CN110198274A (en) * | 2019-06-21 | 2019-09-03 | 苏宁云计算有限公司 | A kind of control method and its system of group flow |
| CN111327461A (en) * | 2020-01-23 | 2020-06-23 | 华为技术有限公司 | Domain name management method, device, equipment and medium based on CDN system |
-
2021
- 2021-06-17 CN CN202110675130.5A patent/CN113364878B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20190253490A1 (en) * | 2016-10-31 | 2019-08-15 | Huawei Technologies Co., Ltd. | Resource load balancing control method and cluster scheduler |
| CN108259376A (en) * | 2018-04-24 | 2018-07-06 | 北京奇艺世纪科技有限公司 | The control method and relevant device of server cluster service traffics |
| CN109194539A (en) * | 2018-08-13 | 2019-01-11 | 中国平安人寿保险股份有限公司 | Data management-control method, device, computer equipment and storage medium |
| CN110086649A (en) * | 2019-03-19 | 2019-08-02 | 深圳壹账通智能科技有限公司 | Detection method, device, computer equipment and the storage medium of abnormal flow |
| CN110198274A (en) * | 2019-06-21 | 2019-09-03 | 苏宁云计算有限公司 | A kind of control method and its system of group flow |
| CN111327461A (en) * | 2020-01-23 | 2020-06-23 | 华为技术有限公司 | Domain name management method, device, equipment and medium based on CDN system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113364878B (en) | 2022-11-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10412170B2 (en) | Retention-based data management in a network-based data store | |
| US10048996B1 (en) | Predicting infrastructure failures in a data center for hosted service mitigation actions | |
| US10735509B2 (en) | Systems and methods for synchronizing microservice data stores | |
| US10652327B2 (en) | Migration of virtual machines | |
| US10860441B2 (en) | Method and system for data backup and restoration in cluster system | |
| US20190235979A1 (en) | Systems and methods for performing computing cluster node switchover | |
| CN110427284B (en) | Data processing method, distributed system, computer system, and medium | |
| US20150081877A1 (en) | Prediction and distribution of resource demand | |
| US11126506B2 (en) | Systems and methods for predictive data protection | |
| CN115277566B (en) | Load balancing method and device for data access, computer equipment and medium | |
| CN111966449B (en) | Virtual machine backup management method, system, terminal and storage medium | |
| US10409495B1 (en) | Data optimization decision control based on file system metadata | |
| CN113867957A (en) | Method and device for realizing elastic expansion of number of cross-cluster containers | |
| US11663054B2 (en) | Autoscaling of data processing computing systems based on predictive queue length | |
| US9471389B2 (en) | Dynamically tuning server placement | |
| CN113590329A (en) | Resource processing method and device | |
| CN104679575A (en) | Control system and control method for input and output flow | |
| CN110692043A (en) | System and method for load balancing backup data | |
| CN113364878B (en) | Data scheduling method and device, electronic device and storage medium | |
| US20230401103A1 (en) | System and method of dynamically adjusting virtual machines for a workload | |
| CN103685359A (en) | Data processing method and device | |
| CN105094947A (en) | Method and system for quotas management of virtual computing resource | |
| US11809299B2 (en) | Predicting storage array capacity | |
| US11381468B1 (en) | Identifying correlated resource behaviors for resource allocation | |
| CN108733511B (en) | Electronic data processing method based on big data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |